moustafaalaa opened a new issue, #5291: URL: https://github.com/apache/hudi/issues/5291
**Describe the problem you faced** I tried to use Hudi `hudi-defaults.conf` with Glue and tried to set the path of the file using Spark Config and Python Environment config and it doesn't work. I checked this issue https://github.com/apache/hudi/pull/4167 but i can't find a clear idea about how to use it. Spark Config: pyspark ``` spark = SparkSession.builder.config('spark.serializer','org.apache.spark.serializer.KryoSerializer') .config('spark.sql.hive.convertMetastoreParquet','false') .config('spark.yarn.appMasterEnv.HUDI_CONF_DIR',args['HUDI_CONF_DIR']) .config('spark.executorEnv.HUDI_CONF_DIR',args['HUDI_CONF_DIR']).getOrCreate() ``` Env Config: ``` os.environ['HUDI_CONF_DIR'] = args['HUDI_CONF_DIR'] ``` I am getting the same error every time, I am not sure if there is a clear example about how to use this feature with spark or Glue. **Expected behavior** A clear and concise description of what you expected to happen. **Environment Description** * Hudi version : 0.10.1 * Spark version : 3.1.1 * Hive version : 2.3.7 * Storage (HDFS/S3/GCS..) : S3 * Running on Docker? (yes/no) : no ```Add the stacktrace of the error.``` ``` WARN DFSPropertiesConfiguration: Cannot find HUDI_CONF_DIR, please set it as the dir of hudi-defaults.conf ``` In the test example, it uses DFSPropertiesConfiguration.refreshGlobalProps(); to refresh, but i am not sure how to use this with pyspark config. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
