moustafaalaa commented on PR #4167:
URL: https://github.com/apache/hudi/pull/4167#issuecomment-1094153012
@nsivabalan @vinothchandar I tried to set the path of the file using Spark
Config and Python Environment config and it doesn't work
Spark Config:
`pyspark`
```
spark =
SparkSession.builder.config('spark.serializer','org.apache.spark.serializer.KryoSerializer').config('spark.sql.hive.convertMetastoreParquet','false').config('spark.yarn.appMasterEnv.HUDI_CONF_DIR',args['HUDI_CONF_DIR']).config('spark.executorEnv.HUDI_CONF_DIR',args['HUDI_CONF_DIR']).getOrCreate()
```
Env Config:
```
os.environ['HUDI_CONF_DIR'] = args['HUDI_CONF_DIR']
```
I am getting the same error every time, I am not sure if there is a clear
example about how to use this feature with spark.
```
WARN DFSPropertiesConfiguration: Cannot find HUDI_CONF_DIR, please set it as
the dir of hudi-defaults.conf
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
