zhedoubushishi commented on issue #5291: URL: https://github.com/apache/hudi/issues/5291#issuecomment-1106118344
I cannot reproduce this issue. I just directly export the external config through the shell something like: `export HUDI_CONF_DIR=s3://wenningd-xxx/hudi/config` and in the following Hudi code I can see it is able to load the config file under that path. I am not sure what is the best way to pass env config to spark, but one thing you can check is can you print out the value of `HUDI_CONF_DIR` inside your python code to verify that this env config is passed to Spark successfully. Also are you using client mode or cluster mode? Currently this external config feature does not support spark on cluster mode. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
