drexler-sky commented on PR #1992: URL: https://github.com/apache/datafusion-comet/pull/1992#issuecomment-3053382953
@comphead @Kontinuation Thank you for the comments. That all makes sense to me. Here’s the plan I propose: 1. Extend fs-hdfs to support passing in a Hadoop configuration programmatically, instead of relying solely on environment variables. 2. Update datafusion-comet-objectstore-hdfs to accept a config map and forward it to fs-hdfs. 3. In Comet, extract relevant Hadoop configurations from SparkConf and pass them through. While it’s currently possible to set configurations via environment variables or spark.hadoop.*, I believe enabling a more explicit and programmatic approach will improve flexibility and user experience. Does this approach sound reasonable to everyone? If so, I’ll start with step 1 and submit a PR to arrow-rs once it’s ready. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
