comphead commented on PR #1992:
URL:
https://github.com/apache/datafusion-comet/pull/1992#issuecomment-3053785263
> @comphead @Kontinuation
>
> Thank you for the comments. That all makes sense to me.
>
> Here’s the plan I propose:
>
> 1. Extend fs-hdfs to support passing in a Hadoop configuration
programmatically, instead of relying solely on environment variables.
> 2. Update datafusion-comet-objectstore-hdfs to accept a config map and
forward it to fs-hdfs.
> 3. In Comet, extract relevant Hadoop configurations from SparkConf and
pass them through.
>
> While it’s currently possible to set configurations via environment
variables or spark.hadoop.*, I believe enabling a more explicit and
programmatic approach will improve flexibility and user experience.
>
> Does this approach sound reasonable to everyone? If so, I’ll start with
step 1 and submit a PR to arrow-rs once it’s ready.
Thanks @drexler-sky I think this approach makes a lot of sense. You probably
also want to cover the cases if multiple config sources are set, like env var,
spark hadoop and programmatical, which one should be overriding others.
@parthchandra I'm not sure if `libhdfs` packaged
with `hdfs` feature and without the `libcomet.dylib` has the same size of
69M, so static linking is unlikely. And checking dynamic libs
with `hdfs` (`libjvm` linked)
```
otool -L native/target/release/libcomet.dylib
native/target/release/libcomet.dylib:
/opt/homebrew/opt/openjdk@11/libexec/openjdk.jdk/Contents/Home/lib/server/libjvm.dylib
(compatibility version 1.0.0, current version 1.0.0)
/System/Library/Frameworks/Security.framework/Versions/A/Security
(compatibility version 1.0.0, current version 61439.120.27)
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
(compatibility version 150.0.0, current version 3502.1.255)
/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current
version 7.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 1351.0.0)
MacBook-Pro-110:arrow-datafusion-comet ovoievodin$ ls -la
native/target/release/
```
without `hdfs`
```
/System/Library/Frameworks/Security.framework/Versions/A/Security
(compatibility version 1.0.0, current version 61439.120.27)
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
(compatibility version 150.0.0, current version 3502.1.255)
/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current
version 7.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 1351.0.0)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
