andygrove opened a new issue, #7637: URL: https://github.com/apache/datafusion-comet/issues/7637
### What is the problem the feature request solves? Native Iceberg writes fall back to iceberg-java for an `hdfs` data location. The gate in `CometIcebergNativeWrite` admits no `hdfs` scheme, so every write to an Iceberg table on HDFS runs on the JVM writer. This no longer waits on iceberg-rust. apache/iceberg-rust#3111 added the `hdfs-native` storage backend, and #5898 already reads Iceberg tables on HDFS natively through it (#5894). The remaining work is on Comet's side. ### Describe the potential solution Admit `hdfs` data locations in the native write gate, and route writes through the same `hdfs-native` backend and configuration the native scan uses. Keep the cases the native scan declines falling back, in particular a cluster with Kerberos and a location whose nameservice cannot be resolved from the Hadoop configuration. ### Additional context Part of #5649. Recorded under "Plans for each fallback" in the Iceberg user guide (#6961). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
