mixermt commented on PR #5898: URL: https://github.com/apache/datafusion-comet/pull/5898#issuecomment-6454295593
1. I'll hold the bump on apache/iceberg-rust#3437 (approved) and move past it once it merges, so the pruned read returns NULL like Spark. 2. Since #7085, main already falls back for every nested column of a name-mapped table; after syncing, this PR inherits that gate and drops its own narrower one. Narrowing it to structs is #6934's follow-up. #7028 carries the same bump; I can sequence this PR after it so the diff is HDFS-only. 3. hdfs-native has `with_user`, but opendal doesn't expose it and iceberg-rust ignores `hdfs.user`. Until that is wired upstream, the planner claims native HDFS only when the executors' identity is known to be the Spark user (`spark.executorEnv.HADOOP_USER_NAME` set to it; in local mode the JVM's user), and falls back naming the setting otherwise. 4. The probe now groups directories by parent and uses one `listStatus` per parent (the NameNode returns each child's encrypted flag), caches per filesystem and directory for the session, and the rule runs once per query (AQE re-plans unexecuted scans only). Errors keep declining; the reason is in EXPLAIN, not only the log. 5. Other changes in the range on default paths: #3364 fixes task failures on short BYTE_ARRAY decimal bounds; #3070 makes an equality delete on case-colliding names error instead of picking one; #3473 reads no columns for an empty projection (COUNT(*) suites pass); #2930 adds `arrow.uuid` field metadata (uuid suites pass). The rest is API, catalog or CI. 6. Added an end-to-end test: metadata on one NameNode, a data file on the HA nameservice, one native scan reads both. 7. An undeclared portless host now resolves to port 8020, as `DFSUtilClient.getNNAddress` does for the JVM client; a name listed only in `dfs.nameservices` still falls back. 8. Description rewritten as current state. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
