andygrove commented on issue #6399: URL: https://github.com/apache/datafusion-comet/issues/6399#issuecomment-5933839570
Phase 3, memory, FFI, config and shims: all 7 PRs have been reviewed against 1.0.0. None of them has a regression to count, so there was nothing to run on the 1.0.0 and rc1 builds for this area. Four findings aren't counted, but are worth knowing: - #6025's IRSA credential provider is on by default and doesn't fall back. 1.0.0 fell back to the node role or Pod Identity when the web-identity call failed, so a cluster with broken IRSA that quietly read S3 as the node role now fails native Iceberg reads with "failed to load signing credential". Not falling back is the point of the PR, and it fixes #6024's 403 storm, but the 1.1.0 section of the upgrade guide doesn't mention it. The documented opt-out is per catalog, so path-loaded Iceberg tables can only opt out with `spark.comet.scan.icebergNative.enabled=false`. - #6191 now rejects an invalid `spark.comet.debug.memory` value such as `yes` instead of ignoring it, which fails every native plan. It's a testing setting, but one sentence in the upgrade guide would cover it. - #5552 sends a broadcast `UNION ALL` uncoalesced when its branches carry different top-level field metadata. The results are correct, and the PR accepted that cost on purpose. - #6162, which isn't in this group, made the allocation-tracking wrapper always on, and it costs about 2% on TPC-H Q21 after #6166. No regression was found in #5368, #5493, #5539, #6166 or #6191's intended changes, which the upgrade guide documents. #5368's code only runs with `spark.comet.exec.pyarrowUDF.enabled`, and #5493's riskiest code was test-only and removed by #6066 before rc1. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
