andygrove opened a new pull request, #6925:
URL: https://github.com/apache/datafusion-comet/pull/6925
## Which issue does this PR close?
Closes #6915.
## Rationale for this change
Native Iceberg write eligibility currently builds two Hadoop configurations
on every planning pass. Each call to `SessionState.newHadoopConf()` copies the
full Hadoop configuration and applies the session SQL configuration, even
though most eligibility rules do not inspect Hadoop settings.
## What changes are included in this PR?
- Build the session Hadoop configuration lazily, only when a rule reads it.
- Derive the effective S3 Hadoop configuration lazily from that same session
configuration, avoiding the second copy.
- Add regression coverage for lazy construction, single construction, and
reuse of the session configuration.
This changes planning cost only. It does not widen or narrow native Iceberg
write eligibility.
## How are these changes tested?
- `make format`
- `make core`
- `JAVA_TOOL_OPTIONS="${JAVA_TOOL_OPTIONS:+$JAVA_TOOL_OPTIONS
}-Dspark.ui.enabled=false" ./mvnw test -Dtest=none
-Dsuites="org.apache.comet.CometIcebergWriteDetectionSuite"` (81 tests passed)
- `./mvnw test-compile -Pspark-3.5 -Pstrict-warnings -DskipTests`
- `git diff --check`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
