viirya opened a new pull request, #7120: URL: https://github.com/apache/datafusion-comet/pull/7120
## Which issue does this PR close? No issue filed. These are two small follow-ups found while reading recent changes; neither changes query results. ## Rationale for this change - `StddevAccumulator::size()` computed `align_of_val(self) - align_of_val(&self.variance) + self.variance.size()`. The sibling accumulators (`regr.rs`, `correlation.rs`) use `size_of_val`. The value is correct today only because the struct has a single field, so the mistake would surface as soon as a field is added. - Since #6838, `CometScanRule.nativeScan` calls `sessionState.newHadoopConf()` for every V1 scan to read the session's `fs.comet.libhdfs.schemes`. `newHadoopConf()` copies the whole Hadoop `Configuration`, and the rule runs again under AQE re-planning, so plans with many scans pay that copy repeatedly. ## What changes are included in this PR? - `stddev.rs`: use `size_of_val` in `StddevAccumulator::size()`. - `NativeConfig`: add a `resolveScanLibhdfsSchemes(sessionSchemes: Set[String], scanConf)` overload. The existing `Configuration` overload delegates to it. - `CometScanRule`: resolve the session's libhdfs schemes once per relation session within one rule application (lazily, on the first V1 scan that reaches the scheme gate) and reuse the set for every scan. Semantics are unchanged: the session list is still read from the session Hadoop conf, so `core-site.xml` is honored, and it is still intersected with each scan's own list. The Delta contrib (`DeltaScanSupport`) builds the session conf per scan in the same way, but it is reached through the `CometScanContrib` SPI, which has no per-application context. It also already builds a per-scan conf with `newHadoopConfWithOptions`, so this PR leaves it as is. ## How are these changes tested? - New Rust unit test `scalar_size_covers_struct` checks that `size()` is at least `size_of::<StddevAccumulator>()`. `cargo test -p datafusion-comet-spark-expr stddev`, `cargo clippy` and `cargo fmt` pass. - `NativeConfigSuite`'s `resolveScanLibhdfsSchemes` test now also checks that the new overload returns the same result on every case. - `CometScanSchemeFallbackSuite` and `NativeConfigSuite` pass (54 tests). This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
