andygrove opened a new issue, #6915: URL: https://github.com/apache/datafusion-comet/issues/6915
### What is the problem the feature request solves? `CometIcebergNativeWrite.checkTriggers` calls `SessionState.newHadoopConf()` twice for every `IcebergWriteExec` it checks: once for `TriggerContext.hadoopConf` and once inside `effectiveHadoopConf`. Each call copies the whole Hadoop configuration and applies the session's SQL configs to it. In a wall-clock profile of the driver running `CometIcebergEndToEndBenchmark` (#6910), the eligibility check took about 5 ms per write statement, most of it in these copies. ### Describe the potential solution Build the session configuration once and derive the effective one from it, and build both lazily, only for the rules that read them (the S3 and GCS checks), or once per query. ### Additional context `checkTriggers` and `effectiveHadoopConf` in `spark/src/main/scala/org/apache/comet/serde/operator/CometIcebergNativeWrite.scala`. Part of #6911. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
