comphead commented on code in PR #7526:
URL: https://github.com/apache/datafusion-comet/pull/7526#discussion_r4488372196
##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometShuffleDependency.scala:
##########
@@ -100,6 +100,9 @@ class CometShuffleDependency[K: ClassTag, V: ClassTag, C:
ClassTag](
mapSideCombine,
shuffleWriterProcessor) {
+ /** The Comet settings used by JVM shuffle writers on executor task threads.
*/
+ private[shuffle] val writerSQLConf: CapturedSQLConf =
CapturedSQLConf.capture()
Review Comment:
For a native shuffle this captures the same Comet configs a second time:
`CometShuffleExchangeExec` already passes `CapturedSQLConf.capture()` into
`NativeShuffleSpec.sqlConf`, and both copies are serialized with the dependency
to every map task. Could this reuse it, for example
`nativeShuffleSpec.map(_.sqlConf).getOrElse(CapturedSQLConf.capture())`, so
each dependency carries one capture and the native and JVM writers cannot see
different snapshots?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
