JingsongLi commented on code in PR #8417:
URL: https://github.com/apache/paimon/pull/8417#discussion_r3519271920
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/commands/PaimonSparkWriter.scala:
##########
@@ -59,6 +59,34 @@ case class PaimonSparkWriter(
batchId: Option[Long] = None)
extends WriteHelper {
+ private case class WriteContext(
Review Comment:
The task closures below capture this whole `WriteContext` via
`ctx.newWrite()` / `ctx.bucketColIdx`. Because the context also stores
driver-side objects (`SparkSession`, the original `DataFrame`, and
`preparedData`), those objects become part of every executor closure even
though they are only needed while building the plan on the driver. This can
make Spark closure serialization fail or drag SparkContext/Dataset state into
tasks. Please keep `WriteContext` limited to serializable task inputs, or
extract only the needed scalars/functions before `mapPartitions` and keep
`SparkSession`/`DataFrame` out of the captured object.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
