ericm-db commented on code in PR #48401:
URL: https://github.com/apache/spark/pull/48401#discussion_r1844389823
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala:
##########
@@ -259,6 +259,24 @@ class IncrementalExecution(
}
}
+ /**
+ * This rule populates the column family schemas for the
TransformWithStateExec
+ * operator to ship them from the driver, where the schema and serializer
objects
+ * are created, to the executor.
+ */
+ object StateStoreColumnFamilySchemasRule extends SparkPlanPartialRule {
+ override val rule: PartialFunction[SparkPlan, SparkPlan] = {
+ case statefulOp: StatefulOperator =>
+ statefulOp match {
+ case op: TransformWithStateExec =>
Review Comment:
Yeah, it will be.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
