Burak Yavuz created SPARK-22238:
-----------------------------------
Summary: EnsureStatefulOpPartitioning shouldn't ask for the child
RDD before planning is completed
Key: SPARK-22238
URL: https://issues.apache.org/jira/browse/SPARK-22238
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 2.3.0
Reporter: Burak Yavuz
Assignee: Burak Yavuz
In EnsureStatefulOpPartitioning, we check that the inputRDD to a SparkPlan has
the expected partitioning for Streaming Stateful Operators. The problem is that
we are not allowed to access this information during planning.
The reason we added that check was because CoalesceExec could actually create
RDDs with 0 partitions. We should fix it such that when CoalesceExec says that
there is a SinglePartition, there is in fact an inputRDD of 1 partition instead
of 0 partitions.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
