Xingcan Cui created SPARK-31838:
-----------------------------------
Summary: The streaming output mode validator didn't treat union
well
Key: SPARK-31838
URL: https://issues.apache.org/jira/browse/SPARK-31838
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 3.0.0
Reporter: Xingcan Cui
{{UnsupportedOperationChecker.checkForStreaming}} didn't treat the union
operator well (it only assumes a linear query plan). To temporarily fix the
broken semantics before we have a complete improvement plan for the output
mode, the validator should do the following.
1. Allow multiple (two for now) aggregations from different branches of the
last union operator.
2. For complete output mode, disallow a union on streams with and without
aggregations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
