[
https://issues.apache.org/jira/browse/SPARK-31838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xingcan Cui updated SPARK-31838:
--------------------------------
Description:
{{UnsupportedOperationChecker.checkForStreaming}} didn't treat the union
operator well (it only assumes a linear query plan). To temporarily fix the
broken semantics before we have a complete improvement plan for the output
mode, the validator should do the following.
1. Allow multiple aggregations from different branches of a union operator.
2. For complete output mode, disallow a union on streams with and without
aggregations.
was:
{{UnsupportedOperationChecker.checkForStreaming}} didn't treat the union
operator well (it only assumes a linear query plan). To temporarily fix the
broken semantics before we have a complete improvement plan for the output
mode, the validator should do the following.
1. Allow multiple aggregations from different branches of the last union
operator.
2. For complete output mode, disallow a union on streams with and without
aggregations.
> The streaming output mode validator didn't treat union well
> -----------------------------------------------------------
>
> Key: SPARK-31838
> URL: https://issues.apache.org/jira/browse/SPARK-31838
> Project: Spark
> Issue Type: Sub-task
> Components: Structured Streaming
> Affects Versions: 3.0.0
> Reporter: Xingcan Cui
> Priority: Major
>
> {{UnsupportedOperationChecker.checkForStreaming}} didn't treat the union
> operator well (it only assumes a linear query plan). To temporarily fix the
> broken semantics before we have a complete improvement plan for the output
> mode, the validator should do the following.
> 1. Allow multiple aggregations from different branches of a union operator.
> 2. For complete output mode, disallow a union on streams with and without
> aggregations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
