Kousuke Saruta created SPARK-19913:
--------------------------------------
Summary: Log warning rather than throw AnalysisException when
output is partitioned although format is memory, console or foreach
Key: SPARK-19913
URL: https://issues.apache.org/jira/browse/SPARK-19913
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 2.2.0
Reporter: Kousuke Saruta
Priority: Minor
When batches are executed with memory, console or foreach format,
`assertNotPartitioned` will check whether output is not partitioned and throw
AnalysisException in case it is.
But I wonder it's better to log warning rather than throw the exception because
partitioning does not affect output for those formats but also does not bring
any negative impacts.
Also, this assertion is not applied when the format is `console`. I think in
this case too, we should assert that .
By fixing them, we can easily switch the format to memory or console for debug
purposes.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
