[GitHub] spark pull request #17252: [SPARK-19913][SS] Log warning rather than throw A...

[sarutak]

Github user sarutak closed the pull request at:

https://github.com/apache/spark/pull/17252


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17252: [SPARK-19913][SS] Log warning rather than throw A...

[sarutak]

GitHub user sarutak opened a pull request:

https://github.com/apache/spark/pull/17252

[SPARK-19913][SS] Log warning rather than throw AnalysisException when 
output is partitioned although format is memory, console or foreach

## What changes were proposed in this pull request?

When batches are executed with memory, console or foreach format, 
`assertNotPartitioned` will check whether output is not partitioned and throw 
AnalysisException in case it is.

But I wonder it's better to log warning rather than throw the exception 
because partitioning does not affect output for those formats but also does not 
bring any negative impacts.

Also, this assertion is not applied when the format is `console`. I think 
in this case too, we should assert that .

By fixing them, we can easily switch the format to memory or console for 
debug purposes.

## How was this patch tested?

I tested manually for memory, foreach and console formats and confirm that 
warning is displayed.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sarutak/spark SPARK-19913

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/17252.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #17252


commit 12b251c8d32d79f74f74b65eb62d08994f83da5a
Author: Kousuke Saruta 
Date:   2017-03-10T23:35:41Z

Log warning rather than throw exception when MemorySink or ForeachSink is 
used with partitioning

commit 7372aa509453d94db5cf267cc8006c78c1be54ef
Author: Kousuke Saruta 
Date:   2017-03-10T23:36:01Z

Log warning when output is partitioned though the format is console




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org