Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/16547
Discussed with @tdas offline. Regarding the changes to the append mode,
it's unknown that if adding the filter is better because it will apply the
filter on all rows but there are usually only few rows out of watermark. It's
better to just leave it as it it.
For aborting StateStore, could you use
`TaskContext.addTaskCompletionListener` instead? So that if the error doesn't
happen in `hasNext` or `next` method, we can still abort it. You can add a flag
to indicate if the StateStore is committed or not.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
