[jira] [Commented] (SPARK-24396) Add Structured Streaming ForeachWriter for python
[ https://issues.apache.org/jira/browse/SPARK-24396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497283#comment-16497283 ] Apache Spark commented on SPARK-24396: -- User 'tdas' has created a pull request for this issue: https://github.com/apache/spark/pull/21477 > Add Structured Streaming ForeachWriter for python > - > > Key: SPARK-24396 > URL: https://issues.apache.org/jira/browse/SPARK-24396 > Project: Spark > Issue Type: New Feature > Components: Structured Streaming >Affects Versions: 2.3.0 >Reporter: Tathagata Das >Assignee: Tathagata Das >Priority: Major > > Users should be able to write ForeachWriter code in python, that is, they > should be able to use the partitionid and the version/batchId/epochId to > conditionally process rows. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24396) Add Structured Streaming ForeachWriter for python
[ https://issues.apache.org/jira/browse/SPARK-24396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491415#comment-16491415 ] Tathagata Das commented on SPARK-24396: --- TaskContext.getLocalProperty in Python is needed for getting the batchId/epochId that is passed on by StreamExecution as Spark job local property. > Add Structured Streaming ForeachWriter for python > - > > Key: SPARK-24396 > URL: https://issues.apache.org/jira/browse/SPARK-24396 > Project: Spark > Issue Type: New Feature > Components: Structured Streaming >Affects Versions: 2.3.0 >Reporter: Tathagata Das >Assignee: Tathagata Das >Priority: Major > > Users should be able to write ForeachWriter code in python, that is, they > should be able to use the partitionid and the version/batchId/epochId to > conditionally process rows. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
