Tathagata Das created SPARK-7385:
------------------------------------
Summary: Add RDD.foreachPartitionWithIndex to allow the index to
be used in transactions in streaming apps
Key: SPARK-7385
URL: https://issues.apache.org/jira/browse/SPARK-7385
Project: Spark
Issue Type: Improvement
Components: Spark Core, Streaming
Reporter: Tathagata Das
Assignee: Tathagata Das
Spark Streaming apps often update external stores transactionally, which
requires it to have an id that uniquely identifies the partition of data to be
inserted. This can be the (time, partition index).
Current work around is to use mapPartitionsWithIndex().count() which is awkward
as it is not an action. This jira is to add foreachPartitionWithIndex().
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
