Chris Fregly created SPARK-5959:
-----------------------------------
Summary: Create a ReliableKinesisReceiver similar to the
ReliableKafkaReceiver
Key: SPARK-5959
URL: https://issues.apache.org/jira/browse/SPARK-5959
Project: Spark
Issue Type: Improvement
Reporter: Chris Fregly
Fix For: 1.4.0
After each block is stored reliably in the WAL (after the store() call
returns), ACK back to Kinesis.
There is still the issue of the ReliableKinesisReceiver dying before the ACK
back to Kinesis, however no data will be lost. Duplicate data is still
possible.
Notes
* Make sure we're not overloading the checkpoint control plane which uses
DynamoDB.
* May need to disable auto-checkpointing and remove the checkpoint interval.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
