[
https://issues.apache.org/jira/browse/SPARK-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Fregly updated SPARK-5959:
--------------------------------
Component/s: Streaming
Description:
After each block is stored reliably in the WAL (after the store() call
returns), ACK back to Kinesis.
There is still the issue of the ReliableKinesisReceiver dying before the ACK
back to Kinesis, however no data will be lost. Duplicate data is still
possible.
Notes:
* Make sure we're not overloading the checkpoint control plane which uses
DynamoDB.
* May need to disable auto-checkpointing and remove the checkpoint interval.
* Maintain compatibility with existing KinesisReceiver-based code.
was:
After each block is stored reliably in the WAL (after the store() call
returns), ACK back to Kinesis.
There is still the issue of the ReliableKinesisReceiver dying before the ACK
back to Kinesis, however no data will be lost. Duplicate data is still
possible.
Notes
* Make sure we're not overloading the checkpoint control plane which uses
DynamoDB.
* May need to disable auto-checkpointing and remove the checkpoint interval.
Target Version/s: 1.4.0
Affects Version/s: 1.1.0
Fix Version/s: (was: 1.4.0)
> Create a ReliableKinesisReceiver similar to the ReliableKafkaReceiver
> ---------------------------------------------------------------------
>
> Key: SPARK-5959
> URL: https://issues.apache.org/jira/browse/SPARK-5959
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
> Affects Versions: 1.1.0
> Reporter: Chris Fregly
>
> After each block is stored reliably in the WAL (after the store() call
> returns), ACK back to Kinesis.
> There is still the issue of the ReliableKinesisReceiver dying before the ACK
> back to Kinesis, however no data will be lost. Duplicate data is still
> possible.
> Notes:
> * Make sure we're not overloading the checkpoint control plane which uses
> DynamoDB.
> * May need to disable auto-checkpointing and remove the checkpoint interval.
> * Maintain compatibility with existing KinesisReceiver-based code.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
