[
https://issues.apache.org/jira/browse/SPARK-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tathagata Das updated SPARK-6599:
---------------------------------
Description:
Usability improvements:
API improvements, AWS SDK upgrades,
Reliability improvements:
Currently, the KinesisReceiver can loose some data in the case of certain
failures (receiver and driver failures). Using the write ahead logs can
mitigate some of the problem, but it is not ideal because WALs dont work with
S3 (eventually consistency, etc.) which is the most likely file system to be
used in the EC2 environment. Hence, we have to take a different approach to
improving reliability for Kinesis. See
https://issues.apache.org/jira/browse/SPARK-9215 for more details.
was:
Usability improvements:
Reliability improvements:
Currently, the KinesisReceiver can loose some data in the case of certain
failures (receiver and driver failures). Using the write ahead logs can
mitigate some of the problem, but it is not ideal because WALs dont work with
S3 (eventually consistency, etc.) which is the most likely file system to be
used in the EC2 environment. Hence, we have to take a different approach to
improving reliability for Kinesis.
A detailed design doc -
> Improve usability of Kinesis-based Spark Streaming
> --------------------------------------------------
>
> Key: SPARK-6599
> URL: https://issues.apache.org/jira/browse/SPARK-6599
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
> Reporter: Tathagata Das
> Assignee: Tathagata Das
>
> Usability improvements:
> API improvements, AWS SDK upgrades,
> Reliability improvements:
> Currently, the KinesisReceiver can loose some data in the case of certain
> failures (receiver and driver failures). Using the write ahead logs can
> mitigate some of the problem, but it is not ideal because WALs dont work with
> S3 (eventually consistency, etc.) which is the most likely file system to be
> used in the EC2 environment. Hence, we have to take a different approach to
> improving reliability for Kinesis. See
> https://issues.apache.org/jira/browse/SPARK-9215 for more details.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
