[
https://issues.apache.org/jira/browse/SPARK-4964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14293129#comment-14293129
]
Tathagata Das edited comment on SPARK-4964 at 1/27/15 7:53 AM:
---------------------------------------------------------------
I am renaming this JIRA to "Exactly-once + WAL-free Kafka Support in Spark
Streaming" because there are two problems that we are trying to solve, which
gets solved by the associated PR. See the design doc for more details.
Also, I updated the description to reflect the two issues, and added references
to the design doc.
was (Author: tdas):
I am renaming this JIRA to "Exactly-once + WAL-free Kafka Support in Spark
Streaming
" because there are two problems that we are trying to solve, which gets solved
by the associated PR. See the design doc for more details.
> Exactly-once + WAL-free Kafka Support in Spark Streaming
> --------------------------------------------------------
>
> Key: SPARK-4964
> URL: https://issues.apache.org/jira/browse/SPARK-4964
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
> Reporter: Cody Koeninger
>
> There are two issues with the current Kafka support
> - Use of Write Ahead Logs in Spark Streaming to ensure no data is lost -
> Causes data replication in both Kafka AND Spark Streaming.
> - Lack of exactly-once semantics - For background, see
> http://apache-spark-developers-list.1001551.n3.nabble.com/Which-committers-care-about-Kafka-td9827.html
> We want to solve both these problem in JIRA. Please see the following design
> doc for the solution.
> https://docs.google.com/a/databricks.com/document/d/1IuvZhg9cOueTf1mq4qwc1fhPb5FVcaRLcyjrtG4XU1k/edit#heading=h.itproy77j3p
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
