GitHub user mbalassi opened a pull request:
https://github.com/apache/flink/pull/459
[streaming] [wip] Fault tolerance prototype (for limited case)
This is the first limited case streaming prototype. The intention is to
provide at least once processing for both states and messages from Kafka. On
failure a whole new ExecutionGraph is submitted to the cluster continuing from
the last known Kafka offset.
The PR contains three separable tasks:
1. Logic in the JobManager and TaskManager to emit barrier triggering state
checkpointing and a new actor to monitor the state (mostly @senorcarbone)
1. New functionality for the record readers handling barriers and state
handling on the stream vertex side (mostly @gyfora)
1. Low level Kafka API connector for resetting the offset on failure and
storing it as a state (mostly @mbalassi)
Heads-up:
1. @senorcarbone has a bug on the JM side
1. Kafka state is hard-coded in the stream vertex
1. The last Kafka commit is unfinished
1. All three of us are still working in the issue
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mbalassi/flink exactly-once
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/459.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #459
----
commit 750e6fb6591b82f5c756ffe2b037b967d4f5c578
Author: Gyula Fora <[email protected]>
Date: 2015-02-20T21:24:27Z
[streaming] Vertex level fault tolerance and state monitor
commit f29b8ebcb1310c4c777358eb5dcdc9c3388da746
Author: Paris Carbone <[email protected]>
Date: 2015-02-20T22:12:57Z
[streaming] Fault tolerance logic in JM and TM
commit f19eb63906c4c754eeb1e8e3b16fb80a35339b81
Author: Paris Carbone <[email protected]>
Date: 2015-03-04T17:30:09Z
[streaming] Operator state checkpointing and injection prototype
commit c879bb08dc205c2c46d40b258547c6690202bae2
Author: mbalassi <[email protected]>
Date: 2015-03-03T20:14:58Z
[FLINK-1638] [streaming] Added connector for low level Kafka Consumer API
commit 764fa5f9fc3b23f0de507d03b6161a67d0f10b06
Author: Gábor Hermann <[email protected]>
Date: 2015-03-04T11:03:14Z
[FLINK-1638] [streaming] Added Kafka topic creator and custom offset
consumer
commit 5c765d3d9899e7f200ddd83ac4064f6439f853b2
Author: mbalassi <[email protected]>
Date: 2015-03-05T10:13:00Z
[streaming] Added persistent Kafka source
Exposed state registering in the public API
commit cbfdadbd1ae810f0b1d30d673328b9d02be57b6a
Author: Gyula Fora <[email protected]>
Date: 2015-03-05T13:53:11Z
[streaming] State interface cleanup
commit dee0825834ac2c5f711f7328ce05b0890eae81fa
Author: mbalassi <[email protected]>
Date: 2015-03-05T16:34:51Z
[streaming] Kafka refactor WIP
commit b8f707e0452a1fe00a2653f6b314a93a586e5745
Author: Gyula Fora <[email protected]>
Date: 2015-03-05T18:55:14Z
Seperated AbstractRecordReader for streaming and batch
commit c64bbdb1e54bfdce28dc978e20710a76295461a1
Author: Gyula Fora <[email protected]>
Date: 2015-03-05T21:04:49Z
[streaming] Barrier sync added to CoRecordReader
Also includes a bugfix
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
