Amit Sela created BEAM-648:
------------------------------
Summary: Persist and restore Aggergator values in case of recovery
from failure
Key: BEAM-648
URL: https://issues.apache.org/jira/browse/BEAM-648
Project: Beam
Issue Type: Bug
Components: runner-spark
Reporter: Amit Sela
Assignee: Amit Sela
While Aggregators are fault-tolerant (pending
https://github.com/apache/incubator-beam/pull/909), they will recover into
their initial value.
The SparkRunner should persist the values to durable storage and recover from
the persisted state.
One way to go would be to persist via Spark's listeners, for example
"onBatchCompleted".
For more details see SPARK-5206.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
