Daniel Chen created SAMZA-2590:
-----------------------------------
Summary: Async State Backup
Key: SAMZA-2590
URL: https://issues.apache.org/jira/browse/SAMZA-2590
Project: Samza
Issue Type: New Feature
Reporter: Daniel Chen
Assignee: Daniel Chen
# We must create a way of committing to the state store that allows for the
remote stores to be uploaded independently from processing.
# The commit phase must be transactional for Kafka changelog commits (preserve
existing behaviour) and remote store commits.
# For remote store commits, we require exclusive, non-overlapping commits so
that the commits that appear in the remote store are in order.
# We need to timeout on the asynchronous commit such that we will stop further
processing until the state in the remote store is caught up to the current
state of the job.
# This feature will be required to be enabled by default of all Samza jobs
regardless of the remote state store backup implementation (Kafka, Azure blob
store, HDFS) and the processing parallelism (async or sync processing) to
improve performance for all present and future remote state commits.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
