[
https://issues.apache.org/jira/browse/SAMZA-2531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharath Kumarasubramanian updated SAMZA-2531:
---------------------------------------------
Description:
We introduced non-blocking commit feature as part of SAMZA-1140. Currently the
commit is only non-blocking and allows process to be run concurrently.
As part of this task, we will need to evaluate what it takes to make it run
concurrently along with window, timer & end of stream and if it is necessary in
the first place. If not, rename the feature to explicitly callout process
exclusive commit instead of async commit.
Some context:
* Transactional state currently doesn't work well w/ async commit turned on.
i.e. Even though async commit, uses low watermark approach to find the offset
to safely commit, it still doesn't guarantee the state snapshots happen during
the commit correspond to the offset that is committed to checkpoint topic.
* Window, timer could potentially mutate the state in the same manner as
process; So it might be okay to phase out (if we decide to support) the changes
to introduce concurrency within window & timer first and follow it up with
synchronizing the state w/ the offset that is committed.
was:
We introduced non-blocking commit feature as part of SAMZA-1140. Currently the
commit is only non-blocking and allows process to be run concurrently.
As part of this task, we will need to evaluate what it takes to make it run
concurrently along with window, timer & end of stream and if it is necessary in
the first place. If not, rename the feature to explicitly callout process
exclusive commit instead of async commit.
> Async Commit is Blocking for Window, Timer & End of Stream
> ----------------------------------------------------------
>
> Key: SAMZA-2531
> URL: https://issues.apache.org/jira/browse/SAMZA-2531
> Project: Samza
> Issue Type: Task
> Reporter: Bharath Kumarasubramanian
> Priority: Major
>
> We introduced non-blocking commit feature as part of SAMZA-1140. Currently
> the commit is only non-blocking and allows process to be run concurrently.
> As part of this task, we will need to evaluate what it takes to make it run
> concurrently along with window, timer & end of stream and if it is necessary
> in the first place. If not, rename the feature to explicitly callout process
> exclusive commit instead of async commit.
> Some context:
> * Transactional state currently doesn't work well w/ async commit turned on.
> i.e. Even though async commit, uses low watermark approach to find the offset
> to safely commit, it still doesn't guarantee the state snapshots happen
> during the commit correspond to the offset that is committed to checkpoint
> topic.
> * Window, timer could potentially mutate the state in the same manner as
> process; So it might be okay to phase out (if we decide to support) the
> changes to introduce concurrency within window & timer first and follow it up
> with synchronizing the state w/ the offset that is committed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
