[
https://issues.apache.org/jira/browse/BEAM-7547?focusedWorklogId=269394&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-269394
]
ASF GitHub Bot logged work on BEAM-7547:
----------------------------------------
Author: ASF GitHub Bot
Created on: 28/Jun/19 17:09
Start Date: 28/Jun/19 17:09
Worklog Time Spent: 10m
Work Description: scwhittle commented on issue #8842: [BEAM-7547] Avoid
WindmillStateCache cache hits for stale work.
URL: https://github.com/apache/beam/pull/8842#issuecomment-506808925
@lukecwik PTAL the tests are passing now. The build issue was due to a
recent commit which I hadn't merged but which apparently Jenkins merged
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 269394)
Time Spent: 2h 20m (was: 2h 10m)
> StreamingDataflowWorker can observe inconsistent cache for stale work items
> ---------------------------------------------------------------------------
>
> Key: BEAM-7547
> URL: https://issues.apache.org/jira/browse/BEAM-7547
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
> Reporter: Sam Whittle
> Assignee: Sam Whittle
> Priority: Minor
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> 1. Dataflow backend generates a work item with a cache token C.
> 2. StreamingDataflowWorker receives the work item and reads the state using
> C, it either hits the cache or performs a read.
> 3. Dataflow backend sends a retry of the work item (possibly because it
> thinks original work item never reached the StreamingDataflowWorker).
> 4. StreamingDataflowWorker commits the work item and gets ack from dataflow
> backend. It caches the state for the key using C.
> 5. StreamingDataflowWorker receives the retried work item with cache token C.
> It uses the cached state and causes possible user consistency failures
> because the cache view is of after the work item completed processing.
> Note that this will not cause corrupted Dataflow persistent state because the
> commit of the retried work item using the inconsistent cache will fail.
> However it may cause failures in user logic for example if they keep the set
> of all seen items in state and throw an exception on duplicates which should
> have been removed by an upstream stage.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
