[
https://issues.apache.org/jira/browse/BEAM-8403?focusedWorklogId=329213&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-329213
]
ASF GitHub Bot logged work on BEAM-8403:
----------------------------------------
Author: ASF GitHub Bot
Created on: 16/Oct/19 15:21
Start Date: 16/Oct/19 15:21
Worklog Time Spent: 10m
Work Description: tweise commented on issue #9800: [BEAM-8403] Guard
request id generation to prevent concurrent worker access
URL: https://github.com/apache/beam/pull/9800#issuecomment-542754461
You are right that due to the GIL there is no real parallelism. That's why
there is the option to use multiple SDK worker processes (configurable via
pipeline option `sdk_worker_parallelism`).
As part of the verification for this change I found that using 2 worker
processes instead of one leads to throughput increase of ~50%. It won't be 100%
because Python is still going to switch threads while waiting for IO etc.
It is still desirable to have a proper abstraction for something as basic as
an atomic counter. It results in code that is more readable and easier to
understand, and helps avoiding issues such as this in first place. Perhaps
that's something the Python SDK experts can consider?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 329213)
Time Spent: 2h (was: 1h 50m)
> Race condition in request id generation of GrpcStateRequestHandler
> ------------------------------------------------------------------
>
> Key: BEAM-8403
> URL: https://issues.apache.org/jira/browse/BEAM-8403
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-harness
> Reporter: Maximilian Michels
> Assignee: Maximilian Michels
> Priority: Major
> Fix For: 2.17.0
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> There is a race condition in {{GrpcStateRequestHandler}} which surfaced after
> the recent changes to process append/clear state request asynchronously. The
> race condition can occur if multiple Runner workers process a transform with
> state requests with the same SDK Harness. For example, this setup occurs with
> Flink when a TaskManager has multiple task slots and two or more of those
> slots process the same stateful stage against an SDK Harness.
> CC [~robertwb]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
