David Snowberger created BEAM-6576:
--------------------------------------
Summary: Python SDK, DataflowRunner streaming job stops without
error.
Key: BEAM-6576
URL: https://issues.apache.org/jira/browse/BEAM-6576
Project: Beam
Issue Type: Bug
Components: runner-dataflow
Affects Versions: 2.9.0
Reporter: David Snowberger
Assignee: Tyler Akidau
Hello, hopefully I'm reporting this in the right location -
I have a simple dataflow job that reads from Pub/Sub, does a few transforms,
and writes to Redis (in another PTransform at the moment, since no Python I/O
connector). It runs perfectly well for several days, then the elements just
stop being processed. If I update the job, or stop and start, it carries on
again for a few more days. I believe I am not the only one to have run into
this issue, as I found mention on [[Stack
Overflow|https://stackoverflow.com/questions/53610876/dataflow-stops-streaming-to-bigquery-without-errors]|https://stackoverflow.com/questions/53610876/dataflow-stops-streaming-to-bigquery-without-errors].
The only indication of workers quitting I've discovered so far are these info
logs:
{code:java}
[0129/074659:INFO:update_manager-inl.h(52)] ChromeOSPolicy::UpdateCheckAllowed:
START
[0129/074659:WARNING:evaluation_context-inl.h(43)] Error reading Variable
update_disabled: "No value set for update_disabled"
[0129/074659:WARNING:evaluation_context-inl.h(43)] Error reading Variable
release_channel_delegated: "No value set for release_channel_delegated"
[0129/074659:INFO:chromeos_policy.cc(317)] Periodic check interval not
satisfied, blocking until 1/29/2019 8:26:37 GMT
[0129/074659:INFO:update_manager-inl.h(74)] ChromeOSPolicy::UpdateCheckAllowed:
END
{code}
I have also filed a Dataflow bug, as I'm not sure where the problem lies. Any
help resolving this would be very welcome, and If I can provide anything
further, please let me know. Thank you!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
