[
https://issues.apache.org/jira/browse/BEAM-6008?focusedWorklogId=371454&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-371454
]
ASF GitHub Bot logged work on BEAM-6008:
----------------------------------------
Author: ASF GitHub Bot
Created on: 14/Jan/20 08:59
Start Date: 14/Jan/20 08:59
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #10566: [BEAM-6008]
Propagate errors during pipeline execution in Java's PortableRunner
URL: https://github.com/apache/beam/pull/10566#discussion_r366216365
##########
File path:
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/JobInvocation.java
##########
@@ -243,6 +249,7 @@ private synchronized void setState(JobState.Enum state) {
}
private synchronized void sendMessage(JobMessage message) {
+ messageHistory.add(message);
Review comment:
> yeah, I'd be wary of storing all of that.
We have the same problem for the state updates which are smaller but also
can arrive at a higher frequency. How about (1) limiting the number of past
invocations? We would keep all the state updates / messages per invocation. (2)
An alternative could be to just store the error-related messages. (3) A third
option would be to subscribe to the message stream early before pipeline
execution starts.
I'm leaning towards (1) because that would still allow the retrieval of past
state updates that I think was a requirement for you @chadrik.
>Meaning there should be only 2 messages per pipeline failure. Unless
there's something I'm missing?
That's right. However, the messages also contain the state updates which we
already save. So there is some room for optimization the storage.
Will follow-up regarding the build time but it looks like it almost doubled
which we should definitely take serious.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 371454)
Time Spent: 5h 40m (was: 5.5h)
> Propagate errors through portable runner
> ----------------------------------------
>
> Key: BEAM-6008
> URL: https://issues.apache.org/jira/browse/BEAM-6008
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core, sdk-py-core
> Reporter: Robert Bradshaw
> Priority: Major
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
