[
https://issues.apache.org/jira/browse/BEAM-6326?focusedWorklogId=183839&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-183839
]
ASF GitHub Bot logged work on BEAM-6326:
----------------------------------------
Author: ASF GitHub Bot
Created on: 10/Jan/19 18:12
Start Date: 10/Jan/19 18:12
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #7453: [BEAM-6326] Fix
portable stateful processing with side input
URL: https://github.com/apache/beam/pull/7453#discussion_r246866056
##########
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPortablePipelineTranslator.java
##########
@@ -632,10 +635,38 @@ private void translateStreamingImpulse(
if (transformedSideInputs.unionTagToView.isEmpty()) {
outputStream = inputDataStream.transform(operatorName,
outputTypeInformation, doFnOperator);
} else {
- outputStream =
- inputDataStream
- .connect(transformedSideInputs.unionedSideInputs.broadcast())
- .transform(operatorName, outputTypeInformation, doFnOperator);
+ DataStream<RawUnionValue> sideInputStream =
+ transformedSideInputs.unionedSideInputs.broadcast();
+ if (stateful) {
+ // We have to manually construct the two-input transform because we're
not
+ // allowed to have only one input keyed, normally. Since Flink 1.5.0
it's
+ // possible to use the Broadcast State Pattern which provides a more
elegant
Review comment:
I don't think it is worth the effort. We have all the logic already in place
and we wouldn't gain anything from using the Broadcast State Pattern. Good
point on keyed side inputs, the current approach is generally much more
flexible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 183839)
Time Spent: 3h (was: 2h 50m)
> Fix test failures in streaming mode of PortableValidatesRunner
> --------------------------------------------------------------
>
> Key: BEAM-6326
> URL: https://issues.apache.org/jira/browse/BEAM-6326
> Project: Beam
> Issue Type: Test
> Components: runner-flink
> Reporter: Maximilian Michels
> Assignee: Maximilian Michels
> Priority: Major
> Time Spent: 3h
> Remaining Estimate: 0h
>
> As of BEAM-6009, the tests are run separately for batch and streaming. This
> has revealed issues with a couple of tests which need to be addressed.
> The Gradle task is: {{validatesPortableRunnerStreaming}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
