[
https://issues.apache.org/jira/browse/BEAM-10097?focusedWorklogId=446071&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446071
]
ASF GitHub Bot logged work on BEAM-10097:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 15/Jun/20 20:15
Start Date: 15/Jun/20 20:15
Worklog Time Spent: 10m
Work Description: lukecwik commented on a change in pull request #11821:
URL: https://github.com/apache/beam/pull/11821#discussion_r440421252
##########
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Combine.java
##########
@@ -1302,13 +1303,35 @@ private GloballyAsSingletonView(
@Override
public PCollectionView<OutputT> expand(PCollection<InputT> input) {
+ // TODO(BEAM-10097): Make this the default expansion for all portable
runners.
+ if (hasExperiment(input.getPipeline().getOptions(), "beam_fn_api")
+ && (hasExperiment(input.getPipeline().getOptions(), "use_runner_v2")
Review comment:
`use_runner_v2` is preferred but internal test framework within Google
has not been fully migrated to use one over the other. Also Googlers typically
use `use_unified_worker`
##########
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkSideInputReader.java
##########
@@ -71,8 +79,23 @@ public FlinkSideInputReader(
tag.getId(), new SideInputInitializer<>(view));
T result = sideInputs.get(window);
if (result == null) {
- ViewFn<MultimapView, T> viewFn = (ViewFn<MultimapView, T>)
view.getViewFn();
- result = viewFn.apply(InMemoryMultimapSideInputView.empty());
Review comment:
In this case the side input is a valid side input but Flink says it has
no data for it by returning null. Also `viewFn.apply` expects a non-null value
which is why we pass in an empty view.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 446071)
Time Spent: 5.5h (was: 5h 20m)
> Migrate PCollection views to use both iterable and multimap
> materializations/access patterns
> --------------------------------------------------------------------------------------------
>
> Key: BEAM-10097
> URL: https://issues.apache.org/jira/browse/BEAM-10097
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core, sdk-java-harness
> Reporter: Luke Cwik
> Assignee: Luke Cwik
> Priority: P2
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> Currently all the PCollection views have a trival mapping from KV<void,
> Iterable<T>> to the view that is being requested (singleton, iterable, list,
> map, multimap.
> We should be using the primitive views (iterable, multimap) directly without
> going through the naive mapping.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
