[
https://issues.apache.org/jira/browse/BEAM-4285?focusedWorklogId=116936&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-116936
]
ASF GitHub Bot logged work on BEAM-4285:
----------------------------------------
Author: ASF GitHub Bot
Created on: 28/Jun/18 18:25
Start Date: 28/Jun/18 18:25
Worklog Time Spent: 10m
Work Description: jkff commented on a change in pull request #5814:
[BEAM-4285] Extend side input handlers to handle multiple access patterns.
URL: https://github.com/apache/beam/pull/5814#discussion_r198939409
##########
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkBatchSideInputHandlerFactory.java
##########
@@ -101,28 +154,53 @@ private FlinkBatchSideInputHandlerFactory(
}
}
- return new SideInputHandler<>(multimap.build(), keyCoder, windowCoder);
+ return new MultimapSideInputHandler(multimap.build(), keyCoder,
valueCoder, windowCoder);
}
- private static class SideInputHandler<K, V, W extends BoundedWindow>
- implements MultimapSideInputHandler<K, V, W> {
+ private <T> List<WindowedValue<T>> getBroadcastVariable(String transformId,
String sideInputId) {
+ PCollectionNode collectionNode =
+ sideInputToCollection.get(
+
SideInputId.newBuilder().setTransformId(transformId).setLocalName(sideInputId).build());
+ checkArgument(collectionNode != null, "No side input for %s/%s",
transformId, sideInputId);
+ return runtimeContext.getBroadcastVariable(collectionNode.getId());
+ }
+
+ private static class MultimapSideInputHandler<K, V, W extends BoundedWindow>
+ implements SideInputHandler<V, W> {
private final Multimap<SideInputKey, V> collection;
private final Coder<K> keyCoder;
+ private final Coder<V> valueCoder;
private final Coder<W> windowCoder;
- private SideInputHandler(
- Multimap<SideInputKey, V> collection, Coder<K> keyCoder, Coder<W>
windowCoder) {
+ private MultimapSideInputHandler(
+ Multimap<SideInputKey, V> collection,
+ Coder<K> keyCoder,
+ Coder<V> valueCoder,
+ Coder<W> windowCoder) {
this.collection = collection;
this.keyCoder = keyCoder;
+ this.valueCoder = valueCoder;
this.windowCoder = windowCoder;
}
@Override
- public Iterable<V> get(K key, W window) {
+ public Iterable<V> get(byte[] keyBytes, W window) {
+ K key;
+ try {
+ // TODO: We could skip decoding and just compare encoded values for
deterministic keyCoders.
+ key = keyCoder.decode(new ByteArrayInputStream(keyBytes));
Review comment:
Here too, could use `CoderUtils`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 116936)
Time Spent: 2.5h (was: 2h 20m)
> Flink batch state request handler
> ---------------------------------
>
> Key: BEAM-4285
> URL: https://issues.apache.org/jira/browse/BEAM-4285
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
> Reporter: Ben Sidhom
> Assignee: Ben Sidhom
> Priority: Major
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> In order to support side inputs Flink needs a state service request handler.
> As in the non-portable we can start by handling batch side inputs by Flink
> broadcast variables.
> [https://github.com/bsidhom/beam/blob/41de3bce60f1ebc9211f299612a20d8e561f9b6f/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkBatchStateRequestHandler.java]
> or
> [https://github.com/bsidhom/beam/blob/41de3bce60f1ebc9211f299612a20d8e561f9b6f/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkBatchStateRequestHandler.java]
> can be used as a starting point.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
