johnyangk commented on a change in pull request #141: [NEMO-252] Fix
CreatViewTransform to emit windowed materialized data
URL: https://github.com/apache/incubator-nemo/pull/141#discussion_r230244319
##########
File path:
compiler/frontend/beam/src/main/java/org/apache/nemo/compiler/frontend/beam/transform/CreateViewTransform.java
##########
@@ -57,23 +63,69 @@ public void prepare(final Context context, final
OutputCollector<WindowedValue<O
}
@Override
- public void onData(final WindowedValue<I> element) {
- // TODO #216: support window in view
- final KV kv = ((WindowedValue<KV>) element).getValue();
- multiView.getDataList().add(kv.getValue());
+ public void onData(final WindowedValue<KV<?, I>> element) {
+ // The key of element is always null (beam's semantic)
+ // because view is a globally materialized data regardless of key
+ for (final BoundedWindow window : element.getWindows()) {
+ windowListMap.putIfAbsent(window, new ArrayList<>());
+ final List<I> list = windowListMap.get(window);
+ list.add(element.getValue().getValue());
+ }
+ }
+
+ @Override
+ public void onWatermark(final Watermark inputWatermark) {
+
+ // If no data, just forwards the watermark
+ if (windowListMap.size() == 0 && outputWatermark <
inputWatermark.getTimestamp()) {
+ outputWatermark = inputWatermark.getTimestamp();
+ outputCollector.emitWatermark(inputWatermark);
+ return;
+ }
+
+ final Iterator<Map.Entry<BoundedWindow, List<I>>> iterator =
windowListMap.entrySet().iterator();
+ long outputTimestamp = Long.MAX_VALUE;
Review comment:
maxOutputTimeStampOfEmittedWindows?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
