github-actions[bot] commented on code in PR #41448:
URL: https://github.com/apache/beam/pull/41448#discussion_r4615413844


##########
runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperatorTest.java:
##########
@@ -2324,6 +2325,138 @@ public void processElement(ProcessContext context) {
     testHarness.close();
   }
 
+  /**
+   * Ensures that the output watermark does not pass the output timestamp of 
an event-time timer
+   * which has fired but is still buffered because of {@code 
@RequiresStableInput}.
+   */

Review Comment:
   **🟢 Nit**: `stateId` and `inputCoder` (on line 2335) are unused in this test 
method and may trigger ErrorProne `UnusedVariable` warnings during compilation.



##########
runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperatorTest.java:
##########
@@ -2324,6 +2325,138 @@ public void processElement(ProcessContext context) {
     testHarness.close();
   }
 
+  /**
+   * Ensures that the output watermark does not pass the output timestamp of 
an event-time timer
+   * which has fired but is still buffered because of {@code 
@RequiresStableInput}.
+   */
+  @Test
+  public void testExactlyOnceBufferingOutputWatermarkHoldsForBufferedTimers() 
throws Exception {
+    FlinkPipelineOptions options = FlinkPipelineOptions.defaults();
+    options.setStreaming(true);
+    options.setMaxBundleSize(1L);
+    options.setCheckpointingInterval(1L);
+    options.setCheckpointingMode("EXACTLY_ONCE");
+    options.setNumConcurrentCheckpoints(1);
+
+    TupleTag<KV<String, String>> outputTag = new TupleTag<>("main-output");
+
+    StringUtf8Coder keyCoder = StringUtf8Coder.of();
+    KvToFlinkKeyKeySelector<String, String> keySelector = new 
KvToFlinkKeyKeySelector<>(keyCoder);
+    KvCoder<String, String> kvCoder = KvCoder.of(keyCoder, 
StringUtf8Coder.of());
+    WindowedValues.FullWindowedValueCoder<KV<String, String>> 
windowedValueCoder =
+        WindowedValues.getFullCoder(kvCoder, GlobalWindow.Coder.INSTANCE);
+
+    final String timerId = "timer";
+    DoFn<KV<String, String>, KV<String, String>> doFn =
+        new DoFn<KV<String, String>, KV<String, String>>() {
+          @TimerId(timerId)
+          private final TimerSpec timerSpec = 
TimerSpecs.timer(TimeDomain.EVENT_TIME);
+
+          @ProcessElement
+          // Use RequiresStableInput to force buffering elements and fired 
timers
+          @RequiresStableInput
+          public void processElement(@TimerId(timerId) Timer timer) {
+            timer.set(new Instant(50));
+          }
+
+          @OnTimer(timerId)
+          public void onTimer(OnTimerContext context) {
+            context.output(KV.of("key", "timer"));
+          }
+        };
+
+    DoFnOperator.MultiOutputOutputManagerFactory<KV<String, String>> 
outputManagerFactory =
+        new DoFnOperator.MultiOutputOutputManagerFactory<>(
+            outputTag,
+            WindowedValues.getFullCoder(kvCoder, GlobalWindow.Coder.INSTANCE),
+            new SerializablePipelineOptions(options));
+
+    DoFnOperator<KV<String, String>, KV<String, String>, KV<String, String>> 
doFnOperator =
+        new DoFnOperator<>(
+            doFn,
+            "stepName",
+            windowedValueCoder,
+            Collections.emptyMap(),
+            outputTag,

Review Comment:
   **🔵 Info**: In this test, `doFn2` is manually wrapped in a 
`KeyedOneInputStreamOperatorTestHarness` with `(KeySelector) new 
WorkItemKeySelector<>(StringUtf8Coder.of())`. Note that when 
`BufferingDoFnRunner.processElement(WindowedValue<InputT> elem)` runs (in 
`BufferingDoFnRunner.java:249`), it calls `keySelector.apply(elem.getValue())` 
(passing the inner `KV<String, String>` rather than the `WindowedValue`), which 
only works here because `keySelector` is raw-cast and `doFn2` is non-portable. 
In portable Flink pipelines (like the Python `ParDoTest` in #24662), a stateful 
stage fused with `@RequiresStableInput` uses `ExecutableStageDoFnOperator` with 
`KvToFlinkKeyKeySelector`, where `keySelector` expects `WindowedValue<KV<...>>` 
and returns `FlinkKey` (while `KeyedBufferingElementsHandler` stores raw keys 
via `keyCoder`). If the goal is to close #24662, we may also want to test/check 
`ExecutableStageDoFnOperator` with `stagePayloadWithUserState` + 
`stagePayloadWithStableInput` an
 d `KvToFlinkKeyKeySelector`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to