alisha-1000 commented on PR #41448:
URL: https://github.com/apache/beam/pull/41448#issuecomment-7024482304

   Thanks @je-ik, I've pushed an update:
   
   - maxBundleSize is now 10, and several elements arrive in one bundle.
   - New elements arrive between checkpoints, and in the last stage the
     fired timer sits in the buffer together with data. The output
     watermark is checked after every stage.
   - Checkpoint interval is 100 ms. In this harness checkpoints are
     triggered manually via snapshot()/notifyCheckpointComplete(), so the
     interval itself doesn't change the behaviour.
   
   The test passes on flink:1.19 and flink:2.2.
   
   Your suggestion also surfaced two things I didn't expect, both
   pre-existing and outside what this test covers:
   
   1. With two keys in the same key group (the harness default
      maxParallelism = 1), replayed buffered elements run under the wrong
      key on the heap backend. k2's timer overwrote k1's, and one timer
      output was lost. With maxParallelism = 128 the same test passes.
      From reading the code, this comes from getElements() in
      KeyedBufferingElementsHandler calling setCurrentKey inside a
      flatMap, whose stream is consumed through an iterator that buffers.
   2. From reading the code, BufferedElements.Coder doesn't encode the
      key of a buffered timer, so with RocksDB the key could come back as
      null after restore. I haven't run this one yet.
   
   I kept this PR's test to a single key. I have a two-key reproducer
   for (1). Would you prefer these as separate issues, or should I look
   into fixing them here? (2) might be related to #24662 itself.
   
   Separately, I checked the bot's note on the portable path: in
   ExecutableStageDoFnOperator the buffering runner gets a lambda that
   casts the input to KV, which matches a stateful stage's input, so I
   don't think the ClassCastException it describes happens there.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to