pnowojski commented on a change in pull request #10435: [FLINK-13955][runtime]
migrate ContinuousFileReaderOperator to the mailbox execution model
URL: https://github.com/apache/flink/pull/10435#discussion_r372930730
##########
File path:
flink-fs-tests/src/test/java/org/apache/flink/hdfstests/ContinuousFileProcessingTest.java
##########
@@ -217,10 +222,11 @@ public void testFileReadingOperatorWithIngestionTime()
throws Exception {
tester.setProcessingTime(nextTimestamp);
// send the next split to be read and wait until it is
fully read, the +1 is for the watermark.
- tester.processElement(new StreamRecord<>(
- new
TimestampedFileInputSplit(modTimes.get(split.getPath().getName()),
- split.getSplitNumber(),
split.getPath(), split.getStart(),
- split.getLength(),
split.getHostnames())));
+ RunnableWithException runnableWithException = () ->
tester.processElement(new StreamRecord<>(
+ new
TimestampedFileInputSplit(modTimes.get(split.getPath().getName()),
+ split.getSplitNumber(),
split.getPath(), split.getStart(),
+ split.getLength(),
split.getHostnames())));
+ tester.enqueueMail(runnableWithException);
Review comment:
Something is strange with this test. The usual pattern I would expect is to
instead of running event loops in separate thread, and delegating actions like
`processElement` to that thread, to make the `main` thread, a thread that
executes and controls the event loop in steps.
```
tester.open();
tester. setProcessingTime(201);
// execute all pending actions
tester.runActions()/triggerAll()/waitForMails()/...;
output = tester.getOutput();
assertTrue(output.toString(), output.peek() instanceof Watermark);
(...)
for (split : splits) {
tester.processElement(...);
}
```
Otherwise we are missing one of the biggest advantage of the single-threaded
execution model: easy to test and to reproduce any desired interleave/order of
of the actions.
For example like in `ManuallyTriggeredScheduledExecutor#triggerAll` from our
code base or `io.netty.channel.embedded.EmbeddedChannel#runPendingTasks` from
netty (`PartitionRequestQueueTest`) - both examples of testing code with async
& single threaded execution models.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
