JingsongLi commented on code in PR #9291:
URL: https://github.com/apache/paimon/pull/9291#discussion_r3818030708
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/service/QueryFileMonitor.java:
##########
@@ -94,12 +96,25 @@ public void start() {
this.read = readBuilder.newRead();
}
+ @Override
+ public CompletableFuture<Void> isAvailable() {
+ return availableFuture;
+ }
+
@Override
public InputStatus pollNext(ReaderOutput<InternalRow> readerOutput)
throws Exception {
boolean isEmpty = doScan(readerOutput);
if (isEmpty) {
- Thread.sleep(monitorInterval);
+ availableFuture =
+ CompletableFuture.runAsync(
Review Comment:
[P2] Do not block the shared common pool for the discovery interval
`CompletableFuture.runAsync` uses `ForkJoinPool.commonPool()`, and this task
occupies one of its workers in `Thread.sleep` for the full interval. When a
TaskManager hosts more idle `QueryFileMonitor` readers than the common-pool
parallelism, the remaining futures stay queued and do not even start their
delay: later readers can wake after roughly 2×, 3×, ... the configured
discovery interval, while unrelated common-pool work is starved as well.
`close()` also leaves a long sleep running. Please use a scheduled/timer
executor that does not block a worker, own/cancel it with the reader lifecycle,
and cover multiple concurrent waits with a controllable scheduler.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
