geruh commented on code in PR #14824:
URL: https://github.com/apache/iceberg/pull/14824#discussion_r2616055281
##########
core/src/main/java/org/apache/iceberg/rest/ScanTaskIterable.java:
##########
@@ -240,10 +239,14 @@ public void close() {
}
private boolean isDone() {
- return taskQueue.isEmpty()
+ // Reorder the conditions to make sure TaskQueue is empty is checked
last.
+ // It may happen that a worker is about to add a new task to the queue,
but before
+ // that happens, taskQueue.isEmpty() is checked then it completes fast
before the
+ // activeWorker is decremented. This would lead to a false negative.
+ return activeWorkers.get() == 0
&& planTasks.isEmpty()
- && activeWorkers.get() == 0
- && initialFileScanTasks.isEmpty();
+ && initialFileScanTasks.isEmpty()
+ && taskQueue.isEmpty();
Review Comment:
Yeah I also wanted to chime in here. I was also able to reproduce the issue
by creating a test that calls this logic repeatedly.
Correct me if I'm wrong but, the `ScanTaskIterable` is in a state today that
allows for "streaming", right? So we're able to start iterating files while
concurrently fetching scan tasks, with retrieving next item. Whereas, utilizing
something like a fork join pool would be closer to a "batch" process,
concurrently ran until all tasks are collected.
Let's say we keep the behavior of "streaming". One thing I noticed while
digging into the current implementation is that we are tracking a lot of the
state separately in `activeWorkers, planTasks, initialFileScanTasks,
taskQueue` and the termination logic has to check all of them.
I wonder if we could simplify this by just counting the work like:
- Set a counter to the number of initial tasks before submitting anything
- When a task discovers nested work, increment the counter before
submitting the nested tasks
- Decrement in the finally block
- When it hits 0, we're done
Might be overkill for this PR though. Just thinking out loud.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
