sejal-gupta-ksolves commented on code in PR #16768:
URL: https://github.com/apache/iceberg/pull/16768#discussion_r3462721742
##########
core/src/main/java/org/apache/iceberg/rest/ScanTaskIterable.java:
##########
@@ -103,11 +104,25 @@ private void submitFixedWorkers() {
@Override
public CloseableIterator<FileScanTask> iterator() {
- return new ScanTasksIterator();
+ ScanTasksIterator iter = new ScanTasksIterator();
+ addCloseable(iter);
+ return iter;
}
@Override
- public void close() throws IOException {}
+ public void close() throws IOException {
+ if (shutdown.compareAndSet(false, true)) {
+ LOG.info(
+ "ScanTaskIterable is closing. Clearing {} queued tasks, {} plan
tasks, and {} initial file scan tasks.",
+ taskQueue.size(),
+ planTasks.size(),
+ initialFileScanTasks.size());
+ super.close();
+ taskQueue.clear();
+ planTasks.clear();
+ initialFileScanTasks.clear();
Review Comment:
Thank you for the extra context regarding StarRocks calling iterator.close()
directly! That makes perfect sense.
I have updated the implementation to move the shutdown signal and
queue-clearing logic directly into ScanTasksIterator.close(). This ensures that
everything is cleaned up properly whether the engine invokes close on the
iterator directly or relies on CloseableGroup cascading it down from the
iterable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
