CalvinKirs commented on code in PR #65318:
URL: https://github.com/apache/doris/pull/65318#discussion_r3534938838
##########
fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/S3CompatibleFileSystem.java:
##########
@@ -555,14 +555,13 @@ private class ObjectStorageFileIterator implements
FileIterator {
@Override
public boolean hasNext() throws IOException {
- if (bufferIdx < buffer.size()) {
- return true;
- }
- if (done) {
- return false;
+ while (bufferIdx >= buffer.size()) {
+ if (done) {
+ return false;
+ }
+ fetchNextPage();
}
- fetchNextPage();
- return bufferIdx < buffer.size();
+ return true;
}
Review Comment:
If a non-AWS-faithful implementation (COS/Ceph/MinIO variants have had such
bugs) returns isTruncated=true with a null or unchanged continuation token,
fetchNextPage() will re-issue the identical list request forever —
listObjects(prefix, null) would restart from page 1 in a tight loop. Since this
PR exists precisely because a backend deviated from AWS behavior, I'd add a
cheap guard in fetchNextPage():
```
if (page.isTruncated()) {
String next = page.getContinuationToken();
if (next == null || next.equals(continuationToken)) {
throw new IOException("list did not advance for prefix " + prefix
+ ", continuation token: " + next);
}
continuationToken = next;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
