anshul98ks123 opened a new pull request, #19863:
URL: https://github.com/apache/pinot/pull/19863
**Problem.** A caller that filters a large prefix by pattern, such as batch
ingestion picking 7 dates out of 500 date
folders, has to page through every key under the prefix, because `PinotFS`
can only start a listing at the first
key. At S3's 1,000 keys per page, 400k keys take 400 sequential requests
even when the caller knows most of them
cannot match.
## Root cause
S3 `ListObjectsV2` returns keys in UTF-8 order and accepts `StartAfter` with
any key, existing or not. `PinotFS`
exposes only prefix plus continuation, so callers cannot use it.
## Approaches
- Filter after a full listing (today): correct, but reads every key.
- Call the S3 SDK from the caller: bypasses `PinotFS` credentials, retries
and configuration.
- Server-side glob (GCS `matchGlob`): one store, one syntax.
- **Chosen: two default methods on `PinotFS`**, `supportsListingAfter()`
(false) and
`listFilesWithMetadataAfter(uri, startAfter, maxResults)`, implemented for
S3 with `StartAfter`. A caller can
jump over a key range in one request.
## Before → after
```mermaid
flowchart LR
subgraph Before
direction TB
a1["caller wants d=0100..0106<br/>of 503 folders"] -->
a2["listFilesWithMetadata(prefix)"] --> a3["pages 1..403<br/>via continuation
token"]
end
subgraph After
direction TB
b1["same caller"] --> b2["listFilesWithMetadataAfter(prefix,<br/>key
just before d=0100, 1000)"] --> b3["S3 StartAfter:<br/>first request lands on
d=0100"]
end
```
## Scope
- `PinotFS`: `supportsListingAfter()` and `listFilesWithMetadataAfter(...)`,
both defaults, so other file
systems are unaffected.
- `S3PinotFS`: `StartAfter` on the first request, continuation after it;
directory markers skipped; page size
capped at 1,000; fewer than `maxResults` only when the listing is
exhausted.
- Not included: GCS (`startOffset` exists, but `GcsPinotFS` paths are
URI-escaped, so path order differs from
listing order), ADLS and HDFS (no start key).
## Related PRs
- First consumer: an automaton-driven skip-scan of batch-ingestion listings
in a downstream distribution. No
consumer in this repository yet.
## Testing
- `S3PinotFSListingAfterTest` (8, mocked client): request shape (prefix,
`StartAfter`, no delimiter, page size),
continuation replaces `StartAfter`, markers skipped while paging,
`maxResults`, a path in another bucket rejected.
- `S3PinotFSTest#testListFilesWithMetadataAfter` (S3Mock, `s3` and `s3a`):
seek past a key that does not exist,
resume after a returned path, start at the first key, start past the end.
- Locally: `S3PinotFSListingAfterTest` and `S3PinotFSPaginatedListTest`
pass; checkstyle, spotless and license
pass. The S3Mock test needs Docker, so CI runs it.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
