dataroaring opened a new pull request, #60414:
URL: https://github.com/apache/doris/pull/60414
## Summary
- For S3 paths without wildcards (`*`, `?`, `[...]`), use HEAD requests
instead of ListObjectsV2 to avoid requiring `s3:ListBucket` permission
- Brace patterns like `{1..10}` are expanded to concrete file paths and
verified individually with HEAD requests
- This enables loading data from S3 when only `s3:GetObject` permission is
granted
## Motivation
S3 `ListBucket` permission is often more restricted than `GetObject` in
enterprise environments. When users specify exact file paths or deterministic
patterns like `file{1..3}.csv`, listing is unnecessary since the file names can
be determined from the input.
## Changes
| File | Description |
|------|-------------|
| `S3Util.java` | Added `isDeterministicPattern()` to detect paths without
wildcards, and `expandBracePatterns()` to expand brace patterns to concrete
paths |
| `S3ObjStorage.java` | Modified `globListInternal()` to use HEAD requests
for deterministic paths |
| `S3UtilTest.java` | Added unit tests for new utility methods |
## Examples
| Path | Deterministic? | Behavior |
|------|----------------|----------|
| `s3://bucket/data/file.csv` | ✅ Yes | Single HEAD request |
| `s3://bucket/data/file{1..3}.csv` | ✅ Yes | 3 HEAD requests |
| `s3://bucket/data/*.csv` | ❌ No | Falls back to LIST |
## Test Plan
- [x] Added unit tests for `isDeterministicPattern()`
- [x] Added unit tests for `expandBracePatterns()`
- [ ] Manual testing with S3 TVF and Broker Load
🤖 Generated with [Claude Code](https://claude.ai/code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
