Doris-Breakwater commented on issue #66527:
URL: https://github.com/apache/doris/issues/66527#issuecomment-5200424969
Breakwater-GitHub-Analysis-Slot: slot_7a0c26a89db8
## Initial triage
**Assessment: high-confidence HTTP TVF compatibility bug.** The issue is
currently unlabelled. The reported failure is consistent with Doris changing a
GET-only pre-signed request into `HEAD` while discovering object size, before
the actual scan begins.
### Verified from the source
- The behavior is present in both the `4.1.2` tag and the inspected `master`
commit (`82646c38c0069eee4b1704eafef416f969bd4b93`).
- `HttpTableValuedFunction.generateFileStatus()` calls
`HttpUtils.getHttpFileSize()` while constructing the TVF. `getHttpFileSize()`
unconditionally executes `connection.setRequestMethod("HEAD")`. Its exception
text exactly matches the error in this issue, so this failure occurs in FE
analysis/planning, not in query execution.
- The scan-side reader has a second occurrence of the same incompatibility:
`be/src/io/fs/http_file_reader.cpp::HttpFileReader::open()` also sends `HEAD`
unless the reader is already initialized. The normal range carries a file size
through `FileReaderOptions.file_size`, but `HttpFileReader::create()`/`open()`
do not consume that option to initialize the reader; the constructor only
recognizes a separate `file_size` entry in the HTTP property map. Therefore,
changing only the FE probe can still leave a later BE `HEAD` failure.
- Actual HTTP reads default to `GET`, and range detection uses a small
ranged `GET`. This is compatible with the reporter's successful GET and
isolates the mismatch to metadata discovery.
- The current HTTP TVF regression server explicitly accepts both `GET` and
`HEAD`; there is no coverage for a GET-authorized endpoint that rejects `HEAD`.
### What is inferred vs. still to verify
It is highly likely that the object store returns 403 because the signature
authorizes `GET` but not `HEAD` (or because the endpoint otherwise forbids
`HEAD`). The Doris-side method substitution is verified; the storage-side
rejection reason still needs confirmation from a direct sanitized request or
storage access log. This should not be treated as a generic credential failure
because the same URL succeeds with GET.
Please provide the following without exposing the full URL or signature
values:
1. Status and response headers from a `HEAD` request and from `GET` with
`Range: bytes=0-0` (especially `Content-Length`, `Content-Range`,
`Accept-Ranges`, and any redirect). For example, run `curl -v -I '<URL>'` and
`curl -v -r 0-0 '<URL>' -o /dev/null`, then redact the query parameter values
before posting.
2. The object-storage vendor/signing scheme, and whether `Range` was
included in the signed headers.
3. The exact Doris commit SHA for the reported `master` test and, if
available, a redacted object-storage authentication/access-log reason for the
rejected `HEAD`.
A Doris Profile is not needed for this case because execution is never
reached.
### Recommended next steps
1. Replace method-changing metadata discovery with a GET-compatible
mechanism. A small `GET` range probe is the natural candidate: for `206`, parse
the total length from `Content-Range` rather than the one-byte
`Content-Length`; also handle an empty object (`416` with `Content-Range: bytes
*/0`) and a server that ignores Range and returns `200`.
2. Cover both metadata paths. Either make the BE reader honor the
already-planned per-file size while preserving its Range/non-Range
initialization semantics, or change its independent metadata probe to the same
GET-compatible mechanism. Ensure that a successful FE fix cannot be followed by
a BE `HEAD`.
3. Add a regression endpoint that accepts GET/ranged GET but rejects HEAD,
then exercise schema inference and full query execution. Include empty files,
Range-supported and Range-ignored responses, redirects, and custom
`http.header.*` propagation.
4. If the fix is accepted on master, assess a branch-4.1 backport because
the same code is present in 4.1.2.
Current workaround: use an endpoint/proxy that permits both HEAD and GET, or
use Doris's object-storage/file TVF with credentials instead of a GET-only
pre-signed HTTP URL. `http.method`, `http.enable.range.request`, and
`http.enable.chunk.response` do not avoid the FE planning-time HEAD probe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
