xmg333 opened a new pull request, #18461:
URL: https://github.com/apache/dolphinscheduler/pull/18461
## Was this PR generated or assisted by AI?
YES. Implementation, tests were drafted with assistance from Claude
(Anthropic); human report the bug, designed the solution and reviewed its code.
## Purpose of the pull request
Resolves #18459. The `maxFrameSize` guard in `TransporterDecoder` (already
on `dev`) prevents OOM by rejecting RPC bodies larger than 64 MB. But
`getTaskInstanceWholeLogFileBytes` returns the entire log as one `byte[]` in a
single RPC — so a task log larger than the frame limit **can no longer be
downloaded** (`TooLongFrameException`). This PR makes large task logs
downloadable by streaming them in 1 MB chunks; each RPC body stays small
regardless of total log size, and memory on both worker and API server stays
bounded (~1 MB).
## Brief change log
- **Common**: `readFileRange(path, offset, length)` + `MAX_LOG_CHUNK_SIZE`
(1 MB, self-defense clamp); `readPartFileContentFromLocal` stops at 64KB; cap
`getFileContentBytesFromLocal` at 64 MB; stream GCS upload.
- **Storage**: GCS/ABS use `InputStream` / `ReadChannel` / `openInputStream`
instead of buffering whole blobs.
- **TaskPlugin**: appId extraction scans only the log head (first 64 MB) —
large files are no longer skipped; cap appInfo at 10000 lines.
- **Extract-Common**: new `getTaskInstanceLogFileChunk` `@RpcMethod` + Chunk
request/response DTOs; worker reads via `readFileRange`, returns `eof` +
`fileSize`; injectable `maxLogDownloadSize` for testing.
- **API**: `download-log` returns `StreamingResponseBody`;
`LogClientDelegate.streamWholeLog` loops 1 MB chunks (worker RPC → remote
storage, no legacy fallback); `RemoteLogClient.prepareLocalLog` syncs remote
once per download; error trailer `[LOG-DOWNLOAD-ERROR]` on mid-stream failure;
`spring.mvc.async.request-timeout` 600 s.
## Verify this pull request
This change added tests and can be verified as follows:
- `dolphinscheduler-common` `LogUtilsTest` — `readFileRange` bounded ranges,
self-defense clamp, EOF, non-existent file.
- `dolphinscheduler-extract-common` `LogServiceImplTest` — chunked fetch
reproduces the full file; oversized rejection (injectable threshold).
- `dolphinscheduler-api` `LogClientDelegateTest` — multi-chunk concat,
worker→remote fallback, partial-write no-fallback (no prefix duplication),
remote synced once.
- `dolphinscheduler-api` `LoggerServiceImplTest` — error trailer on stream
failure.
- `./mvnw spotless:check` passes.
## Pull Request Notice
[Pull Request
Notice](https://github.com/apache/dolphinscheduler/blob/dev/docs/docs/en/contribute/join/pull-request.md)
**no incompatible change**: `download-log` streams instead of buffering,
but the HTTP contract (response body, `Content-Disposition`) is unchanged.
`spring.servlet.multipart` upload-size defaults are left at **1024 MB** (not
modified).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
