fatmanverse opened a new pull request, #18421:
URL: https://github.com/apache/dolphinscheduler/pull/18421
## Was this PR generated or assisted by AI?
YES. The implementation was assisted by AI (Claude). All code was reviewed,
rewritten for production quality, and verified by the author.
## Purpose of the pull request
Closes #18420.
Task/workflow instance logs are currently kept forever on the master/worker
local disk, which grows without bound. This PR adds optional,
operator-controlled archiving of old task instance logs, plus transparent
online viewing of the archived logs.
It delivers the three capabilities requested in the issue:
1. **Toggle whether instance logs are archived** —
`task-log-archive.enabled` (default `false`, so upgraded clusters keep current
behavior).
2. **Choose how old a log must be before archiving** —
`task-log-archive.archive-threshold` (default `30d`); optional expiry via
`task-log-archive.expire-enabled` + `expire-threshold`.
3. **View archived logs online** — the existing log-view path transparently
falls back to reading from the archive zip; no UI change required.
"Archiving" means compressing a day partition into a zip package.
## Brief change log
- `dolphinscheduler-common`: new `common.log.archive` package
- `TaskLogArchiveConfig` — `@ConfigurationProperties(prefix =
"task-log-archive")`, all switches default off.
- `TaskLogArchivePathHelper` — pure path arithmetic mapping
`{base}/{yyyyMMdd}/...` day partitions to `{base}/archive/{yyyyMMdd}.zip`.
- `TaskLogArchiver` — idempotent daemon that packs partitions older than
the threshold into per-day zips and optionally deletes expired zips; implements
`AutoCloseable`.
- `ArchivedLogLocation` — value object for (zip, entry) resolution.
- `CompressionUtils` — JDK `java.util.zip` based zip / single-entry random
read (zip chosen over tar.gz so a single archived log reads cheaply for online
viewing).
- `LogUtils` — read path falls back to the archive zip when the original
log file is absent.
- `MasterServer` / `WorkerServer`: start/stop the archiver following the
existing `AutoCloseable` try-with-resources lifecycle pattern (each node
archives the logs it physically stored; not leader-gated).
- `application.yaml` (master / worker / standalone): documented
`task-log-archive` config block.
- Docs: added the config keys to `configuration.md` (EN + ZH).
## Verify this pull request
This change added tests and can be verified as follows:
- Added `CompressionUtilsTest` (6), `TaskLogArchivePathHelperTest` (8) and
`TaskLogArchiverTest` (3) — 17 tests covering compression/entry read, path
mapping, archive+expire selection, and the archiver `runOnce` flow.
- `./mvnw -pl dolphinscheduler-common test` passes (17/17).
- `dolphinscheduler-master` and `dolphinscheduler-worker` compile with the
archiver wiring.
- `./mvnw spotless:check` passes.
## Pull Request Notice
[Pull Request
Notice](https://github.com/apache/dolphinscheduler/blob/dev/docs/docs/en/contribute/join/pull-request.md)
This change is backward compatible: archiving is disabled by default and no
existing behavior changes until an operator opts in. No incompatible change to
record in `incompatible.md`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
