mrhhsg opened a new pull request, #68032:
URL: https://github.com/apache/doris/pull/68032
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Cloud-mode BEs could only spill to local disks. This PR adds spill to the
instance's S3 storage vault, selected by `be.conf` (`spill_storage_type = local
| s3`, mutually exclusive), while keeping the existing spill operators and
part-file format. The S3 traffic generated by spill is a billing input, so it
is also reported to meta-service and shown in `SHOW DATA`.
**BE**
- Remote `SpillDataDir` bound to the storage vault
(`spill_s3_storage_vault`, default: the instance's default vault); objects live
under `spill/{cloud_unique_id}/{boot_id}/{query_id}/...`. The store becomes
ready lazily (no meta-service sync during startup); older boot generations are
deleted by the GC thread once ready.
- `SpillFileWriter`/`SpillFileReader` reuse `S3FileWriter`/`S3FileReader`.
Atomic capacity reservation (`spill_s3_storage_limit_bytes`) and a submit-time
upload budget (`spill_s3_max_inflight_upload_bytes`) that blocks writers when
too many upload buffers are in flight; `FileWriterOptions` gains an upload gate
+ done callback for this, honoured by `S3FileWriter`.
- Failed parts are drained before the budget is reconciled and multipart
uploads are aborted.
- bvars/metrics: `spill_remote_{read,write}_bytes`,
`spill_remote_{get,put}_requests` and per-second throughput/QPS; profile
counters `SpillRemote*`.
- Since-boot upload totals are reported to meta-service
(`report_spill_stats`) about once a minute and once more at shutdown after all
tasks are done, with bounded retries.
**Meta-service**
- `report_spill_stats` / `get_spill_stats` with key
`stats/{instance_id}/spill/{cloud_unique_id}`: one record per BE; a report of a
new boot folds the previous process' totals into `prior_boots_*`, so the number
of records is bounded by the number of BEs. The range is removed when the
instance is recycled.
- Recycler task `recycle_expired_spill_objects` removes spill objects older
than `spill_objects_expire_time_second` (default 7 days) as a safety net for
crashed BEs (S3/MOCK accessors).
**FE**
- `SHOW DATA PROPERTIES("entire_warehouse"="true")` gains a
`RemoteSpillWriteSize` column; the value is on the `total` row (spill is not
attributable to a database). A meta-service failure is reported instead of
showing 0.
Design notes and the local review records are kept outside the repository.
### Release note
Cloud mode: spill can be written to the S3 storage vault
(`spill_storage_type = s3` in be.conf); `SHOW DATA
PROPERTIES("entire_warehouse"="true")` shows the bytes uploaded by spill in the
new `RemoteSpillWriteSize` column.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. New be.conf options (`spill_storage_type`,
`spill_s3_storage_vault`, `spill_s3_storage_limit_bytes`,
`spill_s3_max_inflight_upload_bytes`), new meta-service config
`spill_objects_expire_time_second`, new column in `SHOW DATA ...
entire_warehouse`.
- Does this need documentation?
- [ ] No.
- [x] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
https://claude.ai/code/session_01Jrdwwwh8bZSnVwoCwykHse
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
