mrhhsg opened a new pull request, #69610: URL: https://github.com/apache/doris/pull/69610
### What problem does this PR solve? Backport cloud-mode query spill to object storage from #68032 to `branch-4.2`. With `spill_storage_type=s3`, query spill uses an S3 storage vault; `local` remains the default. Backends report the bytes currently retained in remote spill, and the FE includes them in warehouse-wide `SHOW DATA` results. Remote spill read/write traffic is also exposed in query statistics and audit records. ### Branch-specific changes - Adapt the spill store, file reader/writer and cleanup paths to the 4.2 object-storage and spill interfaces. Preserve the branch's existing external-spill session behavior and local-spill limits. - Add vault selection, upload back-pressure and coalesced remote reads. Abandoned spill objects and incomplete multipart uploads after a BE crash still require an object-store lifecycle policy. - Extend BE disk reports and query-statistics Thrift fields. Keep the existing 3-column warehouse and 6-column all-database `SHOW DATA` layouts, adding a cloud-only `__remote_spill__` row to warehouse-wide results and totals. - Carry the set of participating backends through forwarded statements so audit records can include final remote-spill traffic, with a bounded wait when backend reports are late. - Add or adapt BE/FE unit tests and `SHOW DATA`/audit regression coverage for the 4.2 APIs and output layouts. ### Local validation - FE build/package, BE build and BE Java extensions build passed. - 82 focused FE unit tests and 121 focused BE unit tests passed. - `test_show_data_warehouse` and `test_audit_log_behavior` each passed on an isolated local, non-cloud cluster. - C++ formatting and committed-diff whitespace checks passed. - A live cloud/S3 end-to-end test was not run locally; S3 behavior is covered by mock-based BE unit tests. ### Release note Cloud-mode query spill can use an S3 storage vault, with current remote spill usage visible in warehouse-wide `SHOW DATA` and remote spill traffic recorded in query statistics and audit logs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
