Gabriel39 commented on code in PR #66348:
URL: https://github.com/apache/doris/pull/66348#discussion_r3698475354
##########
be/src/exec/operator/spill_iceberg_table_sink_operator.cpp:
##########
@@ -55,47 +55,61 @@ size_t
SpillIcebergTableSinkLocalState::get_reserve_mem_size(RuntimeState* state
if (!_writer) {
return 0;
}
- auto current_writer = _writer->current_writer();
- auto* sort_writer =
dynamic_cast<VIcebergSortWriter*>(current_writer.get());
- if (!sort_writer) {
- return 0;
+ std::vector<IcebergSorterReserveMemory> per_partition_reservations;
+ auto active_writers = _writer->active_writers();
+ per_partition_reservations.reserve(active_writers->size());
+ for (const auto& writer : *active_writers) {
+ if (auto* sort_writer =
dynamic_cast<VIcebergSortWriter*>(writer.get())) {
+ auto reservation =
sort_writer->get_reserve_mem_size_components(state, eos);
+ per_partition_reservations.push_back(
+ {.retained_growth = reservation.retained_growth,
+ .transient_workspace = reservation.transient_workspace});
+ }
}
-
- return sort_writer->get_reserve_mem_size(state, eos);
+ // Column growth remains in every touched sorter, while sorting workspace
is reused by serial dispatch.
+ return iceberg_reserve_size(per_partition_reservations,
Review Comment:
Fixed by transferring the remaining pipeline reservation in a move-only
token with every queued block, including the EOS item. The writer thread adopts
it before write, retains the final reservation through finish, and either
consumes, shrinks, or rolls it back when a queued item is discarded.
TransfersReservationBetweenAsyncTasks covers producer-to-consumer handoff and
abandoned-token cleanup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
