milenkovicm commented on code in PR #2091:
URL:
https://github.com/apache/datafusion-ballista/pull/2091#discussion_r3609155488
##########
ballista/core/src/execution_plans/sort_shuffle/writer.rs:
##########
@@ -350,18 +350,35 @@ impl SortShuffleWriterExec {
// Reservation drops naturally; nothing left to free.
drop(reservation);
- debug!(
- "Sort shuffle write for partition {} completed in {} seconds. \
- Output: {:?}, Index: {:?}, Spill events: {}, Spill batches:
{}, \
- Spill bytes: {}",
- input_partition,
- now.elapsed().as_secs(),
- data_path,
- index_path,
- spill_events,
- total_spilled_batches,
- total_bytes_spilled
- );
+ let elapsed_secs = now.elapsed().as_secs();
+ if total_bytes_spilled > 0 {
+ warn!(
+ "Sort shuffle spill: job={} stage={} input_partition={} \
+ spilled {} bytes in {} batches ({} events) under memory \
+ pressure; write completed in {} seconds",
+ job_id,
+ stage_id,
+ input_partition,
+ total_bytes_spilled,
+ total_spilled_batches,
+ spill_events,
+ elapsed_secs,
+ );
+ } else {
+ info!(
Review Comment:
can info be debug in this case?
it might be important at this moment but in day-to-day operations probably
would be just noise? wdyt ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
