andygrove opened a new pull request, #2091:
URL: https://github.com/apache/datafusion-ballista/pull/2091
# Which issue does this PR close?
Part of the observability work tracked in #2081. This extracts the sort
shuffle write/spill logging from the WIP PR #2084 into a small, standalone
change.
# Rationale for this change
Sort shuffle write activity was only logged at `debug!`, so at the default
log level there was no visibility into how long shuffle writes took or
whether a task spilled to disk under memory pressure. Spilling is a key
signal when diagnosing slow or memory-constrained TPC-H stages, and it was
effectively invisible.
# What changes are included in this PR?
- Promote the per-partition shuffle write completion log from `debug!` to
`info!`, and include the output row count in the message.
- When a task spilled to disk, emit a `warn!` instead that names the job,
stage, and input partition and reports the bytes, batches, and events
spilled under memory pressure.
# Are there any user-facing changes?
Executor logs now report sort shuffle write completion at `INFO` and spill
activity at `WARN`. There are no API or configuration changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
