wiedld opened a new pull request, #17943:
URL: https://github.com/apache/datafusion/pull/17943
## Which issue does this PR close?
Drafted/proposed solution for #16904
## Rationale for this change
Various changes to make the OOM error messages more readable.
If we agree with the basic approach, then I'll breakup this draft into
smaller PRs for code review.
## What changes are included in this PR?
General changes, not having to do with the OOM error stack:
* No more magic numbers. Have each number be labeled for what it is:
* Have OOM integration tests assert the whole (normalized) error message:
Changes for the OOM consumer stack:
* add lineage information to each MemoryConsumer (such that we can later on
build traces):
* new `ReportedConsumer` which represents a snapshot:
* reduce lock holding, such that we can use this snapshot in other ways
too (maybe realtime tracking?)
* new `ConsumerStackTrace`
Example usage:
* use the consumer parent/child relationship in ParquetWriter:
* see the changes in the OOM error messages, when we enable for
`TrackConsumersPool::report_top`:
## Are these changes tested?
Yes.
## Are there any user-facing changes?
Only nicer error messages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
