kosiew opened a new pull request, #17021: URL: https://github.com/apache/datafusion/pull/17021
## Which issue does this PR close? * Closes #9246 ## Rationale for this change Understanding and optimizing memory usage is critical for query performance and stability, especially for large or complex queries. This change introduces an experimental but powerful feature to enable fine-grained, on-demand memory profiling in the DataFusion CLI, helping users better analyze and debug memory-intensive workloads. ## What changes are included in this PR? * Added `\memory_profiling` command in the CLI with `enable`, `disable`, and `show` subcommands. * Integrated `MemoryProfilingHandle` and enhanced memory tracking into `SessionContext`. * Implemented `EnhancedMemoryReport` for detailed categorized memory usage analysis. * CLI and SQL command execution pipelines now toggle memory tracking based on user input. * Added test coverage, including snapshot tests and integration-level memory profiling tests. * Introduced a memory profiling example in `datafusion-examples` demonstrating real-world usage. * Updated documentation and user guide to reflect the new command and usage. * Added new `MemoryProfilingMode` config option in `SessionConfig`. ## Are these changes tested? Yes, the following test coverage is provided: * Unit tests for `EnhancedMemoryReport` and `MemoryProfilingHandle` * Integration tests validating correct toggling and memory capture * CLI snapshot tests verifying `\memory_profiling enable/show` output * Example usage in `examples/memory_profiling.rs` ## Are there any user-facing changes? Yes: * New CLI command: `\memory_profiling [enable|disable|show]` * Memory usage report with categorized operators and summary printed to stdout * Documentation and user guide updated These changes are additive, opt-in, and backward compatible. The memory profiling remains disabled by default and must be explicitly enabled per query. --- *NOTE:* The memory profiling feature is currently experimental and should be used for analysis and debugging purposes. It may not yet capture all memory allocations. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
