Hello, please find some notes I took at today's weekly sync meeting
The complete notes are here
https://docs.google.com/document/d/1NBpkIAuU7O9h8Br5CbFksDhX-L9TyO9wmGLPMe0Plc8
Sep 23, 2026
-
Raz – Memory tests using allocators
-
Encountering many memory issues at Flarion
-
Memory PRs are coming after encountering issues, to verify that the
issue was fixed we are using the allocator to ensure the memory pool
accounting
-
For example, there was memory in Filter that was not accounted for
in the pool
-
Matt mentioned that Comet also has been working on similar memory
issues – the accounting in the pool does not reflect the actual process
memory used.
-
Comet document:
https://datafusion.apache.org/comet/contributor-guide/memory_management.html
-
Brent Gardner had a talk about their solution in Coralogix OOMGuard:
https://www.youtube.com/watch?v=XXbwxJeg4Aw&list=PLDOlntcmVnYk&index=2
-
We had memory size test based on allocators
https://github.com/apache/datafusion/issues/22627
-
PR that did it: https://github.com/apache/datafusion/pull/22626
-
reasons why reverted:
https://github.com/apache/datafusion/issues/22723
-
** Andy will find / create a ticket to coordinate efforts on this
-
Possibly related: https://github.com/apache/datafusion/issues/22758
-
Adrian: filter pushdown / dynamic filters / optional filters:
https://claude.ai/artifact/SSz7t6hPyhFWp1MDPecVqt
