milenkovicm commented on PR #18192: URL: https://github.com/apache/datafusion/pull/18192#issuecomment-3447827774
> It depends on how you do the math (I did `(5.7-4)/4`) but that's a detail. lol, Im not sure you can redefine mats, there are strict rules around it 😂 > IMO "control plane" vs "data plane" can get a bit blurry, eg in the case of `InList` it's possible (and I think common) to have relatively large amounts of data in the "control plane". > > Besides: all of this is to further enable an optimization (dynamic filters) that can [make queries 25x faster](https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/). We are in fact discussing using `InList` to push down join hash tables so large `InList` expressions are an excellent example. So what I'm saying is "let's make the data plane 25x faster by at the same time making the control plane use 30% less memory, that requires some breaking API changes, we should figure out what those are". I think that's a pretty compelling story. I'm not sure why this argument is valid in this context 😕 I had sad nothing related to dynamic filters. > DataFusion makes plenty of breaking API changes, I don't even think this is that egregious of one. That's perfectly fine but let's have some kind of high bar when we do that. > Is it API changes in this part of code in general that you're opposed to, or mainly the footgun of re-using a cache / context causing collisions? I'm sure the later can be addressed in some way. Saving a 1.7MB on a executor which use 8GB does not make huge difference, that like 0.02%, yet for that you have introduced a possibility to shoot your foot and made interface more complex -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
