maxburke commented on PR #23032: URL: https://github.com/apache/datafusion/pull/23032#issuecomment-4929186961
> Have you measured peak memory with sliced or partially-full build batches? Do the build sides you see in practice tend to be compact and fully packed? I believe they are fully packed? >Do you have a benchmark for a build side split across many small batches (say a few thousand), versus the same data as one batch? Unfortunately I do not. >A dictionary-encoded or struct/list-heavy build side seems like it could regress more than a constant factor here. Was that exercised? No; we do not have much dictionary or struct data. We have some list data but we haven't seen any performance regressions worth noting. I'm sorry, I don't really have the depth of experience with the internals of the join machinery to really answer your questions deeply :(. I just started attacking the problem where our profiles (and panic backtraces) were pointing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
