2010YOUY01 commented on issue #27109:
URL: https://github.com/apache/datafusion/issues/27109#issuecomment-6646660958
To double check I get the motivation correct -- is this issue trying to
optimize the below plan shape, and avoid the expensive probe to operate at the
default `batch_size` config
```
HJ(limit=3)
Scan(cheap) // build
HJ // probe
Scan(cheap) //build
Scan(expensive) // probe
```
I think we can do `HashJoinExec::with_batch_size()`, it uses local
`batch_size` to override global setting, the change in core would be trivial.
Though the API is less intuitive, the `batch_size` only control the
intermediate batch size when probing, this should be enough to solve the issue.
The concern about `HashJoinExec::with_coalesce_batches(bool)` is it requires
extra logic inside HJ, now HJ is already overly complex, so I'm trying to avoid
extra complexity when possible, while `with_batch_size` requires no inner
change of HJ.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
