GitHub user zanmato1984 added a comment to the discussion: Acero streaming join support
Hi @severinson , I believe Acero does internally do (partially [1]) streaming join. I'm not very familiar with Java binding but you can try to switch the the sides of the tables (and subsequently, if you are doing outer join, change the join type from left to right or vice versa). [1] The right side of the join is chosen to be the "build" side, that is, this table is used to build a hash table to be later probed by the left side. Building the hash table requires full presence of the right side data thus is memory intensive. The left table, on the other hand, is processed in a streaming fashion, because every batch (a subset of the table data) can produce a corresponding result (a subset of the full result). Therefore it is much more efficient to use the small table on the right. GitHub link: https://github.com/apache/arrow/discussions/46370#discussioncomment-13094163 ---- This is an automatically sent email for user@arrow.apache.org. To unsubscribe, please send an email to: user-unsubscr...@arrow.apache.org
