alamb commented on issue #17267: URL: https://github.com/apache/datafusion/issues/17267#issuecomment-5399783691
> I'd like to help with this topic, but I've noticed several related PRs have gone stale and several discussions ended without consensus. Is anyone interested in moving this forward or reviewing the code? > > What's the status of [#22043](https://github.com/apache/datafusion/pull/22043) and GraceHashJoin? Are they stale due to lack of reviewers, or because of unresolved design challenges? I think lack of reviewers But in general, externalized hash join is likely one of the most complex operators in a real database system, and I don't think we have had anyone who has the combination of: 1. Knowledge (or ability to get the knowledge) 2. System skill to implement / explain it 3. Familiarity with DataFusion (and can help get people to comment on the deisgn and review) > Since I'd rather put effort into something that will actually get reviewed, one area I think could be worked on first is benchmarks/tests for external joins. Does anyone who follows this topic closely have a suggestion on where to start — or better yet, would be willing to commit to reviewing it? I think this would be a great place to start. I think @mbutrovich and @2010YOUY01 have been interested recently. @Dandandan also seems to have gotten the join bug recently 🥳 🦜 With that data I would personally recommend writing up a proposal / design. This would include classic things like * problem being solved * Current implementation status (e.g. how large is the largest hash join today) * Proposed design * Breakdown of steps needed to get there * Alternatives considered (and maybe pointers to other systems / what they implement -- e.g. DuckDB, Spark and ClickHouse) I think if you kept the proposal reasonably short (a few pages) you would likely get quite a lot of feedback -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
