Final bump to see whether I can gather more input. It's been a week since the DISCUSS thread was open and I don't think there was a major concern I missed to answer.
If there is no major concern by the end of this week, I'll raise a VOTE early next week. Thanks! On Wed, Oct 7, 2026 at 5:30 PM Jungtaek Lim <[email protected]> wrote: > Hi devs, > > I'm coming up with a proposal for a new join strategy, named "External > Lookup Join". > > tl;dr: > All existing join strategies require the static side (DataFrame/Relation) > to be materialized before the join operator. This doesn't allow us to > achieve low latency join where the storage of the static side is expected > to produce low latency response over on-demand request. (e.g. OLTP > optimized like RDBMS, in-memory storage, NoSQL, etc.) The new join strategy > will issue the query to the external storage "on demand", when processing > the data on the left side. > > The most benefit will come with real-time mode, but it is not limited to > real-time mode. This join strategy would be useful as long as the external > storage of the static side is better suited for on-demand requests (rather > than bulk load with pruning). Both static-static and stream-static can be > benefited by this, though static-static join needs to be careful on doing > this. > > More details are included in the SPIP doc. > > Issue: https://issues.apache.org/jira/browse/SPARK-60031 > SPIP doc: > https://docs.google.com/document/d/1fBPM5cku8UJXacG2fw9df1zqALJ-WFm8E9-b9oiFP5M/edit?tab=t.0 > > Looking forward to hearing from you. > > Thanks! > Jungtaek Lim (HeartSaVioR) >
