+1
2026年10月17日(土) 0:47 Daniel Tenedorio <[email protected]>: > +1 from me on this (non-binding). This functionality is somewhat standard > across relational database management systems in the industry, and has > worked well in my experience when paired with data sources exposing index > key point or range lookups. > > On 2026-10-07 08:30 UTC Jungtaek Lim wrote: > > Hi devs, > > > > I'm coming up with a proposal for a new join strategy, named "External > > Lookup Join". > > > > tl;dr: > > All existing join strategies require the static side (DataFrame/Relation) > > to be materialized before the join operator. This doesn't allow us to > > achieve low latency join where the storage of the static side is expected > > to produce low latency response over on-demand request. (e.g. OLTP > > optimized like RDBMS, in-memory storage, NoSQL, etc.) The new join > strategy > > will issue the query to the external storage "on demand", when processing > > the data on the left side. > > > > The most benefit will come with real-time mode, but it is not limited to > > real-time mode. This join strategy would be useful as long as the > external > > storage of the static side is better suited for on-demand requests > (rather > > than bulk load with pruning). Both static-static and stream-static can be > > benefited by this, though static-static join needs to be careful on doing > > this. > > > > More details are included in the SPIP doc. > > > > Issue: https://issues.apache.org/jira/browse/SPARK-60031 > > SPIP doc: > > > https://docs.google.com/document/d/1fBPM5cku8UJXacG2fw9df1zqALJ-WFm8E9-b9oiFP5M/edit?tab=t.0 > > > > Looking forward to hearing from you. > > > > Thanks! > > Jungtaek Lim (HeartSaVioR) > > > > --------------------------------------------------------------------- > To unsubscribe e-mail: [email protected] > >
