+1 from me on this (non-binding). This functionality is somewhat standard across relational database management systems in the industry, and has worked well in my experience when paired with data sources exposing index key point or range lookups.
On 2026-10-07 08:30 UTC Jungtaek Lim wrote: > Hi devs, > > I'm coming up with a proposal for a new join strategy, named "External > Lookup Join". > > tl;dr: > All existing join strategies require the static side (DataFrame/Relation) > to be materialized before the join operator. This doesn't allow us to > achieve low latency join where the storage of the static side is expected > to produce low latency response over on-demand request. (e.g. OLTP > optimized like RDBMS, in-memory storage, NoSQL, etc.) The new join strategy > will issue the query to the external storage "on demand", when processing > the data on the left side. > > The most benefit will come with real-time mode, but it is not limited to > real-time mode. This join strategy would be useful as long as the external > storage of the static side is better suited for on-demand requests (rather > than bulk load with pruning). Both static-static and stream-static can be > benefited by this, though static-static join needs to be careful on doing > this. > > More details are included in the SPIP doc. > > Issue: https://issues.apache.org/jira/browse/SPARK-60031 > SPIP doc: > https://docs.google.com/document/d/1fBPM5cku8UJXacG2fw9df1zqALJ-WFm8E9-b9oiFP5M/edit?tab=t.0 > > Looking forward to hearing from you. > > Thanks! > Jungtaek Lim (HeartSaVioR) > --------------------------------------------------------------------- To unsubscribe e-mail: [email protected]
