Folks voting here: we opened the VOTE thread! Please vote in here - https://lists.apache.org/thread.html/rc7f5f9ogggv6rqy7w5m2753yq5n5ckq
On Wed, Oct 21, 2026 at 9:50 AM Kousuke Saruta <[email protected]> wrote: > +1 > > > 2026年10月17日(土) 0:47 Daniel Tenedorio <[email protected]>: > >> +1 from me on this (non-binding). This functionality is somewhat standard >> across relational database management systems in the industry, and has >> worked well in my experience when paired with data sources exposing index >> key point or range lookups. >> >> On 2026-10-07 08:30 UTC Jungtaek Lim wrote: >> > Hi devs, >> > >> > I'm coming up with a proposal for a new join strategy, named "External >> > Lookup Join". >> > >> > tl;dr: >> > All existing join strategies require the static side >> (DataFrame/Relation) >> > to be materialized before the join operator. This doesn't allow us to >> > achieve low latency join where the storage of the static side is >> expected >> > to produce low latency response over on-demand request. (e.g. OLTP >> > optimized like RDBMS, in-memory storage, NoSQL, etc.) The new join >> strategy >> > will issue the query to the external storage "on demand", when >> processing >> > the data on the left side. >> > >> > The most benefit will come with real-time mode, but it is not limited to >> > real-time mode. This join strategy would be useful as long as the >> external >> > storage of the static side is better suited for on-demand requests >> (rather >> > than bulk load with pruning). Both static-static and stream-static can >> be >> > benefited by this, though static-static join needs to be careful on >> doing >> > this. >> > >> > More details are included in the SPIP doc. >> > >> > Issue: https://issues.apache.org/jira/browse/SPARK-60031 >> > SPIP doc: >> > >> https://docs.google.com/document/d/1fBPM5cku8UJXacG2fw9df1zqALJ-WFm8E9-b9oiFP5M/edit?tab=t.0 >> > >> > Looking forward to hearing from you. >> > >> > Thanks! >> > Jungtaek Lim (HeartSaVioR) >> > >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: [email protected] >> >>
