Hi Dongjoon, Just a clarification of the second one: I don't think that's needed at this point, given the first decision. External lookup is triggered only when 1) the data source only supports external lookup or 2) users explicitly state the join hint. The fallback case only applies to 2), and I think we generally do not do fallback when users state their intention (and IMO it's the right behavior). Instead, we should help users to guide how to fix their query to work with external lookup, since that's their intention.
I'm open for discussion if we are not reaching a consensus with this; since that's a detailed design rather than a go/no-go decision for the project, I'd still consider the vote to +1. On Wed, Oct 21, 2026 at 2:01 AM Dongjoon Hyun <[email protected]> wrote: > +1 (binding) > > Thank you for driving this, Jungtaek. On-demand lookups fill a real gap > for Real-Time Mode, and the DSv2 sketch and the stream-static scope address > my main concerns from the DISCUSS thread. > > I'd appreciate it if the doc could be updated to match what we agreed in > the thread, so that it stays the reference for the implementation: > > - For sources supporting both `BATCH_READ` and `EXTERNAL_LOOKUP`, the > default stays the existing join strategies (Oct 15). Q4, Q6, and Q8 still > say that Spark chooses the lookup join automatically, and the sentence > about the default behavior in Appendix A is cut off. > - Stream-static joins fall back to `BATCH_READ` instead of failing (Oct 8). > - The cache is tied to the task lifecycle, and the hint options override > the SQL configs (Oct 8 and Oct 16). > > Dongjoon. > > > On 2026/10/20 12:51:57 Jungtaek Lim wrote: > > Starting from my +1 (non-binding) > > > > On Tue, Oct 20, 2026 at 12:04 PM Jungtaek Lim < > [email protected]> > > wrote: > > > > > Hi devs, > > > > > > I would live to start a vote on introducing external lookup join, as > > > proposed. > > > > > > JIRA ticket: https://issues.apache.org/jira/browse/SPARK-60031 > > > SPIP doc: > > > > https://docs.google.com/document/d/1fBPM5cku8UJXacG2fw9df1zqALJ-WFm8E9-b9oiFP5M/edit?tab=t.0 > > > Discussion thread: > > > https://lists.apache.org/thread.html/7s6tgy9cm18jnzykr8htrtrw4nxyrhp1 > > > > > > The vote will be open for at least 72 hours, and passes if a majority > +1 > > > PMC > > > votes are cast, with a minimum of 3 +1 votes. > > > > > > Please vote: > > > [ ] +1: Accept the proposal as an official SPIP > > > [ ] +0 > > > [ ] -1: I don't think this is a good idea because ... > > > > > > Thanks, > > > Jungtaek Lim (HeartSaVioR) > > > > > > > --------------------------------------------------------------------- > To unsubscribe e-mail: [email protected] > >
