+1 (binding) Thank you for driving this, Jungtaek. On-demand lookups fill a real gap for Real-Time Mode, and the DSv2 sketch and the stream-static scope address my main concerns from the DISCUSS thread.
I'd appreciate it if the doc could be updated to match what we agreed in the thread, so that it stays the reference for the implementation: - For sources supporting both `BATCH_READ` and `EXTERNAL_LOOKUP`, the default stays the existing join strategies (Oct 15). Q4, Q6, and Q8 still say that Spark chooses the lookup join automatically, and the sentence about the default behavior in Appendix A is cut off. - Stream-static joins fall back to `BATCH_READ` instead of failing (Oct 8). - The cache is tied to the task lifecycle, and the hint options override the SQL configs (Oct 8 and Oct 16). Dongjoon. On 2026/10/20 12:51:57 Jungtaek Lim wrote: > Starting from my +1 (non-binding) > > On Tue, Oct 20, 2026 at 12:04 PM Jungtaek Lim <[email protected]> > wrote: > > > Hi devs, > > > > I would live to start a vote on introducing external lookup join, as > > proposed. > > > > JIRA ticket: https://issues.apache.org/jira/browse/SPARK-60031 > > SPIP doc: > > https://docs.google.com/document/d/1fBPM5cku8UJXacG2fw9df1zqALJ-WFm8E9-b9oiFP5M/edit?tab=t.0 > > Discussion thread: > > https://lists.apache.org/thread.html/7s6tgy9cm18jnzykr8htrtrw4nxyrhp1 > > > > The vote will be open for at least 72 hours, and passes if a majority +1 > > PMC > > votes are cast, with a minimum of 3 +1 votes. > > > > Please vote: > > [ ] +1: Accept the proposal as an official SPIP > > [ ] +0 > > [ ] -1: I don't think this is a good idea because ... > > > > Thanks, > > Jungtaek Lim (HeartSaVioR) > > > --------------------------------------------------------------------- To unsubscribe e-mail: [email protected]
