Hi Lincoln, Delayed Dim Join is a frequently requested feature, it's exciting to see this feature is on the road. The FLIP looks good to me in general. I only left some minor comments.
1) support retry for sync lookup I'm also fine with the idea proposed by Jingsong. But this doesn't conflict with the FLIP and can be future work. It would be great if we can determine the APIs first. 1) "allow-unordered" => "unordered" I would prefer the "unordered" output mode rather than "allow-unordered". Because this fully aligns with the DataStream behaviors and avoids confusion on the differences. I understand the purpose that adding a "allow" prefix here, but I think the semantic is fine to just use "unordered" here. We didn't see any users confused about OutputMode#UNORDERED. Best, Jark On Fri, 27 May 2022 at 12:58, Jingsong Li <jingsongl...@gmail.com> wrote: > Thanks Lincoln for your proposal. > > Take a look at `strategy: fixed-delay delay: duration, e.g., 10s > max-attempts: integer, e.g., 3`. > > Are these options only for async? It looks like normal lookups work too? > > One thing is: most of the lookup functions seem to be synchronous now? > There are not so many asynchronous ones? > > Best, > Jingsong > > On Tue, May 24, 2022 at 11:48 AM Lincoln Lee <lincoln.8...@gmail.com> > wrote: > > > Hi all, > > > > Considering the new common table option 'lookup.max-retries' proposed in > > FLIP-221[1] which is commonly used for exception handling in connector > > implementation, we should clearly distinguish ASYNC_LOOKUP_RETRY from it > to > > avoid confusing users. > > > > To do so, the name ASYNC_LOOKUP_RETRY can change to > > ASYNC_LOOKUP_MISS_RETRY, and as the name implies, restrict it to support > > retries only for lookup misses and no longer include exceptions (for sql > > connectors, let the connector implementer decide how to handle exceptions > > since there are various kinds of retryable exceptions and can not retry > > ones). > > > > The FLIP[2] has been updated. > > > > [1] > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-221%3A+Abstraction+for+lookup+source+cache+and+metric > > [2] > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-234%3A+Support+Retryable+Lookup+Join+To+Solve+Delayed+Updates+Issue+In+External+Systems > > > > > > Best, > > Lincoln Lee > > > > > > Lincoln Lee <lincoln.8...@gmail.com> 于2022年5月19日周四 18:24写道： > > > > > Dear Flink developers, > > > > > > I would like to open a discussion on FLIP 234 [1] to support retryable > > > lookup join to solve delayed updates issue, as a pre-work for this > > > solution, we proposed FLIP-232[2] which adds a generic retry support > for > > > Async I/O. > > > We prefer to offer this retry capability via query hints, similar to > new > > > join hints proposed in FLINK-27625[3] & FLIP-204[4]. > > > > > > This feature is backwards compatible and transparently to connectors. > For > > > existing connectors which implements AsyncTableFunction, can easily > > enable > > > async retry via the new join hint. > > > > > > [1] > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-234%3A+Support+Retryable+Lookup+Join+To+Solve+Delayed+Updates+Issue+In+External+Systems > > > [2] > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211883963 > > > [3] https://lists.apache.org/thread/jm9kg33wk9z2bvo2b0g5bp3n5kfj6qv8 > > > [4] > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-204:+Introduce+Hash+Lookup+Join > > > > > > Best, > > > Lincoln Lee > > > > > >
