BTW, If there are no other more blocking issue / comments, I would like to
start a VOTE in another thread this wednesday 6.14

Thanks,
Aitozi.

Aitozi <gjying1...@gmail.com> 于2023年6月12日周一 23:34写道:

> Hi, Jing,
>     Thanks for your explanation. I get your point now.
>
> For the performance part, I think it's a good idea to run with returning a
> big table case, the memory consumption
> should be a point to be taken care about. Because in the ordered mode, the
> head element in buffer may affect the
> total memory consumption.
>
>
> Thanks,
> Aitozi.
>
>
>
> Jing Ge <j...@ververica.com.invalid> 于2023年6月12日周一 20:28写道:
>
>> Hi Aitozi,
>>
>> Which key will be used for lookup is not an issue, only one row will be
>> required for each key in order to enrich it. True, it depends on the
>> implementation whether multiple rows or single row for each key will be
>> returned. However, for the lookup & enrichment scenario, one row/key is
>> recommended, otherwise, like I mentioned previously, enrichment won't
>> work.
>>
>> I am a little bit concerned about returning a big table for each key,
>> since
>> it will take the async call longer to return and need more memory. The
>> performance tests should cover this scenario. This is not a blocking issue
>> for this FLIP.
>>
>> Best regards,
>> Jing
>>
>> On Sat, Jun 10, 2023 at 4:11 AM Aitozi <gjying1...@gmail.com> wrote:
>>
>> > Hi Jing,
>> >     I means the join key is not necessary to be the primary key or
>> unique
>> > index of the database.
>> > In this situation, we may queried out multi rows for one join key. I
>> think
>> > that's why the
>> > LookupFunction#lookup will return a collection of RowData.
>> >
>> > BTW, I think the behavior of lookup join will not affect the semantic of
>> > the async udtf.
>> > We use the Async TableFunction here and the table function can collect
>> > multiple rows.
>> >
>> > Thanks,
>> > Atiozi.
>> >
>> >
>> >
>> > Jing Ge <j...@ververica.com.invalid> 于2023年6月10日周六 00:15写道:
>> >
>> > > Hi Aitozi,
>> > >
>> > > The keyRow used in this case contains all keys[1].
>> > >
>> > > Best regards,
>> > > Jing
>> > >
>> > > [1]
>> > >
>> > >
>> >
>> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
>> > >
>> > >
>> > > On Fri, Jun 9, 2023 at 3:42 PM Aitozi <gjying1...@gmail.com> wrote:
>> > >
>> > > > Hi Jing,
>> > > >
>> > > >      The performance test is added to the FLIP.
>> > > >
>> > > >      As I know, The lookup join can return multi rows, it depends on
>> > > > whether  the join key
>> > > > is the primary key of the external database or not. The `lookup` [1]
>> > will
>> > > > return a collection of
>> > > > joined result, and each of them will be collected
>> > > >
>> > > >
>> > > > [1]:
>> > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L52
>> > > >
>> > > >
>> > > > Thanks,
>> > > > Aitozi.
>> > > >
>> > > > Jing Ge <j...@ververica.com.invalid> 于2023年6月9日周五 17:05写道:
>> > > >
>> > > > > Hi Aitozi,
>> > > > >
>> > > > > Thanks for the feedback. Looking forward to the performance tests.
>> > > > >
>> > > > > Afaik, lookup returns one row for each key [1] [2]. Conceptually,
>> the
>> > > > > lookup function is used to enrich column(s) from the dimension
>> table.
>> > > If,
>> > > > > for the given key, there will be more than one row, there will be
>> no
>> > > way
>> > > > to
>> > > > > know which row will be used to enrich the key.
>> > > > >
>> > > > > [1]
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
>> > > > > [2]
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableFunction.java#L196
>> > > > >
>> > > > > Best regards,
>> > > > > Jing
>> > > > >
>> > > > > On Fri, Jun 9, 2023 at 5:18 AM Aitozi <gjying1...@gmail.com>
>> wrote:
>> > > > >
>> > > > > > Hi Jing
>> > > > > >     Thanks for your good questions. I have updated the example
>> to
>> > the
>> > > > > FLIP.
>> > > > > >
>> > > > > > > Only one row for each lookup
>> > > > > > lookup can also return multi rows, based on the query result.
>> [1]
>> > > > > >
>> > > > > > [1]:
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L56
>> > > > > >
>> > > > > > > If we use async calls with lateral join, my gut feeling is
>> > > > > > that we might have many more async calls than lookup join. I am
>> not
>> > > > > really
>> > > > > > sure if we will be facing potential issues in this case or not.
>> > > > > >
>> > > > > > IMO, the work pattern is similar to the lookup function, for
>> each
>> > row
>> > > > > from
>> > > > > > the left table,
>> > > > > > it will evaluate the eval method once, so the async call numbers
>> > will
>> > > > not
>> > > > > > change.
>> > > > > > and the maximum calls in flight is limited by the Async
>> operators
>> > > > buffer
>> > > > > > capacity
>> > > > > > which will be controlled by the option.
>> > > > > >
>> > > > > > BTW, for the naming of these option, I updated the FLIP about
>> this
>> > > you
>> > > > > can
>> > > > > > refer to
>> > > > > > the section of "ConfigOption" and "Rejected Alternatives"
>> > > > > >
>> > > > > > In the end, for the performance evaluation, I'd like to do some
>> > tests
>> > > > and
>> > > > > > will update it to the FLIP doc
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Aitozi.
>> > > > > >
>> > > > > >
>> > > > > > Jing Ge <j...@ververica.com.invalid> 于2023年6月9日周五 07:23写道:
>> > > > > >
>> > > > > > > Hi Aitozi,
>> > > > > > >
>> > > > > > > Thanks for the clarification. The code example looks
>> > interesting. I
>> > > > > would
>> > > > > > > suggest adding them into the FLIP. The description with code
>> > > examples
>> > > > > > will
>> > > > > > > help readers understand the motivation and how to use it.
>> Afaiac,
>> > > it
>> > > > > is a
>> > > > > > > valid feature for Flink users.
>> > > > > > >
>> > > > > > > As we knew, lookup join is based on temporal join, i.e. FOR
>> > > > SYSTEM_TIME
>> > > > > > AS
>> > > > > > > OF which is also used in your code example. Temporal join
>> > performs
>> > > > the
>> > > > > > > lookup based on the processing time match. Only one row for
>> each
>> > > > > > > lookup(afaiu, I need to check the source code to double
>> confirm)
>> > > will
>> > > > > > > return for further enrichment. One the other hand, lateral
>> join
>> > > will
>> > > > > have
>> > > > > > > sub-queries correlated with every individual value of the
>> > reference
>> > > > > table
>> > > > > > > from the preceding part of the query and each sub query will
>> > return
>> > > > > > > multiple rows. If we use async calls with lateral join, my gut
>> > > > feeling
>> > > > > is
>> > > > > > > that we might have many more async calls than lookup join. I
>> am
>> > not
>> > > > > > really
>> > > > > > > sure if we will be facing potential issues in this case or
>> not.
>> > > > > Possible
>> > > > > > > issues I can think of now e.g. too many PRC calls, too many
>> async
>> > > > calls
>> > > > > > > processing, the sub query will return a table which might be
>> > (too)
>> > > > big,
>> > > > > > and
>> > > > > > > might cause performance issues. I would suggest preparing some
>> > use
>> > > > > cases
>> > > > > > > and running some performance tests to check it. These are my
>> > > concerns
>> > > > > > about
>> > > > > > > using async calls with lateral join and I'd like to share with
>> > you,
>> > > > > happy
>> > > > > > > to discuss with you and hear different opinions, hopefully the
>> > > > > > > discussion could help me understand it more deeply. Please
>> > correct
>> > > me
>> > > > > if
>> > > > > > I
>> > > > > > > am wrong.
>> > > > > > >
>> > > > > > > Best regards,
>> > > > > > > Jing
>> > > > > > >
>> > > > > > >
>> > > > > > > On Thu, Jun 8, 2023 at 7:22 AM Aitozi <gjying1...@gmail.com>
>> > > wrote:
>> > > > > > >
>> > > > > > > > Hi Mason,
>> > > > > > > >     Thanks for your input. I think if we support the user
>> > defined
>> > > > > async
>> > > > > > > > table function,
>> > > > > > > > user will be able to use it to hold a batch data then
>> handle it
>> > > at
>> > > > > one
>> > > > > > > time
>> > > > > > > > in the customized function.
>> > > > > > > >
>> > > > > > > > AsyncSink is meant for the sink operator. I have not figure
>> out
>> > > how
>> > > > > to
>> > > > > > > > integrate in this case.
>> > > > > > > >
>> > > > > > > > Thanks,
>> > > > > > > > Atiozi.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > Mason Chen <mas.chen6...@gmail.com> 于2023年6月8日周四 12:40写道:
>> > > > > > > >
>> > > > > > > > > Hi Aitozi,
>> > > > > > > > >
>> > > > > > > > > I think it makes sense to make it easier for SQL users to
>> > make
>> > > > > RPCs.
>> > > > > > Do
>> > > > > > > > you
>> > > > > > > > > think your proposal can extend to the ability to batch
>> data
>> > for
>> > > > the
>> > > > > > > RPC?
>> > > > > > > > > This is also another common strategy to increase
>> throughput.
>> > > > Also,
>> > > > > > have
>> > > > > > > > you
>> > > > > > > > > considered solving this a bit differently by leveraging
>> > Flink's
>> > > > > > > > AsyncSink?
>> > > > > > > > >
>> > > > > > > > > Best,
>> > > > > > > > > Mason
>> > > > > > > > >
>> > > > > > > > > On Mon, Jun 5, 2023 at 1:50 AM Aitozi <
>> gjying1...@gmail.com>
>> > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > One more thing for discussion:
>> > > > > > > > > >
>> > > > > > > > > > In our internal implementation, we reuse the option
>> > > > > > > > > > `table.exec.async-lookup.buffer-capacity` and
>> > > > > > > > > > `table.exec.async-lookup.timeout` to config
>> > > > > > > > > > the async udtf. Do you think we should add two extra
>> option
>> > > to
>> > > > > > > > > distinguish
>> > > > > > > > > > from the lookup option such as
>> > > > > > > > > >
>> > > > > > > > > > `table.exec.async-udtf.buffer-capacity`
>> > > > > > > > > > `table.exec.async-udtf.timeout`
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > Best,
>> > > > > > > > > > Aitozi.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > Aitozi <gjying1...@gmail.com> 于2023年6月5日周一 12:20写道:
>> > > > > > > > > >
>> > > > > > > > > > > Hi Jing,
>> > > > > > > > > > >
>> > > > > > > > > > >     > what is the difference between the RPC call or
>> > query
>> > > > you
>> > > > > > > > > mentioned
>> > > > > > > > > > > and the lookup in a very
>> > > > > > > > > > > general way
>> > > > > > > > > > >
>> > > > > > > > > > > I think the RPC call or query service is quite
>> similar to
>> > > the
>> > > > > > > lookup
>> > > > > > > > > > join.
>> > > > > > > > > > > But lookup join should work
>> > > > > > > > > > > with `LookupTableSource`.
>> > > > > > > > > > >
>> > > > > > > > > > > Let's see how we can perform an async RPC call with
>> > lookup
>> > > > > join:
>> > > > > > > > > > >
>> > > > > > > > > > > (1) Implement an AsyncTableFunction with RPC call
>> logic.
>> > > > > > > > > > > (2) Implement a `LookupTableSource` connector run with
>> > the
>> > > > > async
>> > > > > > > udtf
>> > > > > > > > > > > defined in (1).
>> > > > > > > > > > > (3) Then define a DDL of this look up table in SQL
>> > > > > > > > > > >
>> > > > > > > > > > > CREATE TEMPORARY TABLE Customers (
>> > > > > > > > > > >   id INT,
>> > > > > > > > > > >   name STRING,
>> > > > > > > > > > >   country STRING,
>> > > > > > > > > > >   zip STRING
>> > > > > > > > > > > ) WITH (
>> > > > > > > > > > >   'connector' = 'custom'
>> > > > > > > > > > > );
>> > > > > > > > > > >
>> > > > > > > > > > > (4) Run with the query as below:
>> > > > > > > > > > >
>> > > > > > > > > > > SELECT o.order_id, o.total, c.country, c.zip
>> > > > > > > > > > > FROM Orders AS o
>> > > > > > > > > > >   JOIN Customers FOR SYSTEM_TIME AS OF o.proc_time AS
>> c
>> > > > > > > > > > >     ON o.customer_id = c.id;
>> > > > > > > > > > >
>> > > > > > > > > > > This example is from doc
>> > > > > > > > > > > <
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/queries/joins/#lookup-join
>> > > > > > > > > > >.You
>> > > > > > > > > > > can image the look up process as an async RPC call
>> > process.
>> > > > > > > > > > >
>> > > > > > > > > > > Let's see how we can perform an async RPC call with
>> > lateral
>> > > > > join:
>> > > > > > > > > > >
>> > > > > > > > > > > (1) Implement an AsyncTableFunction with RPC call
>> logic.
>> > > > > > > > > > > (2) Run query with
>> > > > > > > > > > >
>> > > > > > > > > > > Create function f1 as '...' ;
>> > > > > > > > > > >
>> > > > > > > > > > > SELECT o.order_id, o.total, c.country, c.zip FROM
>> Orders
>> > > > > lateral
>> > > > > > > > table
>> > > > > > > > > > > (f1(order_id)) as T(...);
>> > > > > > > > > > >
>> > > > > > > > > > > As you can see, the lateral join version is more
>> simple
>> > and
>> > > > > > > intuitive
>> > > > > > > > > to
>> > > > > > > > > > > users. Users do not have to wrap a
>> > > > > > > > > > > LookupTableSource for the purpose of using async udtf.
>> > > > > > > > > > >
>> > > > > > > > > > > In the end, We can also see the user defined async
>> table
>> > > > > function
>> > > > > > > is
>> > > > > > > > an
>> > > > > > > > > > > enhancement of the current lateral table join
>> > > > > > > > > > > which only supports sync lateral join now.
>> > > > > > > > > > >
>> > > > > > > > > > > Best,
>> > > > > > > > > > > Aitozi.
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > Jing Ge <j...@ververica.com.invalid> 于2023年6月2日周五
>> > 19:37写道:
>> > > > > > > > > > >
>> > > > > > > > > > >> Hi Aitozi,
>> > > > > > > > > > >>
>> > > > > > > > > > >> Thanks for the update. Just out of curiosity, what is
>> > the
>> > > > > > > difference
>> > > > > > > > > > >> between the RPC call or query you mentioned and the
>> > lookup
>> > > > in
>> > > > > a
>> > > > > > > very
>> > > > > > > > > > >> general way? Since Lateral join is used in the FLIP.
>> Is
>> > > > there
>> > > > > > any
>> > > > > > > > > > special
>> > > > > > > > > > >> thought for that? Sorry for asking so many questions.
>> > The
>> > > > FLIP
>> > > > > > > > > contains
>> > > > > > > > > > >> limited information to understand the motivation.
>> > > > > > > > > > >>
>> > > > > > > > > > >> Best regards,
>> > > > > > > > > > >> Jing
>> > > > > > > > > > >>
>> > > > > > > > > > >> On Fri, Jun 2, 2023 at 3:48 AM Aitozi <
>> > > gjying1...@gmail.com
>> > > > >
>> > > > > > > wrote:
>> > > > > > > > > > >>
>> > > > > > > > > > >> > Hi Jing,
>> > > > > > > > > > >> >     I have updated the proposed changes to the
>> FLIP.
>> > > IMO,
>> > > > > > lookup
>> > > > > > > > has
>> > > > > > > > > > its
>> > > > > > > > > > >> > clear
>> > > > > > > > > > >> > async call requirement is due to its IO heavy
>> > operator.
>> > > In
>> > > > > our
>> > > > > > > > > usage,
>> > > > > > > > > > >> sql
>> > > > > > > > > > >> > users have
>> > > > > > > > > > >> > logic to do some RPC call or query the third-party
>> > > service
>> > > > > > which
>> > > > > > > > is
>> > > > > > > > > > >> also IO
>> > > > > > > > > > >> > intensive.
>> > > > > > > > > > >> > In these case, we'd like to leverage the async
>> > function
>> > > to
>> > > > > > > improve
>> > > > > > > > > the
>> > > > > > > > > > >> > throughput.
>> > > > > > > > > > >> >
>> > > > > > > > > > >> > Thanks,
>> > > > > > > > > > >> > Aitozi.
>> > > > > > > > > > >> >
>> > > > > > > > > > >> > Jing Ge <j...@ververica.com.invalid> 于2023年6月1日周四
>> > > > 22:55写道:
>> > > > > > > > > > >> >
>> > > > > > > > > > >> > > Hi Aitozi,
>> > > > > > > > > > >> > >
>> > > > > > > > > > >> > > Sorry for the late reply. Would you like to
>> update
>> > the
>> > > > > > > proposed
>> > > > > > > > > > >> changes
>> > > > > > > > > > >> > > with more details into the FLIP too?
>> > > > > > > > > > >> > > I got your point. It looks like a rational idea.
>> > > > However,
>> > > > > > > since
>> > > > > > > > > > lookup
>> > > > > > > > > > >> > has
>> > > > > > > > > > >> > > its clear async call requirement, are there any
>> real
>> > > use
>> > > > > > cases
>> > > > > > > > > that
>> > > > > > > > > > >> > > need this change? This will help us understand
>> the
>> > > > > > motivation.
>> > > > > > > > > After
>> > > > > > > > > > >> all,
>> > > > > > > > > > >> > > lateral join and temporal lookup join[1] are
>> quite
>> > > > > > different.
>> > > > > > > > > > >> > >
>> > > > > > > > > > >> > > Best regards,
>> > > > > > > > > > >> > > Jing
>> > > > > > > > > > >> > >
>> > > > > > > > > > >> > >
>> > > > > > > > > > >> > > [1]
>> > > > > > > > > > >> > >
>> > > > > > > > > > >> > >
>> > > > > > > > > > >> >
>> > > > > > > > > > >>
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/flink/blob/d90a72da2fd601ca4e2a46700e91ec5b348de2ad/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/AsyncTableFunction.java#L54
>> > > > > > > > > > >> > >
>> > > > > > > > > > >> > > On Wed, May 31, 2023 at 8:53 AM Aitozi <
>> > > > > > gjying1...@gmail.com>
>> > > > > > > > > > wrote:
>> > > > > > > > > > >> > >
>> > > > > > > > > > >> > > > Hi Jing,
>> > > > > > > > > > >> > > >     What do you think about it? Can we move
>> > forward
>> > > > this
>> > > > > > > > > feature?
>> > > > > > > > > > >> > > >
>> > > > > > > > > > >> > > > Thanks,
>> > > > > > > > > > >> > > > Aitozi.
>> > > > > > > > > > >> > > >
>> > > > > > > > > > >> > > > Aitozi <gjying1...@gmail.com> 于2023年5月29日周一
>> > > 09:56写道:
>> > > > > > > > > > >> > > >
>> > > > > > > > > > >> > > > > Hi Jing,
>> > > > > > > > > > >> > > > >     > "Do you mean to support the
>> > > AyncTableFunction
>> > > > > > beyond
>> > > > > > > > the
>> > > > > > > > > > >> > > > > LookupTableSource?"
>> > > > > > > > > > >> > > > > Yes, I mean to support the AyncTableFunction
>> > > beyond
>> > > > > the
>> > > > > > > > > > >> > > > LookupTableSource.
>> > > > > > > > > > >> > > > >
>> > > > > > > > > > >> > > > > The "AsyncTableFunction" is the function with
>> > > > ability
>> > > > > to
>> > > > > > > be
>> > > > > > > > > > >> executed
>> > > > > > > > > > >> > > > async
>> > > > > > > > > > >> > > > > (with AsyncWaitOperator).
>> > > > > > > > > > >> > > > > The async lookup join is a one of usage of
>> this.
>> > > So,
>> > > > > we
>> > > > > > > > don't
>> > > > > > > > > > >> have to
>> > > > > > > > > > >> > > > bind
>> > > > > > > > > > >> > > > > the AyncTableFunction with LookupTableSource.
>> > > > > > > > > > >> > > > > If User-defined AsyncTableFunction is
>> supported,
>> > > > user
>> > > > > > can
>> > > > > > > > > > directly
>> > > > > > > > > > >> > use
>> > > > > > > > > > >> > > > > lateral table syntax to perform async
>> operation.
>> > > > > > > > > > >> > > > >
>> > > > > > > > > > >> > > > > > "It would be better if you could elaborate
>> the
>> > > > > > proposed
>> > > > > > > > > > changes
>> > > > > > > > > > >> wrt
>> > > > > > > > > > >> > > the
>> > > > > > > > > > >> > > > > CorrelatedCodeGenerator with more details"
>> > > > > > > > > > >> > > > >
>> > > > > > > > > > >> > > > > In the proposal, we use lateral table syntax
>> to
>> > > > > support
>> > > > > > > the
>> > > > > > > > > > async
>> > > > > > > > > > >> > table
>> > > > > > > > > > >> > > > > function. So the planner will also treat this
>> > > > > statement
>> > > > > > > to a
>> > > > > > > > > > >> > > > > CommonExecCorrelate node. So the runtime code
>> > > should
>> > > > > be
>> > > > > > > > > > generated
>> > > > > > > > > > >> in
>> > > > > > > > > > >> > > > > CorrelatedCodeGenerator.
>> > > > > > > > > > >> > > > > In CorrelatedCodeGenerator, we will know the
>> > > > > > > TableFunction's
>> > > > > > > > > > Kind
>> > > > > > > > > > >> of
>> > > > > > > > > > >> > > > > `FunctionKind.Table` or
>> > `FunctionKind.ASYNC_TABLE`
>> > > > > > > > > > >> > > > > For  `FunctionKind.ASYNC_TABLE` we can
>> generate
>> > a
>> > > > > > > > > > >> AsyncWaitOperator
>> > > > > > > > > > >> > to
>> > > > > > > > > > >> > > > > execute the async table function.
>> > > > > > > > > > >> > > > >
>> > > > > > > > > > >> > > > >
>> > > > > > > > > > >> > > > > Thanks,
>> > > > > > > > > > >> > > > > Aitozi.
>> > > > > > > > > > >> > > > >
>> > > > > > > > > > >> > > > >
>> > > > > > > > > > >> > > > > Jing Ge <j...@ververica.com.invalid>
>> > > 于2023年5月29日周一
>> > > > > > > 03:22写道:
>> > > > > > > > > > >> > > > >
>> > > > > > > > > > >> > > > >> Hi Aitozi,
>> > > > > > > > > > >> > > > >>
>> > > > > > > > > > >> > > > >> Thanks for the clarification. The naming
>> > "Lookup"
>> > > > > might
>> > > > > > > > > suggest
>> > > > > > > > > > >> > using
>> > > > > > > > > > >> > > it
>> > > > > > > > > > >> > > > >> for table look up. But conceptually what the
>> > > eval()
>> > > > > > > method
>> > > > > > > > > will
>> > > > > > > > > > >> do
>> > > > > > > > > > >> > is
>> > > > > > > > > > >> > > to
>> > > > > > > > > > >> > > > >> get a collection of results(Row, RowData)
>> from
>> > > the
>> > > > > > given
>> > > > > > > > > keys.
>> > > > > > > > > > >> How
>> > > > > > > > > > >> > it
>> > > > > > > > > > >> > > > will
>> > > > > > > > > > >> > > > >> be done depends on the implementation, i.e.
>> you
>> > > can
>> > > > > > > > implement
>> > > > > > > > > > >> your
>> > > > > > > > > > >> > own
>> > > > > > > > > > >> > > > >> Source[1][2]. The example in the FLIP
>> should be
>> > > > able
>> > > > > to
>> > > > > > > be
>> > > > > > > > > > >> handled
>> > > > > > > > > > >> > in
>> > > > > > > > > > >> > > > this
>> > > > > > > > > > >> > > > >> way.
>> > > > > > > > > > >> > > > >>
>> > > > > > > > > > >> > > > >> Do you mean to support the AyncTableFunction
>> > > beyond
>> > > > > the
>> > > > > > > > > > >> > > > LookupTableSource?
>> > > > > > > > > > >> > > > >> It would be better if you could elaborate
>> the
>> > > > > proposed
>> > > > > > > > > changes
>> > > > > > > > > > >> wrt
>> > > > > > > > > > >> > the
>> > > > > > > > > > >> > > > >> CorrelatedCodeGenerator with more details.
>> > > Thanks!
>> > > > > > > > > > >> > > > >>
>> > > > > > > > > > >> > > > >> Best regards,
>> > > > > > > > > > >> > > > >> Jing
>> > > > > > > > > > >> > > > >>
>> > > > > > > > > > >> > > > >> [1]
>> > > > > > > > > > >> > > > >>
>> > > > > > > > > > >> > > > >>
>> > > > > > > > > > >> > > >
>> > > > > > > > > > >> > >
>> > > > > > > > > > >> >
>> > > > > > > > > > >>
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L64
>> > > > > > > > > > >> > > > >> [2]
>> > > > > > > > > > >> > > > >>
>> > > > > > > > > > >> > > > >>
>> > > > > > > > > > >> > > >
>> > > > > > > > > > >> > >
>> > > > > > > > > > >> >
>> > > > > > > > > > >>
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/AsyncTableFunctionProvider.java#L49
>> > > > > > > > > > >> > > > >>
>> > > > > > > > > > >> > > > >> On Sat, May 27, 2023 at 9:48 AM Aitozi <
>> > > > > > > > gjying1...@gmail.com
>> > > > > > > > > >
>> > > > > > > > > > >> > wrote:
>> > > > > > > > > > >> > > > >>
>> > > > > > > > > > >> > > > >> > Hi Jing,
>> > > > > > > > > > >> > > > >> >     Thanks for your response. As stated in
>> > the
>> > > > > FLIP,
>> > > > > > > the
>> > > > > > > > > > >> purpose
>> > > > > > > > > > >> > of
>> > > > > > > > > > >> > > > this
>> > > > > > > > > > >> > > > >> > FLIP is meant to support
>> > > > > > > > > > >> > > > >> > user-defined async table function. As
>> > described
>> > > > in
>> > > > > > > flink
>> > > > > > > > > > >> document
>> > > > > > > > > > >> > > [1]
>> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > >> > > > >> > Async table functions are special
>> functions
>> > for
>> > > > > table
>> > > > > > > > > sources
>> > > > > > > > > > >> that
>> > > > > > > > > > >> > > > >> perform
>> > > > > > > > > > >> > > > >> > > a lookup.
>> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > >> > > > >> > So end user can not directly define and
>> use
>> > > async
>> > > > > > table
>> > > > > > > > > > >> function
>> > > > > > > > > > >> > > now.
>> > > > > > > > > > >> > > > An
>> > > > > > > > > > >> > > > >> > user case is reported in [2]
>> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > >> > > > >> > So, in conclusion, no new interface is
>> > > > introduced,
>> > > > > > but
>> > > > > > > we
>> > > > > > > > > > >> extend
>> > > > > > > > > > >> > the
>> > > > > > > > > > >> > > > >> > ability to support user-defined async
>> table
>> > > > > function.
>> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > >> > > > >> > [1]:
>> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > >> > > > >>
>> > > > > > > > > > >> > > >
>> > > > > > > > > > >> > >
>> > > > > > > > > > >> >
>> > > > > > > > > > >>
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
>> > > > > > > > > > >> > > > >> > [2]:
>> > > > > > > > > > >> > >
>> > > > > > > >
>> > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
>> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > >> > > > >> > Thanks.
>> > > > > > > > > > >> > > > >> > Aitozi.
>> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > >> > > > >> > Jing Ge <j...@ververica.com.invalid>
>> > > > 于2023年5月27日周六
>> > > > > > > > > 06:40写道:
>> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > >> > > > >> > > Hi Aitozi,
>> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > >> > > > >> > > Thanks for your proposal. I am not quite
>> > sure
>> > > > if
>> > > > > I
>> > > > > > > > > > understood
>> > > > > > > > > > >> > your
>> > > > > > > > > > >> > > > >> > thoughts
>> > > > > > > > > > >> > > > >> > > correctly. You described a special case
>> > > > > > > implementation
>> > > > > > > > of
>> > > > > > > > > > the
>> > > > > > > > > > >> > > > >> > > AsyncTableFunction with on public API
>> > > changes.
>> > > > > > Would
>> > > > > > > > you
>> > > > > > > > > > >> please
>> > > > > > > > > > >> > > > >> elaborate
>> > > > > > > > > > >> > > > >> > > your purpose of writing a FLIP
>> according to
>> > > the
>> > > > > > FLIP
>> > > > > > > > > > >> > > > documentation[1]?
>> > > > > > > > > > >> > > > >> > > Thanks!
>> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > >> > > > >> > > [1]
>> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > >> > > > >>
>> > > > > > > > > > >> > > >
>> > > > > > > > > > >> > >
>> > > > > > > > > > >> >
>> > > > > > > > > > >>
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
>> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > >> > > > >> > > Best regards,
>> > > > > > > > > > >> > > > >> > > Jing
>> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > >> > > > >> > > On Wed, May 24, 2023 at 1:07 PM Aitozi <
>> > > > > > > > > > gjying1...@gmail.com
>> > > > > > > > > > >> >
>> > > > > > > > > > >> > > > wrote:
>> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > >> > > > >> > > > May I ask for some feedback  :D
>> > > > > > > > > > >> > > > >> > > >
>> > > > > > > > > > >> > > > >> > > > Thanks,
>> > > > > > > > > > >> > > > >> > > > Aitozi
>> > > > > > > > > > >> > > > >> > > >
>> > > > > > > > > > >> > > > >> > > > Aitozi <gjying1...@gmail.com>
>> > > 于2023年5月23日周二
>> > > > > > > 19:14写道:
>> > > > > > > > > > >> > > > >> > > > >
>> > > > > > > > > > >> > > > >> > > > > Just catch an user case report from
>> > > Giannis
>> > > > > > > Polyzos
>> > > > > > > > > for
>> > > > > > > > > > >> this
>> > > > > > > > > > >> > > > >> usage:
>> > > > > > > > > > >> > > > >> > > > >
>> > > > > > > > > > >> > > > >> > > > >
>> > > > > > > > > > >> > > >
>> > > > > > > > >
>> > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
>> > > > > > > > > > >> > > > >> > > > >
>> > > > > > > > > > >> > > > >> > > > > Aitozi <gjying1...@gmail.com>
>> > > > 于2023年5月23日周二
>> > > > > > > > 17:45写道:
>> > > > > > > > > > >> > > > >> > > > > >
>> > > > > > > > > > >> > > > >> > > > > > Hi guys,
>> > > > > > > > > > >> > > > >> > > > > >     I want to bring up a
>> discussion
>> > > about
>> > > > > > > adding
>> > > > > > > > > > >> support
>> > > > > > > > > > >> > of
>> > > > > > > > > > >> > > > User
>> > > > > > > > > > >> > > > >> > > > > > Defined AsyncTableFunction in
>> Flink.
>> > > > > > > > > > >> > > > >> > > > > > Currently, async table function
>> are
>> > > > special
>> > > > > > > > > functions
>> > > > > > > > > > >> for
>> > > > > > > > > > >> > > > table
>> > > > > > > > > > >> > > > >> > > source
>> > > > > > > > > > >> > > > >> > > > > > to perform
>> > > > > > > > > > >> > > > >> > > > > > async lookup. However, it's worth
>> to
>> > > > > support
>> > > > > > > the
>> > > > > > > > > user
>> > > > > > > > > > >> > > defined
>> > > > > > > > > > >> > > > >> async
>> > > > > > > > > > >> > > > >> > > > > > table function.
>> > > > > > > > > > >> > > > >> > > > > > Because, in this way, the end SQL
>> > user
>> > > > can
>> > > > > > > > leverage
>> > > > > > > > > > it
>> > > > > > > > > > >> to
>> > > > > > > > > > >> > > > >> perform
>> > > > > > > > > > >> > > > >> > the
>> > > > > > > > > > >> > > > >> > > > > > async operation
>> > > > > > > > > > >> > > > >> > > > > > which is useful to maximum the
>> system
>> > > > > > > throughput
>> > > > > > > > > > >> > especially
>> > > > > > > > > > >> > > > for
>> > > > > > > > > > >> > > > >> IO
>> > > > > > > > > > >> > > > >> > > > > > bottleneck case.
>> > > > > > > > > > >> > > > >> > > > > >
>> > > > > > > > > > >> > > > >> > > > > > You can find some more detail in
>> [1].
>> > > > > > > > > > >> > > > >> > > > > >
>> > > > > > > > > > >> > > > >> > > > > > Looking forward to feedback
>> > > > > > > > > > >> > > > >> > > > > >
>> > > > > > > > > > >> > > > >> > > > > >
>> > > > > > > > > > >> > > > >> > > > > > [1]:
>> > > > > > > > > > >> > > > >> > > >
>> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > >> > > > >>
>> > > > > > > > > > >> > > >
>> > > > > > > > > > >> > >
>> > > > > > > > > > >> >
>> > > > > > > > > > >>
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
>> > > > > > > > > > >> > > > >> > > > > >
>> > > > > > > > > > >> > > > >> > > > > > Thanks,
>> > > > > > > > > > >> > > > >> > > > > > Aitozi.
>> > > > > > > > > > >> > > > >> > > >
>> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > >> > > > >>
>> > > > > > > > > > >> > > > >
>> > > > > > > > > > >> > > >
>> > > > > > > > > > >> > >
>> > > > > > > > > > >> >
>> > > > > > > > > > >>
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>

Reply via email to