Re: Regarding state access in UDF

[Kai Fu]

Hi Ingo,

Thank you for your advice, we've not tried it yet, we just thought it may
work that way, but now it seems not then. We'll see how it could match our
use case with the AggregateFunction interface.

On Thu, Jul 1, 2021 at 1:57 PM Ingo Bürk  wrote:

> Hi Kai,
>
> CheckpointedFunction is not an interface meant to be used with UDFs (in
> the Table API / SQL sense[1]), but is rather an interface for DataStream
> API[2]; the term "user-defined function" has a different meaning there. Did
> you actually try it to see if it works? I'd be surprised it it did.
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/table/functions/udfs/
> [2]
> https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/datastream/user_defined_functions/
>
>
> Ingo
>
> On Thu, Jul 1, 2021 at 5:17 AM Kai Fu  wrote:
>
>> Hi Ingo,
>>
>> Thank you for the reply, we actually need more fine-grained control on
>> the states in UDF. Per investigation, we found that the states can be
>> simply created/accessed via implementing `CheckpointedFunction` interface,
>> please advise if there is any side-effect by doing that.
>>
>> On Wed, Jun 30, 2021 at 10:33 PM Ingo Bürk  wrote:
>>
>>> Hi Kai,
>>>
>>> AggregateFunction and TableAggregateFunction are both stateful UDF
>>> interfaces. This should cover most scenarios given where they would be
>>> used. If you need more fine-grained control you can also always drop down
>>> into the DataStream API (using #toDataStream) and work there. Table API /
>>> SQL in general are higher-level abstractions where you cannot directly
>>> interact with operators.
>>>
>>> If this doesn't answer your question it would also be great if you could
>>> explain your use case more so we can understand it. Thanks!
>>>
>>>
>>> Best
>>> Ingo
>>>
>>> On Wed, Jun 30, 2021 at 3:37 PM Kai Fu  wrote:
>>>
 Hi team,

 We've a use case that needs to create/access state in UDF, while per
 the documentation
 
 and UDF interface
 .
 It does not provide such a way for that. We want to know if it is by design
 and is there any other approach for it.

 --
 *Best wishes,*
 *- Kai*

>>>
>>
>> --
>> *Best wishes,*
>> *- Kai*
>>
>

-- 
*Best wishes,*
*- Kai*

Re: Regarding state access in UDF

[Ingo Bürk]

Hi Kai,

CheckpointedFunction is not an interface meant to be used with UDFs (in the
Table API / SQL sense[1]), but is rather an interface for DataStream
API[2]; the term "user-defined function" has a different meaning there. Did
you actually try it to see if it works? I'd be surprised it it did.

[1]
https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/table/functions/udfs/
[2]
https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/datastream/user_defined_functions/


Ingo

On Thu, Jul 1, 2021 at 5:17 AM Kai Fu  wrote:

> Hi Ingo,
>
> Thank you for the reply, we actually need more fine-grained control on the
> states in UDF. Per investigation, we found that the states can be simply
> created/accessed via implementing `CheckpointedFunction` interface, please
> advise if there is any side-effect by doing that.
>
> On Wed, Jun 30, 2021 at 10:33 PM Ingo Bürk  wrote:
>
>> Hi Kai,
>>
>> AggregateFunction and TableAggregateFunction are both stateful UDF
>> interfaces. This should cover most scenarios given where they would be
>> used. If you need more fine-grained control you can also always drop down
>> into the DataStream API (using #toDataStream) and work there. Table API /
>> SQL in general are higher-level abstractions where you cannot directly
>> interact with operators.
>>
>> If this doesn't answer your question it would also be great if you could
>> explain your use case more so we can understand it. Thanks!
>>
>>
>> Best
>> Ingo
>>
>> On Wed, Jun 30, 2021 at 3:37 PM Kai Fu  wrote:
>>
>>> Hi team,
>>>
>>> We've a use case that needs to create/access state in UDF, while per the
>>> documentation
>>> 
>>> and UDF interface
>>> .
>>> It does not provide such a way for that. We want to know if it is by design
>>> and is there any other approach for it.
>>>
>>> --
>>> *Best wishes,*
>>> *- Kai*
>>>
>>
>
> --
> *Best wishes,*
> *- Kai*
>

Re: Regarding state access in UDF

[Kai Fu]

Hi Ingo,

Thank you for the reply, we actually need more fine-grained control on the
states in UDF. Per investigation, we found that the states can be simply
created/accessed via implementing `CheckpointedFunction` interface, please
advise if there is any side-effect by doing that.

On Wed, Jun 30, 2021 at 10:33 PM Ingo Bürk  wrote:

> Hi Kai,
>
> AggregateFunction and TableAggregateFunction are both stateful UDF
> interfaces. This should cover most scenarios given where they would be
> used. If you need more fine-grained control you can also always drop down
> into the DataStream API (using #toDataStream) and work there. Table API /
> SQL in general are higher-level abstractions where you cannot directly
> interact with operators.
>
> If this doesn't answer your question it would also be great if you could
> explain your use case more so we can understand it. Thanks!
>
>
> Best
> Ingo
>
> On Wed, Jun 30, 2021 at 3:37 PM Kai Fu  wrote:
>
>> Hi team,
>>
>> We've a use case that needs to create/access state in UDF, while per the
>> documentation
>> 
>> and UDF interface
>> .
>> It does not provide such a way for that. We want to know if it is by design
>> and is there any other approach for it.
>>
>> --
>> *Best wishes,*
>> *- Kai*
>>
>

-- 
*Best wishes,*
*- Kai*

Re: Regarding state access in UDF

[Ingo Bürk]

Hi Kai,

AggregateFunction and TableAggregateFunction are both stateful UDF
interfaces. This should cover most scenarios given where they would be
used. If you need more fine-grained control you can also always drop down
into the DataStream API (using #toDataStream) and work there. Table API /
SQL in general are higher-level abstractions where you cannot directly
interact with operators.

If this doesn't answer your question it would also be great if you could
explain your use case more so we can understand it. Thanks!


Best
Ingo

On Wed, Jun 30, 2021 at 3:37 PM Kai Fu  wrote:

> Hi team,
>
> We've a use case that needs to create/access state in UDF, while per the
> documentation
> 
> and UDF interface
> .
> It does not provide such a way for that. We want to know if it is by design
> and is there any other approach for it.
>
> --
> *Best wishes,*
> *- Kai*
>

Regarding state access in UDF

[Kai Fu]

Hi team,

We've a use case that needs to create/access state in UDF, while per the
documentation

and UDF interface
.
It does not provide such a way for that. We want to know if it is by design
and is there any other approach for it.

-- 
*Best wishes,*
*- Kai*