Re: Getting runtime context from scalar and table functions

[Fabian Hueske]

Yes, metrics won't work for this use case.
Before we had proper metrics support, accumulators were often used as a
work around.

In general, the Table API & SQL in Flink are designed to keep all data in
tables and not "leak" data on the side.

Best, Fabian

2018-04-04 22:08 GMT+02:00 Darshan Singh :

> I doubt the metrics will work as I will need to get the String output. I
> will need to use figure out something else.
>
> Basically I will pass my function a string and will get some columns back
> but if my string is something special. Then I will need to get extra
> information. I am using the data set so I think I wont be able to use
> side-output.
>
> As of now I return my POJO along-with the String(only if it is special)
> and then i select these from table itself. I was just wondering if there
> are other better alternate where my existing table return my POJO and I
> somehow find those special strings once job is done. The number of these
> strings is very small. I thought I could create a custom accumulator which
> will work more like list and in the end I will get all values.
>
> I guess I will need to keep trying something.
>
> Thanks
>
> On Wed, Apr 4, 2018 at 9:01 PM, Darshan Singh 
> wrote:
>
>> Thanks for getting back to me. I was trying to see various options to
>> accumulate(sort of list append) some of my data. The amount of appended
>> data is quite small so I wanted to see if I can use accumulators and once
>> the job is done I can get all the data and then use the way I want to use.
>>
>> I guess I will try the metrics as well to see how that goes.
>>
>> Thanks
>>
>> On Wed, Apr 4, 2018 at 8:51 PM, Fabian Hueske  wrote:
>>
>>> Hi Darshan,
>>>
>>> Accumulators are not exposed to UDFs of the Table API / SQL.
>>> What's your use case for these? Would metrics do the job as well?
>>>
>>> Best, Fabian
>>>
>>> 2018-04-04 21:31 GMT+02:00 Darshan Singh :
>>>
 Hi,

 I would like to use accumulators with table /scalar functions. However,
 I am not able to figure out how to get the runtime context from inside of
 scalar function open method.
 Only thing i see is function context which can not provide the runtime
 context.

 I tried using AbstractRichFunction.getRuntimeContext()
 
 .  which I guess is not allowed from open of scalar or table functions.

 Any help will be appreciated.

 Thanks

>>>
>>>
>>
>

Re: Getting runtime context from scalar and table functions

[Darshan Singh]

I doubt the metrics will work as I will need to get the String output. I
will need to use figure out something else.

Basically I will pass my function a string and will get some columns back
but if my string is something special. Then I will need to get extra
information. I am using the data set so I think I wont be able to use
side-output.

As of now I return my POJO along-with the String(only if it is special) and
then i select these from table itself. I was just wondering if there are
other better alternate where my existing table return my POJO and I somehow
find those special strings once job is done. The number of these strings is
very small. I thought I could create a custom accumulator which will work
more like list and in the end I will get all values.

I guess I will need to keep trying something.

Thanks

On Wed, Apr 4, 2018 at 9:01 PM, Darshan Singh 
wrote:

> Thanks for getting back to me. I was trying to see various options to
> accumulate(sort of list append) some of my data. The amount of appended
> data is quite small so I wanted to see if I can use accumulators and once
> the job is done I can get all the data and then use the way I want to use.
>
> I guess I will try the metrics as well to see how that goes.
>
> Thanks
>
> On Wed, Apr 4, 2018 at 8:51 PM, Fabian Hueske  wrote:
>
>> Hi Darshan,
>>
>> Accumulators are not exposed to UDFs of the Table API / SQL.
>> What's your use case for these? Would metrics do the job as well?
>>
>> Best, Fabian
>>
>> 2018-04-04 21:31 GMT+02:00 Darshan Singh :
>>
>>> Hi,
>>>
>>> I would like to use accumulators with table /scalar functions. However,
>>> I am not able to figure out how to get the runtime context from inside of
>>> scalar function open method.
>>> Only thing i see is function context which can not provide the runtime
>>> context.
>>>
>>> I tried using AbstractRichFunction.getRuntimeContext()
>>> 
>>> .  which I guess is not allowed from open of scalar or table functions.
>>>
>>> Any help will be appreciated.
>>>
>>> Thanks
>>>
>>
>>
>

Re: Getting runtime context from scalar and table functions

[Fabian Hueske]

Hi Darshan,

Accumulators are not exposed to UDFs of the Table API / SQL.
What's your use case for these? Would metrics do the job as well?

Best, Fabian

2018-04-04 21:31 GMT+02:00 Darshan Singh :

> Hi,
>
> I would like to use accumulators with table /scalar functions. However, I
> am not able to figure out how to get the runtime context from inside of
> scalar function open method.
> Only thing i see is function context which can not provide the runtime
> context.
>
> I tried using AbstractRichFunction.getRuntimeContext()
> 
> .  which I guess is not allowed from open of scalar or table functions.
>
> Any help will be appreciated.
>
> Thanks
>