RE: DI in flink

[Schwalbe Matthias]

Hi Yashoda,

I use Spring-Boot to setup my job networks and DI-compose streaming components 
like operators/functions etc.
Important part is that all components need to be serializable in order for this 
to work.
Specific task implementations are a little more difficult (little experience) 
to set up in a DI way. If I’m not mistaken, Flink uses factories for this.

Sincere greetings

Thias


From: Yashoda Krishna T 
Sent: Wednesday, February 15, 2023 6:19 AM
To: Austin Cawley-Edwards 
Cc: user 
Subject: Re: DI in flink

Thanks Austin.
I can make use of Rich functions to solve my problem.

Thanks
Yashoda

On Wed, Feb 15, 2023 at 12:42 AM Austin Cawley-Edwards 
mailto:austin.caw...@gmail.com>> wrote:
(note: please keep user@flink.apache.org<mailto:user@flink.apache.org> included 
in replies)

Ah, I see. Then no, this is not provided by Flink. When I've used dependency 
inject with Flink in the past, I instantiated everything in the `open()` method 
of the Flink Rich* classes. Could you solve this by having a common base Sink 
class or builder that does the configuring? I'm just wondering why it's 
necessary to solve it in Flink itself.

Best,
Austin

On Tue, Feb 14, 2023 at 11:05 AM Yashoda Krishna T 
mailto:yashoda.kris...@unbxd.com>> wrote:
This is my use case.
I have a sink function to push streaming data to S3. And I have a class lets 
call S3ConnProvider that provides me a connection object to S3, and a class 
lets say S3Util that has functions over S3 which injects S3ConnProvider.
If dependency injection works I can inject S3Util alone in my SinkFunction 
class. If not I have to initialize S3ConnProvider first and then S3Util.
This can become complex if there are too many initializations required 
depending on the use case.
Diese Nachricht ist ausschliesslich für den Adressaten bestimmt und beinhaltet 
unter Umständen vertrauliche Mitteilungen. Da die Vertraulichkeit von 
e-Mail-Nachrichten nicht gewährleistet werden kann, übernehmen wir keine 
Haftung für die Gewährung der Vertraulichkeit und Unversehrtheit dieser 
Mitteilung. Bei irrtümlicher Zustellung bitten wir Sie um Benachrichtigung per 
e-Mail und um Löschung dieser Nachricht sowie eventueller Anhänge. Jegliche 
unberechtigte Verwendung oder Verbreitung dieser Informationen ist streng 
verboten.

This message is intended only for the named recipient and may contain 
confidential or privileged information. As the confidentiality of email 
communication cannot be guaranteed, we do not accept any responsibility for the 
confidentiality and the intactness of this message. If you have received it in 
error, please advise the sender by return e-mail and delete this message and 
any attachments. Any unauthorised use or dissemination of this information is 
strictly prohibited.

Re: DI in flink

[Yashoda Krishna T]

Thanks Austin.
I can make use of Rich functions to solve my problem.

Thanks
Yashoda

On Wed, Feb 15, 2023 at 12:42 AM Austin Cawley-Edwards <
austin.caw...@gmail.com> wrote:

> (note: please keep user@flink.apache.org included in replies)
>
> Ah, I see. Then no, this is not provided by Flink. When I've used
> dependency inject with Flink in the past, I instantiated everything in the
> `open()` method of the Flink Rich* classes. Could you solve this by having
> a common base Sink class or builder that does the configuring? I'm just
> wondering why it's necessary to solve it in Flink itself.
>
> Best,
> Austin
>
> On Tue, Feb 14, 2023 at 11:05 AM Yashoda Krishna T <
> yashoda.kris...@unbxd.com> wrote:
>
>> This is my use case.
>> I have a sink function to push streaming data to S3. And I have a class
>> lets call S3ConnProvider that provides me a connection object to S3, and a
>> class lets say S3Util that has functions over S3 which injects
>> S3ConnProvider.
>> If dependency injection works I can inject S3Util alone in my
>> SinkFunction class. If not I have to initialize S3ConnProvider first and
>> then S3Util.
>> This can become complex if there are too many initializations
>> required depending on the use case.
>>
>>>

Re: DI in flink

[Austin Cawley-Edwards]

(note: please keep user@flink.apache.org included in replies)

Ah, I see. Then no, this is not provided by Flink. When I've used
dependency inject with Flink in the past, I instantiated everything in the
`open()` method of the Flink Rich* classes. Could you solve this by having
a common base Sink class or builder that does the configuring? I'm just
wondering why it's necessary to solve it in Flink itself.

Best,
Austin

On Tue, Feb 14, 2023 at 11:05 AM Yashoda Krishna T <
yashoda.kris...@unbxd.com> wrote:

> This is my use case.
> I have a sink function to push streaming data to S3. And I have a class
> lets call S3ConnProvider that provides me a connection object to S3, and a
> class lets say S3Util that has functions over S3 which injects
> S3ConnProvider.
> If dependency injection works I can inject S3Util alone in my SinkFunction
> class. If not I have to initialize S3ConnProvider first and then S3Util.
> This can become complex if there are too many initializations
> required depending on the use case.
>
>>

Re: DI in flink

[Austin Cawley-Edwards]

What would be the benefits and features over what can be done in user land?

On Tue, Feb 14, 2023 at 10:41 Yashoda Krishna T 
wrote:

> Hi Austin
>
> Yes this can be done in Usrr land.
> Can we do it in flink land too?
>
> Thanks
> Yashoda
>
> On Tue, 14 Feb 2023, 9:05 pm Austin Cawley-Edwards, <
> austin.caw...@gmail.com> wrote:
>
>> Hey Yashoda,
>>
>> This can be done in userland (eg with Dagger )
>> unless you're wanting Flink to do something in addition?
>>
>> Best,
>> Austin
>>
>> On Tue, Feb 14, 2023 at 10:01 AM Yashoda Krishna T <
>> yashoda.kris...@unbxd.com> wrote:
>>
>>> Does flink support dependency injection in flink task functions in java?
>>> If not is there an alternative?
>>>
>>

Re: DI in flink

[Austin Cawley-Edwards]

Hey Yashoda,

This can be done in userland (eg with Dagger ) unless
you're wanting Flink to do something in addition?

Best,
Austin

On Tue, Feb 14, 2023 at 10:01 AM Yashoda Krishna T <
yashoda.kris...@unbxd.com> wrote:

> Does flink support dependency injection in flink task functions in java?
> If not is there an alternative?
>