Re: [External] Re: Source vs SourceFunction and testing
Hi Carlos, You might want to join the discussion about FLIP-238[1] to share your thoughts with us. Thanks! Best regards, Jing [1] https://lists.apache.org/thread/7gjxto1rmkpff4kl54j8nlg5db2rqhkt On Thu, Jun 9, 2022 at 2:13 PM Sanabria, Carlos < carlos.sanab...@accenture.com> wrote: > Thanks for your quick response! > > Yes, this is exactly what we were looking for! > Seems like a really nice feature. Even better than the FromElementsSource > we were asking for, because it allows to generate the events dynamically. > > Is there any way we can vote for the FLIP-238 to be accepted? > > -Original Message- > From: Qingsheng Ren > Sent: jueves, 9 de junio de 2022 12:16 > To: Sanabria, Carlos > Cc: user > Subject: Re: [External] Re: Source vs SourceFunction and testing > > Hi Carlos, > > FLIP-238 [1] is proposing a FLIP-27-based data generator source and I > think this is what you are looking for. This FLIP was created just days ago > so it may take some time to get accepted and released. > > [1] > https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_FLINK_FLIP-2D238-253A-2BIntroduce-2BFLIP-2D27-2Dbased-2BData-2BGenerator-2BSource=DwIFaQ=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU=Q6157zGhiDIuCzxlSpEZgTNbdEC4jbNL0iaPBqIxifg=HzoMy2jOOV5oDa2SuDfcm4cy2LK-9DOQ530Dmvi2r4f0jhOVKmqOTToQ6ArM3q1F=ZRTHrrDGp1m0Po50VeAFjAEQBjCM28naJRNWM4CZQoA= > > Cheers, > > Qingsheng > > On Thu, Jun 9, 2022 at 6:05 PM Sanabria, Carlos < > carlos.sanab...@accenture.com> wrote: > > > > Hi everyone! > > > > Sorry for reopening the thread, but I am having some problems related to > this case while migrating our code from Flink 1.12 to Flink 1.15. > > > > We have a base project that encapsulates a ton of common code and > configurations. One of the abstractions we have is an AbstractDataStreamJob > class that has generic Sources and Sinks. We implemented it like this since > Flink 1.8, following the recommendations of the Flink documentation [1]: > > > > "Apache Flink provides a JUnit rule called MiniClusterWithClientResource > for testing complete jobs against a local, embedded mini cluster. called > MiniClusterWithClientResource. > > ... > > A few remarks on integration testing with MiniClusterWithClientResource: > > - In order not to copy your whole pipeline code from production to test, > make sources and sinks pluggable in your production code and inject special > test sources and test sinks in your tests. > > ..." > > > > This way, we can create the real Kafka Sources and Sinks in the Main > class of the job, and also create the test Sources and Sinks in the Junit > tests, and inject them in the AbstractDataStreamJob class. > > > > The problem comes with the new Source interface and the end to end tests > against the local embedded mini cluster. Prior to Flink 1.15, we used the > FromElementsFunction to create the test SourceFunction. Now that we changed > the code to use the new Source interface, we cannot use the > FromElementsFunction anymore, and we haven't found an equivalent > FromElementsSource class with the same functionality but implemented using > the new Source API. > > > > We want to keep the same structure in the AbstractDataStreamJob class > (with generic and pluggable sources and sinks), as we think it is the most > elegant and generic solution. > > > > Is it planned to implement a FromElementsSource class that extends the > new Source API? Is there any other alternative that may serve as a > workaround for the moment? > > > > We have tried to implement a custom Source for this use case, but it > seems like an overwhelming task and we do not want to reinvent the wheel > either. If it is planned to implement the FromElementsSource we'd rather > prefer to wait for it. > > > > Thanks! > > Carlos > > > > [1] > > https://urldefense.proofpoint.com/v2/url?u=https-3A__nightlies.apache. > > org_flink_flink-2Ddocs-2Drelease-2D1.15_docs_dev_datastream_testing_-2 > > 3junit-2Drule-2Dminiclusterwithclientresource=DwIFaQ=eIGjsITfXP_y- > > DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU=Q6157zGhiDIuCzxlSpEZgTNbdEC4jbNL0iaPB > > qIxifg=HzoMy2jOOV5oDa2SuDfcm4cy2LK-9DOQ530Dmvi2r4f0jhOVKmqOTToQ6ArM3 > > q1F=RKTpSSHRudC_BMmTz9xhGOT91uAAbp7HPEejuTihHvU= > > > > -Original Message- > > From: Qingsheng Ren > > Sent: miércoles, 25 de mayo de 2022 12:10 > > To: Piotr Domagalski > > Cc: user@flink.apache.org > > Subject: [External] Re: Source vs SourceFunction and testing > > > > This message is from an EXTERNAL SENDER - be CAUTIOUS, particularly with >
RE: [External] Re: Source vs SourceFunction and testing
Thanks for your quick response! Yes, this is exactly what we were looking for! Seems like a really nice feature. Even better than the FromElementsSource we were asking for, because it allows to generate the events dynamically. Is there any way we can vote for the FLIP-238 to be accepted? -Original Message- From: Qingsheng Ren Sent: jueves, 9 de junio de 2022 12:16 To: Sanabria, Carlos Cc: user Subject: Re: [External] Re: Source vs SourceFunction and testing Hi Carlos, FLIP-238 [1] is proposing a FLIP-27-based data generator source and I think this is what you are looking for. This FLIP was created just days ago so it may take some time to get accepted and released. [1] https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_FLINK_FLIP-2D238-253A-2BIntroduce-2BFLIP-2D27-2Dbased-2BData-2BGenerator-2BSource=DwIFaQ=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU=Q6157zGhiDIuCzxlSpEZgTNbdEC4jbNL0iaPBqIxifg=HzoMy2jOOV5oDa2SuDfcm4cy2LK-9DOQ530Dmvi2r4f0jhOVKmqOTToQ6ArM3q1F=ZRTHrrDGp1m0Po50VeAFjAEQBjCM28naJRNWM4CZQoA= Cheers, Qingsheng On Thu, Jun 9, 2022 at 6:05 PM Sanabria, Carlos wrote: > > Hi everyone! > > Sorry for reopening the thread, but I am having some problems related to this > case while migrating our code from Flink 1.12 to Flink 1.15. > > We have a base project that encapsulates a ton of common code and > configurations. One of the abstractions we have is an AbstractDataStreamJob > class that has generic Sources and Sinks. We implemented it like this since > Flink 1.8, following the recommendations of the Flink documentation [1]: > > "Apache Flink provides a JUnit rule called MiniClusterWithClientResource for > testing complete jobs against a local, embedded mini cluster. called > MiniClusterWithClientResource. > ... > A few remarks on integration testing with MiniClusterWithClientResource: > - In order not to copy your whole pipeline code from production to test, make > sources and sinks pluggable in your production code and inject special test > sources and test sinks in your tests. > ..." > > This way, we can create the real Kafka Sources and Sinks in the Main class of > the job, and also create the test Sources and Sinks in the Junit tests, and > inject them in the AbstractDataStreamJob class. > > The problem comes with the new Source interface and the end to end tests > against the local embedded mini cluster. Prior to Flink 1.15, we used the > FromElementsFunction to create the test SourceFunction. Now that we changed > the code to use the new Source interface, we cannot use the > FromElementsFunction anymore, and we haven't found an equivalent > FromElementsSource class with the same functionality but implemented using > the new Source API. > > We want to keep the same structure in the AbstractDataStreamJob class (with > generic and pluggable sources and sinks), as we think it is the most elegant > and generic solution. > > Is it planned to implement a FromElementsSource class that extends the new > Source API? Is there any other alternative that may serve as a workaround for > the moment? > > We have tried to implement a custom Source for this use case, but it seems > like an overwhelming task and we do not want to reinvent the wheel either. If > it is planned to implement the FromElementsSource we'd rather prefer to wait > for it. > > Thanks! > Carlos > > [1] > https://urldefense.proofpoint.com/v2/url?u=https-3A__nightlies.apache. > org_flink_flink-2Ddocs-2Drelease-2D1.15_docs_dev_datastream_testing_-2 > 3junit-2Drule-2Dminiclusterwithclientresource=DwIFaQ=eIGjsITfXP_y- > DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU=Q6157zGhiDIuCzxlSpEZgTNbdEC4jbNL0iaPB > qIxifg=HzoMy2jOOV5oDa2SuDfcm4cy2LK-9DOQ530Dmvi2r4f0jhOVKmqOTToQ6ArM3 > q1F=RKTpSSHRudC_BMmTz9xhGOT91uAAbp7HPEejuTihHvU= > > -Original Message----- > From: Qingsheng Ren > Sent: miércoles, 25 de mayo de 2022 12:10 > To: Piotr Domagalski > Cc: user@flink.apache.org > Subject: [External] Re: Source vs SourceFunction and testing > > This message is from an EXTERNAL SENDER - be CAUTIOUS, particularly with > links and attachments. > > Glad to see you have resolved the issue! > > If you want to learn more about the Source API, the Flink document [1] has a > detailed description about it. The original proposal FLIP-27 [2] is also a > good reference. > > [1] > https://urldefense.proofpoint.com/v2/url?u=https-3A__nightlies.apache. > org_flink_flink-2Ddocs-2Drelease-2D1.15_docs_dev_datastream_sources_ > =DwIFaQ=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU=Q6157zGhiDIuCz > xlSpEZgTNbdEC4jbNL0iaPBqIxifg=UaxcpZWDroSZiLenzhGnRRQockpCt3Hg9jXU50 > aRVltFNinifOKvurHPTzdPL1da=lQGFDQJRG2BADprHFhkCefHCPTjDTh-OGIz4xFl-1 > W8= [2
Re: [External] Re: Source vs SourceFunction and testing
Hi Carlos,
FLIP-238 [1] is proposing a FLIP-27-based data generator source and I
think this is what you are looking for. This FLIP was created just
days ago so it may take some time to get accepted and released.
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-238%3A+Introduce+FLIP-27-based+Data+Generator+Source
Cheers,
Qingsheng
On Thu, Jun 9, 2022 at 6:05 PM Sanabria, Carlos
wrote:
>
> Hi everyone!
>
> Sorry for reopening the thread, but I am having some problems related to this
> case while migrating our code from Flink 1.12 to Flink 1.15.
>
> We have a base project that encapsulates a ton of common code and
> configurations. One of the abstractions we have is an AbstractDataStreamJob
> class that has generic Sources and Sinks. We implemented it like this since
> Flink 1.8, following the recommendations of the Flink documentation [1]:
>
> "Apache Flink provides a JUnit rule called MiniClusterWithClientResource for
> testing complete jobs against a local, embedded mini cluster. called
> MiniClusterWithClientResource.
> ...
> A few remarks on integration testing with MiniClusterWithClientResource:
> - In order not to copy your whole pipeline code from production to test, make
> sources and sinks pluggable in your production code and inject special test
> sources and test sinks in your tests.
> ..."
>
> This way, we can create the real Kafka Sources and Sinks in the Main class of
> the job, and also create the test Sources and Sinks in the Junit tests, and
> inject them in the AbstractDataStreamJob class.
>
> The problem comes with the new Source interface and the end to end tests
> against the local embedded mini cluster. Prior to Flink 1.15, we used the
> FromElementsFunction to create the test SourceFunction. Now that we changed
> the code to use the new Source interface, we cannot use the
> FromElementsFunction anymore, and we haven't found an equivalent
> FromElementsSource class with the same functionality but implemented using
> the new Source API.
>
> We want to keep the same structure in the AbstractDataStreamJob class (with
> generic and pluggable sources and sinks), as we think it is the most elegant
> and generic solution.
>
> Is it planned to implement a FromElementsSource class that extends the new
> Source API? Is there any other alternative that may serve as a workaround for
> the moment?
>
> We have tried to implement a custom Source for this use case, but it seems
> like an overwhelming task and we do not want to reinvent the wheel either. If
> it is planned to implement the FromElementsSource we'd rather prefer to wait
> for it.
>
> Thanks!
> Carlos
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/datastream/testing/#junit-rule-miniclusterwithclientresource
>
> -----Original Message-
> From: Qingsheng Ren
> Sent: miércoles, 25 de mayo de 2022 12:10
> To: Piotr Domagalski
> Cc: user@flink.apache.org
> Subject: [External] Re: Source vs SourceFunction and testing
>
> This message is from an EXTERNAL SENDER - be CAUTIOUS, particularly with
> links and attachments.
>
> Glad to see you have resolved the issue!
>
> If you want to learn more about the Source API, the Flink document [1] has a
> detailed description about it. The original proposal FLIP-27 [2] is also a
> good reference.
>
> [1]
> https://urldefense.proofpoint.com/v2/url?u=https-3A__nightlies.apache.org_flink_flink-2Ddocs-2Drelease-2D1.15_docs_dev_datastream_sources_=DwIFaQ=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU=Q6157zGhiDIuCzxlSpEZgTNbdEC4jbNL0iaPBqIxifg=UaxcpZWDroSZiLenzhGnRRQockpCt3Hg9jXU50aRVltFNinifOKvurHPTzdPL1da=lQGFDQJRG2BADprHFhkCefHCPTjDTh-OGIz4xFl-1W8=
> [2]
> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_FLINK_FLIP-2D27-253A-2BRefactor-2BSource-2BInterface=DwIFaQ=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU=Q6157zGhiDIuCzxlSpEZgTNbdEC4jbNL0iaPBqIxifg=UaxcpZWDroSZiLenzhGnRRQockpCt3Hg9jXU50aRVltFNinifOKvurHPTzdPL1da=SqQEnABQt5ZGeX8rUZVEI8wyNDe2GlRNBHtZv5V3MIQ=
>
> Cheers,
>
> Qingsheng
>
> > On May 25, 2022, at 17:54, Piotr Domagalski wrote:
> >
> > Thank you Qingsheng, this context helps a lot!
> >
> > And once again thank you all for being such a helpful community!
> >
> > P.S. I actually struggled for a bit trying to understand why my refactored
> > solution which accepts DataStream<> wouldn't work ("no operators defined in
> > the streaming topology"). Turns out, my assumption that I can call
> > StreamExecutionEnvironment.getExecutionEnvironment() multiple times and get
> > the same environment, was wrong. I had env.addSource and env.fromSource
> > calls us
RE: [External] Re: Source vs SourceFunction and testing
Hi everyone!
Sorry for reopening the thread, but I am having some problems related to this
case while migrating our code from Flink 1.12 to Flink 1.15.
We have a base project that encapsulates a ton of common code and
configurations. One of the abstractions we have is an AbstractDataStreamJob
class that has generic Sources and Sinks. We implemented it like this since
Flink 1.8, following the recommendations of the Flink documentation [1]:
"Apache Flink provides a JUnit rule called MiniClusterWithClientResource for
testing complete jobs against a local, embedded mini cluster. called
MiniClusterWithClientResource.
...
A few remarks on integration testing with MiniClusterWithClientResource:
- In order not to copy your whole pipeline code from production to test, make
sources and sinks pluggable in your production code and inject special test
sources and test sinks in your tests.
..."
This way, we can create the real Kafka Sources and Sinks in the Main class of
the job, and also create the test Sources and Sinks in the Junit tests, and
inject them in the AbstractDataStreamJob class.
The problem comes with the new Source interface and the end to end tests
against the local embedded mini cluster. Prior to Flink 1.15, we used the
FromElementsFunction to create the test SourceFunction. Now that we changed the
code to use the new Source interface, we cannot use the FromElementsFunction
anymore, and we haven't found an equivalent FromElementsSource class with the
same functionality but implemented using the new Source API.
We want to keep the same structure in the AbstractDataStreamJob class (with
generic and pluggable sources and sinks), as we think it is the most elegant
and generic solution.
Is it planned to implement a FromElementsSource class that extends the new
Source API? Is there any other alternative that may serve as a workaround for
the moment?
We have tried to implement a custom Source for this use case, but it seems like
an overwhelming task and we do not want to reinvent the wheel either. If it is
planned to implement the FromElementsSource we'd rather prefer to wait for it.
Thanks!
Carlos
[1]
https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/datastream/testing/#junit-rule-miniclusterwithclientresource
-Original Message-
From: Qingsheng Ren
Sent: miércoles, 25 de mayo de 2022 12:10
To: Piotr Domagalski
Cc: user@flink.apache.org
Subject: [External] Re: Source vs SourceFunction and testing
This message is from an EXTERNAL SENDER - be CAUTIOUS, particularly with links
and attachments.
Glad to see you have resolved the issue!
If you want to learn more about the Source API, the Flink document [1] has a
detailed description about it. The original proposal FLIP-27 [2] is also a good
reference.
[1]
https://urldefense.proofpoint.com/v2/url?u=https-3A__nightlies.apache.org_flink_flink-2Ddocs-2Drelease-2D1.15_docs_dev_datastream_sources_=DwIFaQ=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU=Q6157zGhiDIuCzxlSpEZgTNbdEC4jbNL0iaPBqIxifg=UaxcpZWDroSZiLenzhGnRRQockpCt3Hg9jXU50aRVltFNinifOKvurHPTzdPL1da=lQGFDQJRG2BADprHFhkCefHCPTjDTh-OGIz4xFl-1W8=
[2]
https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_FLINK_FLIP-2D27-253A-2BRefactor-2BSource-2BInterface=DwIFaQ=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU=Q6157zGhiDIuCzxlSpEZgTNbdEC4jbNL0iaPBqIxifg=UaxcpZWDroSZiLenzhGnRRQockpCt3Hg9jXU50aRVltFNinifOKvurHPTzdPL1da=SqQEnABQt5ZGeX8rUZVEI8wyNDe2GlRNBHtZv5V3MIQ=
Cheers,
Qingsheng
> On May 25, 2022, at 17:54, Piotr Domagalski wrote:
>
> Thank you Qingsheng, this context helps a lot!
>
> And once again thank you all for being such a helpful community!
>
> P.S. I actually struggled for a bit trying to understand why my refactored
> solution which accepts DataStream<> wouldn't work ("no operators defined in
> the streaming topology"). Turns out, my assumption that I can call
> StreamExecutionEnvironment.getExecutionEnvironment() multiple times and get
> the same environment, was wrong. I had env.addSource and env.fromSource calls
> using one instance of the environment, but then called env.execute() on
> another instance :facepalm:
>
> On Wed, May 25, 2022 at 6:04 AM Qingsheng Ren wrote:
> Hi Piotr,
>
> I’d like to share my understanding about this. Source and SourceFunction are
> both interfaces to data sources. SourceFunction was designed and introduced
> earlier and as the project evolved, many shortcomings emerged. Therefore, the
> community re-designed the source interface and introduced the new Source API
> in FLIP-27 [1].
>
> Finally we will deprecate the SourceFunction and use Source as the only
> interface for all data sources, but considering the huge cost of migration
> you’ll see SourceFunction and Source co-exist for some time, like the
> ParallelTestSource you mentioned is still on Sour
