Re: Add me to slack

[Chengxuan Wang]

Hi Jing,
Could you also send the invite to me? wcxz...@gmail.com

Thanks,
Chengxuan

Zain Haider Nemati  于2022年6月5日周日 20:03写道：

> Hi Jing,
> Could you also send the invite to me?
> zain.hai...@retailo.co
>
>
> On Mon, 6 Jun 2022 at 7:04 AM Jing Ge  wrote:
>
>> Hi Xiao,
>>
>> Just done, please check. Thanks!
>>
>> Best regards,
>> Jing
>>
>>
>> On Mon, Jun 6, 2022 at 3:59 AM Xiao Ma  wrote:
>>
>>> Hi Jing,
>>>
>>> Could you please add me to the slack channel also?
>>>
>>> Thank you.
>>>
>>>
>>> Best,
>>> Mark Ma
>>>
>>> On Sun, Jun 5, 2022 at 9:57 PM Jing Ge  wrote:
>>>
 Hi Raghunadh,

 Just did, please check your email. Thanks!

 Best regards,
 Jing

 On Mon, Jun 6, 2022 at 3:51 AM Raghunadh Nittala <
 raghunitt...@gmail.com> wrote:

> Team, Kindly add me to the slack channel.
>
> Best Regards.
>
 --
>>> Xiao Ma
>>> Geotab
>>> Software Developer, Data Engineering | B.Sc, M.Sc
>>> Direct +1 (416) 836 - 3541
>>> Toll-free  +1 (877) 436 - 8221
>>> Visit   www.geotab.com
>>> Twitter | Facebook | YouTube | LinkedIn
>>>
>>

Best way to perform look up with historical data in Flink

[Surendra Lalwani]

Hi Team,

Since Flink does not support Stream-Batch Join, we need to perform a lookup
with some historical data, what could be the best way in Flink to do that.
I am aware of one such approach using Broadcasted stream, not sure if there
are any other proper solutions. Any help in this will be helpful.

Regards ,
Surendra Lalwani

-- 

IMPORTANT NOTICE: This e-mail, including any attachments, may contain 
confidential information and is intended only for the addressee(s) named 
above. If you are not the intended recipient(s), you should not 
disseminate, distribute, or copy this e-mail. Please notify the sender by 
reply e-mail immediately if you have received this e-mail in error and 
permanently delete all copies of the original message from your system. 
E-mail transmission cannot be guaranteed to be secure as it could be 
intercepted, corrupted, lost, destroyed, arrive late or incomplete, or 
contain viruses. Company accepts no liability for any damage or loss of 
confidential information caused by this email or due to any virus 
transmitted by this email or otherwise.

Re: flink-ml algorithms

[Natia Chachkhiani]

Thanks for the reply Zhipeng and Jing.
Running the OnlineKmeans with a fixed initial model removed the randomness!


On Sun, Jun 5, 2022 at 6:19 PM Zhipeng Zhang 
wrote:

> Hi Natia,
>
> As I understand, the processing order of onlineKmeans is the same the
> input data.
>
> Are you running OnlineKmeans with using one data point with random initial
> KmeansModel? Could you use a fixed initial model following [1] and try out?
>
> [1]
> https://github.com/apache/flink-ml/blob/239788f2b1f1f3a4e55ca112517980b598705a15/flink-ml-lib/src/test/java/org/apache/flink/ml/clustering/OnlineKMeansTest.java#L354
>
> Jing Ge  于2022年6月3日周五 17:04写道：
>
>> Hi,
>>
>> It seems like an evaluation with a small dataset. In this case, would you
>> like to share your data sample and code? In addition, have you tried KMeans
>> with the same dataset and got inconsistent results too?
>>
>> Best regards,
>> Jing
>>
>> On Fri, Jun 3, 2022 at 4:29 AM Natia Chachkhiani <
>> natia.chachkhia...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I am running OnlineKmeans from flink-ml repo on a small dataset. I've
>>> noticed that I don't get consistent results, assignments to clusters,
>>> across different runs. I have set both parallelism and globalBatchSize to 1.
>>> I am doing simple fit and transform on each data point ingested. Is the
>>> order of processing not guaranteed? Or am I missing something?
>>>
>>> Thanks,
>>> Natia
>>>
>>
>
> --
> best,
> Zhipeng
>
>

Re: Add me to slack

[Zain Haider Nemati]

Hi Jing,
Could you also send the invite to me?
zain.hai...@retailo.co


On Mon, 6 Jun 2022 at 7:04 AM Jing Ge  wrote:

> Hi Xiao,
>
> Just done, please check. Thanks!
>
> Best regards,
> Jing
>
>
> On Mon, Jun 6, 2022 at 3:59 AM Xiao Ma  wrote:
>
>> Hi Jing,
>>
>> Could you please add me to the slack channel also?
>>
>> Thank you.
>>
>>
>> Best,
>> Mark Ma
>>
>> On Sun, Jun 5, 2022 at 9:57 PM Jing Ge  wrote:
>>
>>> Hi Raghunadh,
>>>
>>> Just did, please check your email. Thanks!
>>>
>>> Best regards,
>>> Jing
>>>
>>> On Mon, Jun 6, 2022 at 3:51 AM Raghunadh Nittala 
>>> wrote:
>>>
 Team, Kindly add me to the slack channel.

 Best Regards.

>>> --
>> Xiao Ma
>> Geotab
>> Software Developer, Data Engineering | B.Sc, M.Sc
>> Direct +1 (416) 836 - 3541
>> Toll-free  +1 (877) 436 - 8221
>> Visit   www.geotab.com
>> Twitter | Facebook | YouTube | LinkedIn
>>
>

?????? [ANNOUNCE] Apache Flink Kubernetes Operator 1.0.0 released

[????]

----
??: 
   "dev"

<1996fan...@gmail.com;
:2022??6??5??(??) 5:19
??:"Yang Wang"https://flink.apache.org/news/2022/06/05/release-kubernetes-operator-1.0.0.html

 The release is available for download at:
 https://flink.apache.org/downloads.html

 Maven artifacts for Flink Kubernetes Operator can be found at:

 
https://search.maven.org/artifact/org.apache.flink/flink-kubernetes-operator

 Official Docker image for Flink Kubernetes Operator applications can be
 found at:
 https://hub.docker.com/r/apache/flink-kubernetes-operator

 The full release notes are available in Jira:

 
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522version=12351500

 We would like to thank all contributors of the Apache Flink community who
 made this release possible!

 Regards,
 Gyula  Yang

Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.0.0 released

[Aitozi]

Thanks Yang and Nice to see it happen.

Best,
Aitozi.

Yang Wang  于2022年6月5日周日 16:14写道：

> The Apache Flink community is very happy to announce the release of Apache
> Flink Kubernetes Operator 1.0.0.
>
> The Flink Kubernetes Operator allows users to manage their Apache Flink
> applications and their lifecycle through native k8s tooling like kubectl.
> This is the first production ready release and brings numerous
> improvements and new features to almost every aspect of the operator.
>
> Please check out the release blog post for an overview of the release:
>
> https://flink.apache.org/news/2022/06/05/release-kubernetes-operator-1.0.0.html
>
> The release is available for download at:
> https://flink.apache.org/downloads.html
>
> Maven artifacts for Flink Kubernetes Operator can be found at:
>
> https://search.maven.org/artifact/org.apache.flink/flink-kubernetes-operator
>
> Official Docker image for Flink Kubernetes Operator applications can be
> found at:
> https://hub.docker.com/r/apache/flink-kubernetes-operator
>
> The full release notes are available in Jira:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351500
>
> We would like to thank all contributors of the Apache Flink community who
> made this release possible!
>
> Regards,
> Gyula & Yang
>

Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.0.0 released

[Aitozi]

Thanks Yang and Nice to see it happen.

Best,
Aitozi.

Yang Wang  于2022年6月5日周日 16:14写道：

> The Apache Flink community is very happy to announce the release of Apache
> Flink Kubernetes Operator 1.0.0.
>
> The Flink Kubernetes Operator allows users to manage their Apache Flink
> applications and their lifecycle through native k8s tooling like kubectl.
> This is the first production ready release and brings numerous
> improvements and new features to almost every aspect of the operator.
>
> Please check out the release blog post for an overview of the release:
>
> https://flink.apache.org/news/2022/06/05/release-kubernetes-operator-1.0.0.html
>
> The release is available for download at:
> https://flink.apache.org/downloads.html
>
> Maven artifacts for Flink Kubernetes Operator can be found at:
>
> https://search.maven.org/artifact/org.apache.flink/flink-kubernetes-operator
>
> Official Docker image for Flink Kubernetes Operator applications can be
> found at:
> https://hub.docker.com/r/apache/flink-kubernetes-operator
>
> The full release notes are available in Jira:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351500
>
> We would like to thank all contributors of the Apache Flink community who
> made this release possible!
>
> Regards,
> Gyula & Yang
>

Re: Flink source Code Explanation

[sri hari kali charan Tummala]

I am getting a connection timed out error in firefox and google chrome can
you double-check whether the weblink is working or not?

Thanks
Sri




On Sun, Jun 5, 2022 at 7:01 PM Jing Ge  wrote:

> Hi Sri,
>
> Flink is very well documented. You can find it under e.g.
> https://nightlies.apache.org/flink/flink-docs-master/
>
> Best regards,
> Jing
>
> On Mon, Jun 6, 2022 at 3:39 AM sri hari kali charan Tummala <
> kali.tumm...@gmail.com> wrote:
>
>> Hi Flink Community,
>>
>> I want to go through flink source code in my free time is there a
>> document that I can go through that explains to me where to start? other
>> than Java doc is there anything else to start my reserve engineering.
>>
>> Thanks & Regards
>> Sri Tummala
>>
>>

-- 
Thanks & Regards
Sri Tummala

Re: Not able to see std output in console/.out files with table API

[Shengkai Fang]

Hi.

The TableResult.print() only prints the result to the client console. How
do you redirect the output to the .out file? Can you get the output without
redirection?

Best,
Shengkai

Xuyang  于2022年6月2日周四 21:17写道：

> Could you find that the input amount of the node `sink` is
> being accumulated in Flink UI? Is it possible that there are no data
> actually after join.
>
> If you have checked the above and everything seems ok, you can try again
> with  a connector named `print` as the sink table and check whether the out
> file has data.
>
>
> --
> Best！
> Xuyang
>
>
> At 2022-06-02 18:10:28, "Zain Haider Nemati" 
> wrote:
>
> Hi,
> We are using table apis to integrate and transform data sources and
> converting them to datastream. We want to see the data formatting and
> adding a .print() sink to the datastream but the .out files do not show any
> output.
> We do see records coming in from the metrics in flink UI though.
> Suggestions on where to look at for potential issues?
>
> code:
> tEnv.executeSql(“CREATE TABLE orders (\n” +
> ”id  BIGINT,\n” +
> ”customer_id BIGINT\n” +
> “) WITH (\n” +
> ”‘connector’ = ‘kafka’,\n” +
> ”‘topic’ = ‘orders’,\n” +
> ”‘properties.bootstrap.servers’ = ‘...’,\n” +
> ”‘scan.startup.mode’ = ‘earliest-offset’,\n” +
> ”‘format’= ‘json’\n” +
> “)”);
> Table result = tEnv.sqlQuery(“SELECT o.id AS order_id,\n” +
> ” dbo.batch_id AS batch_id,\n” +
> ” o.customer_id AS customer_id,\n” +
> ” dbo.delivery_priority AS delivery_priority\n” +
> ” FROM orders o\n” +
> ” INNER JOIN delivery_batch_orders dbo ON o.id =
> dbo.order_id\n”
> );
>
> tEnv.toAppendStream(result, StringValue.class).print();
> env.execute();
>
> Flink Version : 1.13.1
>
>

Re: Add me to slack

[Jing Ge]

Hi Xiao,

Just done, please check. Thanks!

Best regards,
Jing


On Mon, Jun 6, 2022 at 3:59 AM Xiao Ma  wrote:

> Hi Jing,
>
> Could you please add me to the slack channel also?
>
> Thank you.
>
>
> Best,
> Mark Ma
>
> On Sun, Jun 5, 2022 at 9:57 PM Jing Ge  wrote:
>
>> Hi Raghunadh,
>>
>> Just did, please check your email. Thanks!
>>
>> Best regards,
>> Jing
>>
>> On Mon, Jun 6, 2022 at 3:51 AM Raghunadh Nittala 
>> wrote:
>>
>>> Team, Kindly add me to the slack channel.
>>>
>>> Best Regards.
>>>
>> --
> Xiao Ma
> Geotab
> Software Developer, Data Engineering | B.Sc, M.Sc
> Direct +1 (416) 836 - 3541
> Toll-free  +1 (877) 436 - 8221
> Visit   www.geotab.com
> Twitter | Facebook | YouTube | LinkedIn
>

Re: Add me to slack

[Xiao Ma]

Received. Thank you very much

On Sun, Jun 5, 2022 at 9:59 PM Xiao Ma  wrote:

> Hi Jing,
>
> Could you please add me to the slack channel also?
>
> Thank you.
>
>
> Best,
> Mark Ma
>
> On Sun, Jun 5, 2022 at 9:57 PM Jing Ge  wrote:
>
>> Hi Raghunadh,
>>
>> Just did, please check your email. Thanks!
>>
>> Best regards,
>> Jing
>>
>> On Mon, Jun 6, 2022 at 3:51 AM Raghunadh Nittala 
>> wrote:
>>
>>> Team, Kindly add me to the slack channel.
>>>
>>> Best Regards.
>>>
>> --
> Xiao Ma
> Geotab
> Software Developer, Data Engineering | B.Sc, M.Sc
> Direct +1 (416) 836 - 3541
> Toll-free  +1 (877) 436 - 8221
> Visit   www.geotab.com
> Twitter | Facebook | YouTube | LinkedIn
>
-- 
Xiao Ma
Geotab
Software Developer, Data Engineering | B.Sc, M.Sc
Direct +1 (416) 836 - 3541
Toll-free  +1 (877) 436 - 8221
Visit   www.geotab.com
Twitter | Facebook | YouTube | LinkedIn

Re: Flink source Code Explanation

[Jing Ge]

Hi Sri,

Flink is very well documented. You can find it under e.g.
https://nightlies.apache.org/flink/flink-docs-master/

Best regards,
Jing

On Mon, Jun 6, 2022 at 3:39 AM sri hari kali charan Tummala <
kali.tumm...@gmail.com> wrote:

> Hi Flink Community,
>
> I want to go through flink source code in my free time is there a document
> that I can go through that explains to me where to start? other than Java
> doc is there anything else to start my reserve engineering.
>
> Thanks & Regards
> Sri Tummala
>
>

Re: Add me to slack

[Xiao Ma]

Hi Jing,

Could you please add me to the slack channel also?

Thank you.


Best,
Mark Ma

On Sun, Jun 5, 2022 at 9:57 PM Jing Ge  wrote:

> Hi Raghunadh,
>
> Just did, please check your email. Thanks!
>
> Best regards,
> Jing
>
> On Mon, Jun 6, 2022 at 3:51 AM Raghunadh Nittala 
> wrote:
>
>> Team, Kindly add me to the slack channel.
>>
>> Best Regards.
>>
> --
Xiao Ma
Geotab
Software Developer, Data Engineering | B.Sc, M.Sc
Direct +1 (416) 836 - 3541
Toll-free  +1 (877) 436 - 8221
Visit   www.geotab.com
Twitter | Facebook | YouTube | LinkedIn

Re: Need help to join Apache Flink community on.Slack

[Sucheth S]

I got it, thanks!
Regards,
Sucheth Shivakumar
website : https://sucheths.com
mobile : +1(650)-576-8050
San Mateo, United States


On Sun, Jun 5, 2022 at 6:48 PM Jing Ge  wrote:

> Hi Sucheth,
>
> Just invited you, please check. Thanks!
>
> Best Regards,
> Jing
>
> On Sun, Jun 5, 2022 at 6:06 PM Sucheth S  wrote:
>
>> Hello Jing,
>>
>> Can you please add me - suchet...@gmail.com
>>
>>
>> On Sun, Jun 5, 2022 at 9:02 AM sri hari kali charan Tummala <
>> kali.tumm...@gmail.com> wrote:
>>
>>> Hi Jing,
>>>
>>> Please add me kali.tumm...@gmail.com.
>>>
>>> Thanks
>>> Sri
>>>
>>> On Sat, Jun 4, 2022 at 4:47 PM Jing Ge  wrote:
>>>
 Hi Santhosh,

 just invited you. Please check your email. Looking forward to knowing
 your story! Thanks!

 To anyone else who wants to join, please send an email to
 user@flink.apache.org, you might have a better chance to get the
 invite. Thanks.

 Regards,
 Jing

 On Sat, Jun 4, 2022 at 10:37 PM santhosh venkat <
 santhoshvenkat1...@gmail.com> wrote:

> Hi,
>
> Can you please invite me to join the apache flink slack community
> channel. We have adopted apache flink and would like to participate in the
> community forum.
>
> Thank you.
>
> Regards
>

>>>
>>> --
>>> Thanks & Regards
>>> Sri Tummala
>>>
>>> --
>> Regards,
>> Sucheth Shivakumar
>> website: https://sucheths.com
>> Mobile : +1(650)-576-8050
>> San Mateo, United States
>>
>

Re: Add me to slack

[Jing Ge]

Hi Raghunadh,

Just did, please check your email. Thanks!

Best regards,
Jing

On Mon, Jun 6, 2022 at 3:51 AM Raghunadh Nittala 
wrote:

> Team, Kindly add me to the slack channel.
>
> Best Regards.
>

Add me to slack

[Raghunadh Nittala]

Team, Kindly add me to the slack channel.

Best Regards.

Re: Need help to join Apache Flink community on.Slack

[Jing Ge]

Hi Sucheth,

Just invited you, please check. Thanks!

Best Regards,
Jing

On Sun, Jun 5, 2022 at 6:06 PM Sucheth S  wrote:

> Hello Jing,
>
> Can you please add me - suchet...@gmail.com
>
>
> On Sun, Jun 5, 2022 at 9:02 AM sri hari kali charan Tummala <
> kali.tumm...@gmail.com> wrote:
>
>> Hi Jing,
>>
>> Please add me kali.tumm...@gmail.com.
>>
>> Thanks
>> Sri
>>
>> On Sat, Jun 4, 2022 at 4:47 PM Jing Ge  wrote:
>>
>>> Hi Santhosh,
>>>
>>> just invited you. Please check your email. Looking forward to knowing
>>> your story! Thanks!
>>>
>>> To anyone else who wants to join, please send an email to
>>> user@flink.apache.org, you might have a better chance to get the
>>> invite. Thanks.
>>>
>>> Regards,
>>> Jing
>>>
>>> On Sat, Jun 4, 2022 at 10:37 PM santhosh venkat <
>>> santhoshvenkat1...@gmail.com> wrote:
>>>
 Hi,

 Can you please invite me to join the apache flink slack community
 channel. We have adopted apache flink and would like to participate in the
 community forum.

 Thank you.

 Regards

>>>
>>
>> --
>> Thanks & Regards
>> Sri Tummala
>>
>> --
> Regards,
> Sucheth Shivakumar
> website: https://sucheths.com
> Mobile : +1(650)-576-8050
> San Mateo, United States
>

Re: Need help to join Apache Flink community on.Slack

[Jing Ge]

done, please check. Thanks

Best regards,
Jing

On Sun, Jun 5, 2022 at 6:05 PM Deepak Sharma  wrote:

> I need the invite as well .
> kdq...@gmail.com
>
> On Sun, 5 Jun 2022 at 9:32 PM, sri hari kali charan Tummala <
> kali.tumm...@gmail.com> wrote:
>
>> Hi Jing,
>>
>> Please add me kali.tumm...@gmail.com.
>>
>> Thanks
>> Sri
>>
>> On Sat, Jun 4, 2022 at 4:47 PM Jing Ge  wrote:
>>
>>> Hi Santhosh,
>>>
>>> just invited you. Please check your email. Looking forward to knowing
>>> your story! Thanks!
>>>
>>> To anyone else who wants to join, please send an email to
>>> user@flink.apache.org, you might have a better chance to get the
>>> invite. Thanks.
>>>
>>> Regards,
>>> Jing
>>>
>>> On Sat, Jun 4, 2022 at 10:37 PM santhosh venkat <
>>> santhoshvenkat1...@gmail.com> wrote:
>>>
 Hi,

 Can you please invite me to join the apache flink slack community
 channel. We have adopted apache flink and would like to participate in the
 community forum.

 Thank you.

 Regards

>>>
>>
>> --
>> Thanks & Regards
>> Sri Tummala
>>
>> --
> Thanks
> Deepak
> www.bigdatabig.com
> www.keosha.net
>

Re: Need help to join Apache Flink community on.Slack

[Jing Ge]

Hi Sri,

I have invited you, please check. Thanks!

Best regards,
Jing

On Sun, Jun 5, 2022 at 6:02 PM sri hari kali charan Tummala <
kali.tumm...@gmail.com> wrote:

> Hi Jing,
>
> Please add me kali.tumm...@gmail.com.
>
> Thanks
> Sri
>
> On Sat, Jun 4, 2022 at 4:47 PM Jing Ge  wrote:
>
>> Hi Santhosh,
>>
>> just invited you. Please check your email. Looking forward to knowing
>> your story! Thanks!
>>
>> To anyone else who wants to join, please send an email to
>> user@flink.apache.org, you might have a better chance to get the invite.
>> Thanks.
>>
>> Regards,
>> Jing
>>
>> On Sat, Jun 4, 2022 at 10:37 PM santhosh venkat <
>> santhoshvenkat1...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Can you please invite me to join the apache flink slack community
>>> channel. We have adopted apache flink and would like to participate in the
>>> community forum.
>>>
>>> Thank you.
>>>
>>> Regards
>>>
>>
>
> --
> Thanks & Regards
> Sri Tummala
>
>

Flink source Code Explanation

[sri hari kali charan Tummala]

Hi Flink Community,

I want to go through flink source code in my free time is there a document
that I can go through that explains to me where to start? other than Java
doc is there anything else to start my reserve engineering.

Thanks & Regards
Sri Tummala

Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.0.0 released

[rui fan]

Thanks Yang for driving the release, and thanks to
all contributors for making this release happen!

Best wishes
Rui Fan

On Sun, Jun 5, 2022 at 4:14 PM Yang Wang  wrote:

> The Apache Flink community is very happy to announce the release of Apache
> Flink Kubernetes Operator 1.0.0.
>
> The Flink Kubernetes Operator allows users to manage their Apache Flink
> applications and their lifecycle through native k8s tooling like kubectl.
> This is the first production ready release and brings numerous
> improvements and new features to almost every aspect of the operator.
>
> Please check out the release blog post for an overview of the release:
>
> https://flink.apache.org/news/2022/06/05/release-kubernetes-operator-1.0.0.html
>
> The release is available for download at:
> https://flink.apache.org/downloads.html
>
> Maven artifacts for Flink Kubernetes Operator can be found at:
>
> https://search.maven.org/artifact/org.apache.flink/flink-kubernetes-operator
>
> Official Docker image for Flink Kubernetes Operator applications can be
> found at:
> https://hub.docker.com/r/apache/flink-kubernetes-operator
>
> The full release notes are available in Jira:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351500
>
> We would like to thank all contributors of the Apache Flink community who
> made this release possible!
>
> Regards,
> Gyula & Yang
>

Re: flink-ml algorithms

[Zhipeng Zhang]

Hi Natia,

As I understand, the processing order of onlineKmeans is the same the input
data.

Are you running OnlineKmeans with using one data point with random initial
KmeansModel? Could you use a fixed initial model following [1] and try out?

[1]
https://github.com/apache/flink-ml/blob/239788f2b1f1f3a4e55ca112517980b598705a15/flink-ml-lib/src/test/java/org/apache/flink/ml/clustering/OnlineKMeansTest.java#L354

Jing Ge  于2022年6月3日周五 17:04写道：

> Hi,
>
> It seems like an evaluation with a small dataset. In this case, would you
> like to share your data sample and code? In addition, have you tried KMeans
> with the same dataset and got inconsistent results too?
>
> Best regards,
> Jing
>
> On Fri, Jun 3, 2022 at 4:29 AM Natia Chachkhiani <
> natia.chachkhia...@gmail.com> wrote:
>
>> Hi,
>>
>> I am running OnlineKmeans from flink-ml repo on a small dataset. I've
>> noticed that I don't get consistent results, assignments to clusters,
>> across different runs. I have set both parallelism and globalBatchSize to 1.
>> I am doing simple fit and transform on each data point ingested. Is the
>> order of processing not guaranteed? Or am I missing something?
>>
>> Thanks,
>> Natia
>>
>

-- 
best,
Zhipeng

Re: FW: Issue Facing While Using EmbeddedRocksDbCheckpointing FlinkVersion(1.15.0)

[Dian Fu]

None of the attachments are logs of the TaskManger. The TaskManger log
should be located in the directory
`E:\pythonProject16\lib\site-packages\pyflink\log`.

On Fri, Jun 3, 2022 at 8:41 PM harshit.varsh...@iktara.ai <
harshit.varsh...@iktara.ai> wrote:

>
>
>
>
> *From:* Shuiqiang Chen [mailto:acqua@gmail.com]
> *Sent:* Friday, June 3, 2022 2:11 PM
> *To:* harshit.varsh...@iktara.ai
> *Cc:* user
> *Subject:* Re: Issue Facing While Using EmbeddedRocksDbCheckpointing
> FlinkVersion(1.15.0)
>
>
>
> Hi,
>
>
>
> I guess that the traceback log you provided might not be the root cause of
> the failure, could you please provide the complete log of the Taskmanager？
>
>
>
> Best,
>
> Shuiqiang
>
>
>
> harshit.varsh...@iktara.ai  于2022年6月2日周四 22:04
> 写道：
>
> Dear Team,
>
>
>
> I am new to pyflink and request for your support in issue I am facing with
> Pyflink. I am using Pyflink version 1.15.0 & using reference code from
> pyflink reference code.
>
> The errors I am getting
>
> Traceback (most recent call last):
>
>   File
> "E:\pythonProject16\lib\site-packages\apache_beam\runners\worker\data_plane.py",
> line 470, in input_elements
>
> element = received.get(timeout=1)
>
>   File
> "C:\Users\Admin\AppData\Local\Programs\Python\Python38\lib\queue.py", line
> 178, in get
>
> raise Empty
>
> _queue.Empty
>
> RuntimeError: Channel closed prematurely.
>
> My code is:
>
> import json
>
> import os
>
> import time
>
> from datetime import datetime
>
>
>
> from pyflink.common import SimpleStringSchema,
> JsonRowDeserializationSchema, Types, JsonRowSerializationSchema
>
> from pyflink.datastream import StreamExecutionEnvironment, WindowFunction,
> HashMapStateBackend, CheckpointingMode, \
>
> FileSystemCheckpointStorage, KeyedProcessFunction, RuntimeContext,
> EmbeddedRocksDBStateBackend, RocksDBStateBackend, \
>
> ExternalizedCheckpointCleanup
>
> from pyflink.datastream.connectors import FlinkKafkaConsumer,
> FlinkKafkaProducer
>
> from pyflink.datastream.state import ValueStateDescriptor, StateTtlConfig,
> ListStateDescriptor
>
> from sklearn.preprocessing import LabelEncoder
>
> import pickle
>
> import pandas as pd
>
> from pyflink.common import Row
>
>
>
> import argparse
>
> from typing import Iterable
>
>
>
> from pyflink.datastream.connectors import FileSink, OutputFileConfig,
> RollingPolicy
>
>
>
> from pyflink.common import Types, WatermarkStrategy, Time, Encoder
>
> from pyflink.common.watermark_strategy import TimestampAssigner
>
> from pyflink.datastream import StreamExecutionEnvironment,
> ProcessWindowFunction
>
> from pyflink.datastream.window import TumblingEventTimeWindows,
> TimeWindow, TumblingProcessingTimeWindows
>
>
>
>
>
> class MyTimestampAssigner(TimestampAssigner):
>
> def extract_timestamp(self, value, record_timestamp) -> int:
>
> return int(value[0])
>
>
>
>
>
> class CountWindowFunction(WindowFunction[tuple, tuple, str, TimeWindow]):
>
> def apply(self, key: str, window: TimeWindow, inputs: Iterable[tuple]):
>
> # return [(key, result)]
>
> return [(key, len([e for e in inputs]))]
>
>
>
>
>
> class Storage(KeyedProcessFunction):
>
>
>
> def __init__(self):
>
> self.state = None
>
>
>
> def open(self, runtime_context: RuntimeContext):
>
> state_descriptor = ValueStateDescriptor("state", Types.FLOAT())
>
> state_ttl_config = StateTtlConfig \
>
> .new_builder(Time.days(7)) \
>
> .set_update_type(StateTtlConfig.UpdateType.OnReadAndWrite) \
>
> .disable_cleanup_in_background() \
>
> .build()
>
> state_descriptor.enable_time_to_live(state_ttl_config)
>
> self.state = runtime_context.get_state(state_descriptor)
>
>
>
> def process_element(self, value, ctx: 'KeyedProcessFunction.Context'):
>
> # retrieve the current count
>
> current = self.state.value()
>
> if current is None:
>
> current = 0
>
> current = value[1]
>
> self.state.update(current)
>
>
>
> yield current,time.time()
>
>
>
>
>
> def write_to_kafka():
>
> env = StreamExecutionEnvironment.get_execution_environment()
>
> env.set_parallelism(1)
>
> env.enable_checkpointing(1)
>
> env.get_checkpoint_config().set_min_pause_between_checkpoints(5000)
>
>
> env.set_state_backend(EmbeddedRocksDBStateBackend(enable_incremental_checkpointing=True))
>
>
> env.get_checkpoint_config().set_checkpointing_mode(CheckpointingMode.EXACTLY_ONCE)
>
> env.get_checkpoint_config().enable_unaligned_checkpoints()
>
> env.get_checkpoint_config().set_max_concurrent_checkpoints(1)
>
> env.get_checkpoint_config().set_checkpoint_timeout(5)
>
>
> env.get_checkpoint_config().enable_externalized_checkpoints(ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION)
>
> check = os.path.join(os.path.abspath(os.path.dirname(__file__)),
>
>  'checkpoint-dir11')
>
>
> 

Re: Need help to join Apache Flink community on.Slack

[Sucheth S]

Hello Jing,

Can you please add me - suchet...@gmail.com


On Sun, Jun 5, 2022 at 9:02 AM sri hari kali charan Tummala <
kali.tumm...@gmail.com> wrote:

> Hi Jing,
>
> Please add me kali.tumm...@gmail.com.
>
> Thanks
> Sri
>
> On Sat, Jun 4, 2022 at 4:47 PM Jing Ge  wrote:
>
>> Hi Santhosh,
>>
>> just invited you. Please check your email. Looking forward to knowing
>> your story! Thanks!
>>
>> To anyone else who wants to join, please send an email to
>> user@flink.apache.org, you might have a better chance to get the invite.
>> Thanks.
>>
>> Regards,
>> Jing
>>
>> On Sat, Jun 4, 2022 at 10:37 PM santhosh venkat <
>> santhoshvenkat1...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Can you please invite me to join the apache flink slack community
>>> channel. We have adopted apache flink and would like to participate in the
>>> community forum.
>>>
>>> Thank you.
>>>
>>> Regards
>>>
>>
>
> --
> Thanks & Regards
> Sri Tummala
>
> --
Regards,
Sucheth Shivakumar
website: https://sucheths.com
Mobile : +1(650)-576-8050
San Mateo, United States

Re: Need help to join Apache Flink community on.Slack

[Deepak Sharma]

I need the invite as well .
kdq...@gmail.com

On Sun, 5 Jun 2022 at 9:32 PM, sri hari kali charan Tummala <
kali.tumm...@gmail.com> wrote:

> Hi Jing,
>
> Please add me kali.tumm...@gmail.com.
>
> Thanks
> Sri
>
> On Sat, Jun 4, 2022 at 4:47 PM Jing Ge  wrote:
>
>> Hi Santhosh,
>>
>> just invited you. Please check your email. Looking forward to knowing
>> your story! Thanks!
>>
>> To anyone else who wants to join, please send an email to
>> user@flink.apache.org, you might have a better chance to get the invite.
>> Thanks.
>>
>> Regards,
>> Jing
>>
>> On Sat, Jun 4, 2022 at 10:37 PM santhosh venkat <
>> santhoshvenkat1...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Can you please invite me to join the apache flink slack community
>>> channel. We have adopted apache flink and would like to participate in the
>>> community forum.
>>>
>>> Thank you.
>>>
>>> Regards
>>>
>>
>
> --
> Thanks & Regards
> Sri Tummala
>
> --
Thanks
Deepak
www.bigdatabig.com
www.keosha.net

Re: Need help to join Apache Flink community on.Slack

[sri hari kali charan Tummala]

Hi Jing,

Please add me kali.tumm...@gmail.com.

Thanks
Sri

On Sat, Jun 4, 2022 at 4:47 PM Jing Ge  wrote:

> Hi Santhosh,
>
> just invited you. Please check your email. Looking forward to knowing your
> story! Thanks!
>
> To anyone else who wants to join, please send an email to
> user@flink.apache.org, you might have a better chance to get the invite.
> Thanks.
>
> Regards,
> Jing
>
> On Sat, Jun 4, 2022 at 10:37 PM santhosh venkat <
> santhoshvenkat1...@gmail.com> wrote:
>
>> Hi,
>>
>> Can you please invite me to join the apache flink slack community
>> channel. We have adopted apache flink and would like to participate in the
>> community forum.
>>
>> Thank you.
>>
>> Regards
>>
>

-- 
Thanks & Regards
Sri Tummala

filesink part files roll over

[Sucheth S]

Hi,

Can someone please help me with this please -
https://stackoverflow.com/q/72496963/9125940 ?

Regards,
Sucheth Shivakumar
website : https://sucheths.com
mobile : +1(650)-576-8050
San Mateo, United States

Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.0.0 released

[Jing Ge]

Amazing! Thanks Yang for driving this! Thanks all for your effort!

Best regards,
Jing

On Sun, Jun 5, 2022 at 11:30 AM tison  wrote:

> Congrats! Thank you all for making this release happen.
>
> Best,
> tison.
>
>
> rui fan <1996fan...@gmail.com> 于2022年6月5日周日 17:19写道：
>
>> Thanks Yang for driving the release, and thanks to
>> all contributors for making this release happen!
>>
>> Best wishes
>> Rui Fan
>>
>> On Sun, Jun 5, 2022 at 4:14 PM Yang Wang  wrote:
>>
>> > The Apache Flink community is very happy to announce the release of
>> Apache
>> > Flink Kubernetes Operator 1.0.0.
>> >
>> > The Flink Kubernetes Operator allows users to manage their Apache Flink
>> > applications and their lifecycle through native k8s tooling like
>> kubectl.
>> > This is the first production ready release and brings numerous
>> > improvements and new features to almost every aspect of the operator.
>> >
>> > Please check out the release blog post for an overview of the release:
>> >
>> >
>> https://flink.apache.org/news/2022/06/05/release-kubernetes-operator-1.0.0.html
>> >
>> > The release is available for download at:
>> > https://flink.apache.org/downloads.html
>> >
>> > Maven artifacts for Flink Kubernetes Operator can be found at:
>> >
>> >
>> https://search.maven.org/artifact/org.apache.flink/flink-kubernetes-operator
>> >
>> > Official Docker image for Flink Kubernetes Operator applications can be
>> > found at:
>> > https://hub.docker.com/r/apache/flink-kubernetes-operator
>> >
>> > The full release notes are available in Jira:
>> >
>> >
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351500
>> >
>> > We would like to thank all contributors of the Apache Flink community
>> who
>> > made this release possible!
>> >
>> > Regards,
>> > Gyula & Yang
>> >
>>
>

Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.0.0 released

[Jing Ge]

Amazing! Thanks Yang for driving this! Thanks all for your effort!

Best regards,
Jing

On Sun, Jun 5, 2022 at 11:30 AM tison  wrote:

> Congrats! Thank you all for making this release happen.
>
> Best,
> tison.
>
>
> rui fan <1996fan...@gmail.com> 于2022年6月5日周日 17:19写道：
>
>> Thanks Yang for driving the release, and thanks to
>> all contributors for making this release happen!
>>
>> Best wishes
>> Rui Fan
>>
>> On Sun, Jun 5, 2022 at 4:14 PM Yang Wang  wrote:
>>
>> > The Apache Flink community is very happy to announce the release of
>> Apache
>> > Flink Kubernetes Operator 1.0.0.
>> >
>> > The Flink Kubernetes Operator allows users to manage their Apache Flink
>> > applications and their lifecycle through native k8s tooling like
>> kubectl.
>> > This is the first production ready release and brings numerous
>> > improvements and new features to almost every aspect of the operator.
>> >
>> > Please check out the release blog post for an overview of the release:
>> >
>> >
>> https://flink.apache.org/news/2022/06/05/release-kubernetes-operator-1.0.0.html
>> >
>> > The release is available for download at:
>> > https://flink.apache.org/downloads.html
>> >
>> > Maven artifacts for Flink Kubernetes Operator can be found at:
>> >
>> >
>> https://search.maven.org/artifact/org.apache.flink/flink-kubernetes-operator
>> >
>> > Official Docker image for Flink Kubernetes Operator applications can be
>> > found at:
>> > https://hub.docker.com/r/apache/flink-kubernetes-operator
>> >
>> > The full release notes are available in Jira:
>> >
>> >
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351500
>> >
>> > We would like to thank all contributors of the Apache Flink community
>> who
>> > made this release possible!
>> >
>> > Regards,
>> > Gyula & Yang
>> >
>>
>

Re: slack invite link

[Jing Ge]

Hi,

Invites have been sent. Please check your emails. Thanks!

Regards,
JIng

On Sun, Jun 5, 2022 at 12:38 PM Jay Ghiya  wrote:

> Request community to share invite link for me at ghiya6...@gmail.com and
> jay.gh...@ge.com
>

Re: slack invite link

[Jay Ghiya]

Request community to share invite link for me at ghiya6...@gmail.com and
jay.gh...@ge.com

Re: Questions regarding classpath loading order in YarnClusterDescriptor

[Geng Biao]

Hi Leon,
You are welcome. ‘Each plugin is loaded through its own classloader’(see 
doc)
 and as a result, they are not added to the flink system classpath. If I 
understand correctly, you do not need to do extra work if you set them 
correctly in the flink-conf.yaml.
If you have some dependency jars for a specific flink job, since 1.15.0, you 
can put those jars under ‘usrlib’ (if the dir does not exist, you can create it 
by yourself) which will be shipped automatically as well.

Best,
Biao Geng

From: Leon Xu 
Date: Sunday, June 5, 2022 at 4:04 PM
To: Biao Geng 
Cc: user 
Subject: Re: Questions regarding classpath loading order in 
YarnClusterDescriptor

Hi Biao,

I really appreciate your thorough answers. And yes for now I took the 
workaround by manipulating the directory names.
To follow up with one more question if you don't mind:
What is the recommended way of managing plugins in YarnClusterDescriptor? 
Currently I am placing the plugins (e.g. flink-s3-fs-hadoop) under the system 
jars setting, which works. But I am also seeing this comment in the 
code
 so I am a bit confused.


Thanks
Leon

On Sat, Jun 4, 2022 at 11:03 PM Biao Geng 
mailto:biaoge...@gmail.com>> wrote:
Hi Leon,

For your question1, in the classpath, there are 2 types of jars: user jars and 
flink system jars(i.e. jars in flink/lib). System jars are sorted 
alphabetically. For user jars, there are 3 choices to add user jars in the 
final classpath: ORDER, FIRST, LAST(See the 
doc
 for more details). To my best knowledge, there is no way to pass a sort 
function for this for now. One workaround is managing your jar paths. You can 
put the jar that you want to load first in an alphabetical smaller 
directory(e.g a-flink/user-jar).
For your question2, flink-dist.jar is always at the end of the system jars. 
Depending on your choices of adding user jars, it is not always at the end of 
the final generated classpath. flink-dist.jar is special and mandatory as we 
need it to launch java process to run ClusterEntrypoint on the cluster side. 
Other jars in the flink/lib can somehow be compromised.

I have met a similar problem as well. My previous woraround is managing the 
directory name, which is not so elegant. It can be useful to add the ability to 
customize loading orders of jars in classpath while it is also important to 
package the jars more carefully to avoid the conflicts.

Best,
Biao Geng


Leon Xu mailto:l...@attentivemobile.com>> 
于2022年6月5日周日 03:21写道：
Hi Flink Community,

We are building on top of  org.apache.flink.yarn.YarnClusterDescriptor to 
submit a flink application from Java code to YARN cluster, in the application 
mode. We are setting the classpath as the value of the yarn.provided.lib.dirs 
property under the yarn configuration.

By playing with the YarnClusterDescriptor code I have two questions that I hope 
to get some answers:
1. YarnClusterDescriptor seems to force the classpath loading in alphabetical 
order. See code 
here.
 Is there any specific reason for doing that? If I'd like to enforce my own 
order is it possible now?
2. Looks like the flink-dist.jar is treated separately from the other classpath 
classes. In the YarnApplicationFileUploader class, the 
registerMultipleLocalResources method will skip the jar if it is a dist jar. 
See the code 
here.
 With the current behavior it seems it will always place the flink-dist.jar at 
the end of the classpath. Is there any reason that Flink wants to treat the 
flink-dist.jar separately from other jars?

In our classpath loading we are hoping to enforce certain order because 
different jars may contain the same dependent library but with different 
versions. We hope to force the order so that we can load the correct library 
version as we want.


Thanks
Leon

Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.0.0 released

[tison]

Congrats! Thank you all for making this release happen.

Best,
tison.


rui fan <1996fan...@gmail.com> 于2022年6月5日周日 17:19写道：

> Thanks Yang for driving the release, and thanks to
> all contributors for making this release happen!
>
> Best wishes
> Rui Fan
>
> On Sun, Jun 5, 2022 at 4:14 PM Yang Wang  wrote:
>
> > The Apache Flink community is very happy to announce the release of
> Apache
> > Flink Kubernetes Operator 1.0.0.
> >
> > The Flink Kubernetes Operator allows users to manage their Apache Flink
> > applications and their lifecycle through native k8s tooling like kubectl.
> > This is the first production ready release and brings numerous
> > improvements and new features to almost every aspect of the operator.
> >
> > Please check out the release blog post for an overview of the release:
> >
> >
> https://flink.apache.org/news/2022/06/05/release-kubernetes-operator-1.0.0.html
> >
> > The release is available for download at:
> > https://flink.apache.org/downloads.html
> >
> > Maven artifacts for Flink Kubernetes Operator can be found at:
> >
> >
> https://search.maven.org/artifact/org.apache.flink/flink-kubernetes-operator
> >
> > Official Docker image for Flink Kubernetes Operator applications can be
> > found at:
> > https://hub.docker.com/r/apache/flink-kubernetes-operator
> >
> > The full release notes are available in Jira:
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351500
> >
> > We would like to thank all contributors of the Apache Flink community who
> > made this release possible!
> >
> > Regards,
> > Gyula & Yang
> >
>

Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.0.0 released

[tison]

Congrats! Thank you all for making this release happen.

Best,
tison.


rui fan <1996fan...@gmail.com> 于2022年6月5日周日 17:19写道：

> Thanks Yang for driving the release, and thanks to
> all contributors for making this release happen!
>
> Best wishes
> Rui Fan
>
> On Sun, Jun 5, 2022 at 4:14 PM Yang Wang  wrote:
>
> > The Apache Flink community is very happy to announce the release of
> Apache
> > Flink Kubernetes Operator 1.0.0.
> >
> > The Flink Kubernetes Operator allows users to manage their Apache Flink
> > applications and their lifecycle through native k8s tooling like kubectl.
> > This is the first production ready release and brings numerous
> > improvements and new features to almost every aspect of the operator.
> >
> > Please check out the release blog post for an overview of the release:
> >
> >
> https://flink.apache.org/news/2022/06/05/release-kubernetes-operator-1.0.0.html
> >
> > The release is available for download at:
> > https://flink.apache.org/downloads.html
> >
> > Maven artifacts for Flink Kubernetes Operator can be found at:
> >
> >
> https://search.maven.org/artifact/org.apache.flink/flink-kubernetes-operator
> >
> > Official Docker image for Flink Kubernetes Operator applications can be
> > found at:
> > https://hub.docker.com/r/apache/flink-kubernetes-operator
> >
> > The full release notes are available in Jira:
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351500
> >
> > We would like to thank all contributors of the Apache Flink community who
> > made this release possible!
> >
> > Regards,
> > Gyula & Yang
> >
>

Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.0.0 released

[rui fan]

Thanks Yang for driving the release, and thanks to
all contributors for making this release happen!

Best wishes
Rui Fan

On Sun, Jun 5, 2022 at 4:14 PM Yang Wang  wrote:

> The Apache Flink community is very happy to announce the release of Apache
> Flink Kubernetes Operator 1.0.0.
>
> The Flink Kubernetes Operator allows users to manage their Apache Flink
> applications and their lifecycle through native k8s tooling like kubectl.
> This is the first production ready release and brings numerous
> improvements and new features to almost every aspect of the operator.
>
> Please check out the release blog post for an overview of the release:
>
> https://flink.apache.org/news/2022/06/05/release-kubernetes-operator-1.0.0.html
>
> The release is available for download at:
> https://flink.apache.org/downloads.html
>
> Maven artifacts for Flink Kubernetes Operator can be found at:
>
> https://search.maven.org/artifact/org.apache.flink/flink-kubernetes-operator
>
> Official Docker image for Flink Kubernetes Operator applications can be
> found at:
> https://hub.docker.com/r/apache/flink-kubernetes-operator
>
> The full release notes are available in Jira:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351500
>
> We would like to thank all contributors of the Apache Flink community who
> made this release possible!
>
> Regards,
> Gyula & Yang
>

Re: slack invite link

[yuxia]

I have send the invitation to the email address shmily...@gmail.com. Please 
check you email! Look forward your joining. 

Best regards, 
Yuxia 


发件人: "shmily"  
收件人: "User"  
发送时间: 星期日, 2022年 6 月 05日 下午 4:55:11 
主题: slack invite link 

hi, 
can someone please send me a slack invite link, the one provided by the 
community has expired~ 

many thanks! 

slack invite link

[shmily]

hi,

can someone please send me a slack invite link, the one provided by the
community has expired~

many thanks!

[ANNOUNCE] Apache Flink Kubernetes Operator 1.0.0 released

[Yang Wang]

The Apache Flink community is very happy to announce the release of Apache
Flink Kubernetes Operator 1.0.0.

The Flink Kubernetes Operator allows users to manage their Apache Flink
applications and their lifecycle through native k8s tooling like kubectl.
This is the first production ready release and brings numerous improvements
and new features to almost every aspect of the operator.

Please check out the release blog post for an overview of the release:
https://flink.apache.org/news/2022/06/05/release-kubernetes-operator-1.0.0.html

The release is available for download at:
https://flink.apache.org/downloads.html

Maven artifacts for Flink Kubernetes Operator can be found at:
https://search.maven.org/artifact/org.apache.flink/flink-kubernetes-operator

Official Docker image for Flink Kubernetes Operator applications can be
found at:
https://hub.docker.com/r/apache/flink-kubernetes-operator

The full release notes are available in Jira:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351500

We would like to thank all contributors of the Apache Flink community who
made this release possible!

Regards,
Gyula & Yang

[ANNOUNCE] Apache Flink Kubernetes Operator 1.0.0 released

[Yang Wang]

The Apache Flink community is very happy to announce the release of Apache
Flink Kubernetes Operator 1.0.0.

The Flink Kubernetes Operator allows users to manage their Apache Flink
applications and their lifecycle through native k8s tooling like kubectl.
This is the first production ready release and brings numerous improvements
and new features to almost every aspect of the operator.

Please check out the release blog post for an overview of the release:
https://flink.apache.org/news/2022/06/05/release-kubernetes-operator-1.0.0.html

The release is available for download at:
https://flink.apache.org/downloads.html

Maven artifacts for Flink Kubernetes Operator can be found at:
https://search.maven.org/artifact/org.apache.flink/flink-kubernetes-operator

Official Docker image for Flink Kubernetes Operator applications can be
found at:
https://hub.docker.com/r/apache/flink-kubernetes-operator

The full release notes are available in Jira:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351500

We would like to thank all contributors of the Apache Flink community who
made this release possible!

Regards,
Gyula & Yang

Re: Questions regarding classpath loading order in YarnClusterDescriptor

[Leon Xu]

Hi Biao,

I really appreciate your thorough answers. And yes for now I took the
workaround by manipulating the directory names.
To follow up with one more question if you don't mind:
What is the recommended way of managing plugins in YarnClusterDescriptor?
Currently I am placing the plugins (e.g. flink-s3-fs-hadoop) under the
system jars setting, which works. But I am also seeing this comment in the
code

so I am a bit confused.


Thanks
Leon

On Sat, Jun 4, 2022 at 11:03 PM Biao Geng  wrote:

> Hi Leon,
>
> For your question1, in the classpath, there are 2 types of jars: user jars
> and flink system jars(i.e. jars in flink/lib). System jars are sorted
> alphabetically. For user jars, there are 3 choices to add user jars in the
> final classpath: ORDER, FIRST, LAST(See the doc
> 
> for more details). To my best knowledge, there is no way to pass a sort
> function for this for now. One workaround is managing your jar paths. You
> can put the jar that you want to load first in an alphabetical smaller
> directory(e.g a-flink/user-jar).
> For your question2, flink-dist.jar is always at the end of the system
> jars. Depending on your choices of adding user jars, it is not always at
> the end of the final generated classpath. flink-dist.jar is special and
> mandatory as we need it to launch java process to run ClusterEntrypoint on
> the cluster side. Other jars in the flink/lib can somehow be compromised.
>
> I have met a similar problem as well. My previous woraround is managing
> the directory name, which is not so elegant. It can be useful to add the
> ability to customize loading orders of jars in classpath while it is also
> important to package the jars more carefully to avoid the conflicts.
>
> Best,
> Biao Geng
>
>
> Leon Xu  于2022年6月5日周日 03:21写道：
>
>> Hi Flink Community,
>>
>> We are building on top of  *org.apache.flink.yarn.YarnClusterDescriptor *to
>> submit a flink application from Java code to YARN cluster, in the
>> application mode. We are setting the classpath as the value of *the 
>> yarn.provided.lib.dirs
>> *property under the yarn configuration.
>>
>> By playing with the YarnClusterDescriptor code I have two questions that
>> I hope to get some answers:
>> 1. YarnClusterDescriptor seems to force the classpath loading in
>> alphabetical order. See code here
>> .
>> Is there any specific reason for doing that? If I'd like to enforce my own
>> order is it possible now?
>> 2. Looks like the *flink-dist.jar* is treated separately from the other
>> classpath classes. In the *YarnApplicationFileUploader* class,
>> the registerMultipleLocalResources method will skip the jar if it is a dist
>> jar. See the code here
>> .
>> With the current behavior it seems it will always place the flink-dist.jar
>> at the end of the classpath. Is there any reason that Flink wants to treat
>> the *flink-dist.jar* separately from other jars?
>>
>> In our classpath loading we are hoping to enforce certain order because
>> different jars may contain the same dependent library but with different
>> versions. We hope to force the order so that we can load the correct
>> library version as we want.
>>
>>
>> Thanks
>> Leon
>>
>

Re: Questions regarding classpath loading order in YarnClusterDescriptor

[Biao Geng]

Hi Leon,

For your question1, in the classpath, there are 2 types of jars: user jars
and flink system jars(i.e. jars in flink/lib). System jars are sorted
alphabetically. For user jars, there are 3 choices to add user jars in the
final classpath: ORDER, FIRST, LAST(See the doc

for more details). To my best knowledge, there is no way to pass a sort
function for this for now. One workaround is managing your jar paths. You
can put the jar that you want to load first in an alphabetical smaller
directory(e.g a-flink/user-jar).
For your question2, flink-dist.jar is always at the end of the system jars.
Depending on your choices of adding user jars, it is not always at the end
of the final generated classpath. flink-dist.jar is special and mandatory
as we need it to launch java process to run ClusterEntrypoint on the
cluster side. Other jars in the flink/lib can somehow be compromised.

I have met a similar problem as well. My previous woraround is managing the
directory name, which is not so elegant. It can be useful to add the
ability to customize loading orders of jars in classpath while it is also
important to package the jars more carefully to avoid the conflicts.

Best,
Biao Geng


Leon Xu  于2022年6月5日周日 03:21写道：

> Hi Flink Community,
>
> We are building on top of  *org.apache.flink.yarn.YarnClusterDescriptor *to
> submit a flink application from Java code to YARN cluster, in the
> application mode. We are setting the classpath as the value of *the 
> yarn.provided.lib.dirs
> *property under the yarn configuration.
>
> By playing with the YarnClusterDescriptor code I have two questions that I
> hope to get some answers:
> 1. YarnClusterDescriptor seems to force the classpath loading in
> alphabetical order. See code here
> .
> Is there any specific reason for doing that? If I'd like to enforce my own
> order is it possible now?
> 2. Looks like the *flink-dist.jar* is treated separately from the other
> classpath classes. In the *YarnApplicationFileUploader* class,
> the registerMultipleLocalResources method will skip the jar if it is a dist
> jar. See the code here
> .
> With the current behavior it seems it will always place the flink-dist.jar
> at the end of the classpath. Is there any reason that Flink wants to treat
> the *flink-dist.jar* separately from other jars?
>
> In our classpath loading we are hoping to enforce certain order because
> different jars may contain the same dependent library but with different
> versions. We hope to force the order so that we can load the correct
> library version as we want.
>
>
> Thanks
> Leon
>