Re: Need help to join Apache Flink community on.Slack

2022-06-04 Thread Jing Ge 
Hi Santhosh,

just invited you. Please check your email. Looking forward to knowing your
story! Thanks!

To anyone else who wants to join, please send an email to
user@flink.apache.org, you might have a better chance to get the invite.
Thanks.

Regards,
Jing

On Sat, Jun 4, 2022 at 10:37 PM santhosh venkat <
santhoshvenkat1...@gmail.com> wrote:

> Hi,
>
> Can you please invite me to join the apache flink slack community channel.
> We have adopted apache flink and would like to participate in the
> community forum.
>
> Thank you.
>
> Regards
>

Questions regarding classpath loading order in YarnClusterDescriptor

2022-06-04 Thread Leon Xu 
Hi Flink Community,

We are building on top of  *org.apache.flink.yarn.YarnClusterDescriptor *to
submit a flink application from Java code to YARN cluster, in the
application mode. We are setting the classpath as the value of *the
yarn.provided.lib.dirs
*property under the yarn configuration.

By playing with the YarnClusterDescriptor code I have two questions that I
hope to get some answers:
1. YarnClusterDescriptor seems to force the classpath loading in
alphabetical order. See code here
.
Is there any specific reason for doing that? If I'd like to enforce my own
order is it possible now?
2. Looks like the *flink-dist.jar* is treated separately from the other
classpath classes. In the *YarnApplicationFileUploader* class,
the registerMultipleLocalResources method will skip the jar if it is a dist
jar. See the code here
.
With the current behavior it seems it will always place the flink-dist.jar
at the end of the classpath. Is there any reason that Flink wants to treat
the *flink-dist.jar* separately from other jars?

In our classpath loading we are hoping to enforce certain order because
different jars may contain the same dependent library but with different
versions. We hope to force the order so that we can load the correct
library version as we want.


Thanks
Leon

Re: [ANNOUNCE] Welcome to join the Apache Flink community on Slack

2022-06-04 Thread Jing Ge 
Hi Xingtong,

While inviting new members, there are two options: "From another company"
vs "Your coworker". In this case, we should always choose "Your coworker"
to add new members to the Apache Flink workspace, right?

Best regards,
Jing

On Fri, Jun 3, 2022 at 1:10 PM Yuan Mei  wrote:

> Thanks, Xintong and Jark the great effort driving this, and everyone for
> making this possible.
>
> I've also Twittered this announcement on our Apache Flink Twitter account.
>
> Best
>
> Yuan
>
>
>
> On Fri, Jun 3, 2022 at 12:54 AM Jing Ge  wrote:
>
>> Thanks everyone for your effort!
>>
>> Best regards,
>> Jing
>>
>> On Thu, Jun 2, 2022 at 4:17 PM Martijn Visser 
>> wrote:
>>
>>> Thanks everyone for joining! It's good to see so many have joined in
>>> such a short time already. I've just refreshed the link which you can
>>> always find on the project website [1]
>>>
>>> Best regards, Martijn
>>>
>>> [1] https://flink.apache.org/community.html#slack
>>>
>>> Op do 2 jun. 2022 om 11:42 schreef Jingsong Li :
>>>
 Thanks Xingtong, Jark, Martijn and Robert for making this possible!

 Best,
 Jingsong


 On Thu, Jun 2, 2022 at 5:32 PM Jark Wu  wrote:

> Thank Xingtong for making this possible!
>
> Cheers,
> Jark Wu
>
> On Thu, 2 Jun 2022 at 15:31, Xintong Song 
> wrote:
>
> > Hi everyone,
> >
> > I'm very happy to announce that the Apache Flink community has
> created a
> > dedicated Slack workspace [1]. Welcome to join us on Slack.
> >
> > ## Join the Slack workspace
> >
> > You can join the Slack workspace by either of the following two ways:
> > 1. Click the invitation link posted on the project website [2].
> > 2. Ask anyone who already joined the Slack workspace to invite you.
> >
> > We recommend 2), if available. Due to Slack limitations, the
> invitation
> > link in 1) expires and needs manual updates after every 100 invites.
> If it
> > is expired, please reach out to the dev / user mailing lists.
> >
> > ## Community rules
> >
> > When using the community Slack workspace, please follow these
> community
> > rules:
> > * *Be respectful* - This is the most important rule!
> > * All important decisions and conclusions *must be reflected back to
> the
> > mailing lists*. "If it didn’t happen on a mailing list, it didn’t
> happen."
> > - The Apache Mottos [3]
> > * Use *Slack threads* to keep parallel conversations from
> overwhelming a
> > channel.
> > * Please *do not direct message* people for troubleshooting, Jira
> assigning
> > and PR review. These should be picked-up voluntarily.
> >
> >
> > ## Maintenance
> >
> >
> > Committers can refer to this wiki page [4] for information needed for
> > maintaining the Slack workspace.
> >
> >
> > Thanks Jark, Martijn and Robert for helping setting up the Slack
> workspace.
> >
> >
> > Best,
> >
> > Xintong
> >
> >
> > [1] https://apache-flink.slack.com/
> >
> > [2] https://flink.apache.org/community.html#slack
> >
> > [3] http://theapacheway.com/on-list/
> >
> > [4]
> https://cwiki.apache.org/confluence/display/FLINK/Slack+Management
> >
>

Re: Cannot cast GoogleHadoopFileSystem to hadoop.fs.FileSystem to list file in Flink 1.15

2022-06-04 Thread 陳昌倬 
On Thu, Jun 02, 2022 at 06:23:20PM +0800, Qingsheng Ren wrote:
> Thanks for the input ChangZhuo.
> 
> Could you check if the configuration "classloader.resolve-order” is
> set to “parent-first” in your Flink 1.14 cluster? I didn’t notice any
> changes related to the user code classloader in Flink 1.15. If my
> assumption is correct, you package the gcs-connector into your job JAR
> but the Hadoop FS dependencies are not included, so
> org.apache.hadoop.fs.FileSystem is loaded by app classloader from
> flink-s3-fs-hadoop.jar under the lib of Flink, but
> GoogleHadoopFileSystem is loaded by user code classloader from job
> JAR. Setting the resolve order to "parent-first" could bypass the
> issue [1] so I assume you have this config in 1.14 but not in 1.15.
> Please forgive me if I understand incorrectly!

No, we do not config classloader.resolve-order in both 1.14, and 1.15
setup. We will check if "parent-first" can solve the problem, thanks for
the advise.


Also, in 1.14, we include the following jars into /opt/flink/lib to
support GCS:

* flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar
* gcs-connector-hadoop3-2.2.2-shaded.jar

In 1.15, we add flink-gs-fs-hadoop-1.15.0.jar to /opt/flink/lib to
support GCS. Maybe this different causes problem?


-- 
ChangZhuo Chen (陳昌倬) czchen@{czchen,debian}.org
http://czchen.info/
Key fingerprint = BA04 346D C2E1 FE63 C790  8793 CC65 B0CD EC27 5D5B


signature.asc
Description: PGP signature