date:20220501

Re: How to debug Metaspace exception?

2022-05-01 Thread Chesnay Schepler

And you do should make sure that it is set for both processes!

On 02/05/2022 08:43, Chesnay Schepler wrote:
The setting itself isn't taskmanager specific; it applies to both the
job- and taskmanager process.

On 02/05/2022 05:29, John Smith wrote:

Also just to be sure this is a Task Manager setting right?

On Thu, Apr 28, 2022 at 11:13 AM John Smith
wrote:

I assume you will take action on your side to track and fix the
doc? :)

On Thu, Apr 28, 2022 at 11:12 AM John Smith
wrote:

Ok so to summarize...

- Build my job jar and have the JDBC driver as a compile only
dependency and copy the JDBC driver to flink lib folder.

- Build my job jar and include JDBC driver in the shadow,
plus copy the JDBC driver in the flink lib folder, plus make
an entry in config for
|classloader.parent-first-patterns-additional|

On Thu, Apr 28, 2022 at 10:17 AM Chesnay Schepler
wrote:

I think what I meant was "either add it to /lib, or [if
it is already in /lib but also bundled in the jar] add it
to the parent-first patterns."

On 28/04/2022 15:56, Chesnay Schepler wrote:

Pretty sure, even though I seemingly documented it
incorrectly :)

On 28/04/2022 15:49, John Smith wrote:

You sure?

/JDBC/: JDBC drivers leak references outside the
user code classloader. To ensure that these classes
are only loaded once you should either add the
driver jars to Flink’s |lib/| folder, or add the
driver classes to the list of parent-first loaded
class via
|classloader.parent-first-patterns-additional|

It says either or

On Wed, Apr 27, 2022 at 3:44 AM Chesnay Schepler
wrote:

You're misinterpreting the docs.

The parent/child-first classloading controls where
Flink looks for a class /first/, specifically
whether we first load from /lib or the user-jar.
It does not allow you to load something from the
user-jar in the parent classloader. That's just not
how it works.

It must be in /lib.

On 27/04/2022 04:59, John Smith wrote:

Hi Chesnay as per the docs...

https://nightlies.apache.org/flink/flink-docs-master/docs/ops/debugging/debugging_classloading/

You can either put the jars in task manager lib
folder or use
|classloader.parent-first-patterns-additional|

I prefer the latter like this: the
dependency stays with the user-jar and not on the
task manager.

On Tue, Apr 26, 2022 at 9:52 PM John Smith
wrote:

Ok so I should put the Apache ignite and my
Microsoft drivers in the lib folders of my
task managers?

And then in my job jar only include them as
compile time dependencies?

On Tue, Apr 26, 2022 at 10:42 AM Chesnay
Schepler wrote:

JDBC drivers are well-known for leaking
classloaders unfortunately.

You have correctly identified your
alternatives.

You must put the jdbc driver into /lib
instead. Setting only the parent-first
pattern shouldn't affect anything.
That is only relevant if something is in
both in /lib and the user-jar, telling
Flink to prioritize what is in lib.

On 26/04/2022 15:35, John Smith wrote:

So I
put classloader.parent-first-patterns.additional:
"org.apache.ignite." in the task config
and so far I don't think I'm getting
"java.lang.OutOfMemoryError: Metaspace"
any more.

Or it's too early to tell.

Though now, the task managers are
shutting down due to some other failures.

So maybe because tasks were f

Re: How to debug Metaspace exception?

2022-05-01 Thread Chesnay Schepler

The setting itself isn't taskmanager specific; it applies to both the
job- and taskmanager process.

On 02/05/2022 05:29, John Smith wrote:

Also just to be sure this is a Task Manager setting right?

On Thu, Apr 28, 2022 at 11:13 AM John Smith
wrote:

I assume you will take action on your side to track and fix the
doc? :)

On Thu, Apr 28, 2022 at 11:12 AM John Smith
wrote:

Ok so to summarize...

- Build my job jar and have the JDBC driver as a compile only
dependency and copy the JDBC driver to flink lib folder.

- Build my job jar and include JDBC driver in the shadow, plus
copy the JDBC driver in the flink lib folder, plus make an
entry in config for
|classloader.parent-first-patterns-additional|

On Thu, Apr 28, 2022 at 10:17 AM Chesnay Schepler
wrote:

I think what I meant was "either add it to /lib, or [if it
is already in /lib but also bundled in the jar] add it to
the parent-first patterns."

On 28/04/2022 15:56, Chesnay Schepler wrote:

Pretty sure, even though I seemingly documented it
incorrectly :)

On 28/04/2022 15:49, John Smith wrote:

You sure?

It says either or

On Wed, Apr 27, 2022 at 3:44 AM Chesnay Schepler
wrote:

You're misinterpreting the docs.

It must be in /lib.

On 27/04/2022 04:59, John Smith wrote:

Hi Chesnay as per the docs...

https://nightlies.apache.org/flink/flink-docs-master/docs/ops/debugging/debugging_classloading/

You can either put the jars in task manager lib
folder or use
|classloader.parent-first-patterns-additional|

I prefer the latter like this: the dependency stays
with the user-jar and not on the task manager.

On Tue, Apr 26, 2022 at 9:52 PM John Smith
wrote:

Ok so I should put the Apache ignite and my
Microsoft drivers in the lib folders of my task
managers?

And then in my job jar only include them as
compile time dependencies?

On Tue, Apr 26, 2022 at 10:42 AM Chesnay
Schepler wrote:

JDBC drivers are well-known for leaking
classloaders unfortunately.

You have correctly identified your
alternatives.

On 26/04/2022 15:35, John Smith wrote:

So I
put classloader.parent-first-patterns.additional:
"org.apache.ignite." in the task config
and so far I don't think I'm getting
"java.lang.OutOfMemoryError: Metaspace"
any more.

Or it's too early to tell.

Though now, the task managers are shutting
down due to some other failures.

So maybe because tasks were failing and
reloading often the task manager was
running out of Metspace. But n

About job execution

2022-05-01 Thread Ww J

Hello,

I read some articles on internet about job execution of Flink and have some 
questions. When the dispatcher receives a job, the dispatcher will start a 
JobManager. After the job is finished, will the JobManager be shut down? For 
the TaskManager, after the job is finished, will the TaskManager be shut down? 
Or is the TaskManager shared among all the tasks?

Thanks.

Jack

Re: How to debug Metaspace exception?

2022-05-01 Thread John Smith

Also just to be sure this is a Task Manager setting right?

On Thu, Apr 28, 2022 at 11:13 AM John Smith  wrote:

> I assume you will take action on your side to track and fix the doc? :)
>
> On Thu, Apr 28, 2022 at 11:12 AM John Smith 
> wrote:
>
>> Ok so to summarize...
>>
>> - Build my job jar and have the JDBC driver as a compile only
>> dependency and copy the JDBC driver to flink lib folder.
>>
>> Or
>>
>> - Build my job jar and include JDBC driver in the shadow, plus copy the
>> JDBC driver in the flink lib folder, plus  make an entry in config for
>> classloader.parent-first-patterns-additional
>> 
>>
>>
>> On Thu, Apr 28, 2022 at 10:17 AM Chesnay Schepler 
>> wrote:
>>
>>> I think what I meant was "either add it to /lib, or [if it is already in
>>> /lib but also bundled in the jar] add it to the parent-first patterns."
>>>
>>> On 28/04/2022 15:56, Chesnay Schepler wrote:
>>>
>>> Pretty sure, even though I seemingly documented it incorrectly :)
>>>
>>> On 28/04/2022 15:49, John Smith wrote:
>>>
>>> You sure?
>>>
>>>-
>>>
>>>*JDBC*: JDBC drivers leak references outside the user code
>>>classloader. To ensure that these classes are only loaded once you should
>>>either add the driver jars to Flink’s lib/ folder, or add the driver
>>>classes to the list of parent-first loaded class via
>>>classloader.parent-first-patterns-additional
>>>
>>> 
>>>.
>>>
>>>It says either or
>>>
>>>
>>> On Wed, Apr 27, 2022 at 3:44 AM Chesnay Schepler 
>>> wrote:
>>>
 You're misinterpreting the docs.

 The parent/child-first classloading controls where Flink looks for a
 class *first*, specifically whether we first load from /lib or the
 user-jar.
 It does not allow you to load something from the user-jar in the parent
 classloader. That's just not how it works.

 It must be in /lib.

 On 27/04/2022 04:59, John Smith wrote:

 Hi Chesnay as per the docs...
 https://nightlies.apache.org/flink/flink-docs-master/docs/ops/debugging/debugging_classloading/

 You can either put the jars in task manager lib folder or use
 classloader.parent-first-patterns-additional
 

 I prefer the latter like this: the dependency stays with the user-jar
 and not on the task manager.

 On Tue, Apr 26, 2022 at 9:52 PM John Smith 
 wrote:

> Ok so I should put the Apache ignite and my Microsoft drivers in the
> lib folders of my task managers?
>
> And then in my job jar only include them as compile time dependencies?
>
>
> On Tue, Apr 26, 2022 at 10:42 AM Chesnay Schepler 
> wrote:
>
>> JDBC drivers are well-known for leaking classloaders unfortunately.
>>
>> You have correctly identified your alternatives.
>>
>> You must put the jdbc driver into /lib instead. Setting only the
>> parent-first pattern shouldn't affect anything.
>> That is only relevant if something is in both in /lib and the
>> user-jar, telling Flink to prioritize what is in lib.
>>
>>
>>
>> On 26/04/2022 15:35, John Smith wrote:
>>
>> So I put classloader.parent-first-patterns.additional:
>> "org.apache.ignite." in the task config and so far I don't think I'm
>> getting "java.lang.OutOfMemoryError: Metaspace" any more.
>>
>> Or it's too early to tell.
>>
>> Though now, the task managers are shutting down due to some
>> other failures.
>>
>> So maybe because tasks were failing and reloading often the task
>> manager was running out of Metspace. But now maybe it's just
>> cleanly shutting down.
>>
>> On Wed, Apr 20, 2022 at 11:35 AM John Smith 
>> wrote:
>>
>>> Or I can put in the config to treat org.apache.ignite. classes as
>>> first class?
>>>
>>> On Tue, Apr 19, 2022 at 10:18 PM John Smith 
>>> wrote:
>>>
 Ok, so I loaded the dump into Eclipse Mat and followed:
 https://cwiki.apache.org/confluence/display/FLINK/Debugging+ClassLoader+leaks

 - On the Histogram, I got over 30 entries for: ChildFirstClassLoader
 - Then I clicked on one of them "Merge Shortest Path..." and picked
 "Exclude all phantom/weak/soft references"
 - Which then gave me: SqlDriverManager > Apache Ignite JdbcThin
 Driver

 So i'm guessing anything JDBC based. I should copy into the task
 manager libs folder and my jobs make the dependencies as compile only?

 On Tue, Apr 19, 2022 at 12:18 PM Yaroslav Tkachenko <
 yaros...@goldsky.i

Re: How to debug Metaspace exception?

Re: How to debug Metaspace exception?

About job execution

Re: How to debug Metaspace exception?

4 matches

Site Navigation

Mail list logo

Footer information