The setting itself isn't taskmanager specific; it applies to both the
job- and taskmanager process.
On 02/05/2022 05:29, John Smith wrote:
Also just to be sure this is a Task Manager setting right?
On Thu, Apr 28, 2022 at 11:13 AM John Smith <java.dev....@gmail.com>
wrote:
I assume you will take action on your side to track and fix the
doc? :)
On Thu, Apr 28, 2022 at 11:12 AM John Smith
<java.dev....@gmail.com> wrote:
Ok so to summarize...
- Build my job jar and have the JDBC driver as a compile only
dependency and copy the JDBC driver to flink lib folder.
Or
- Build my job jar and include JDBC driver in the shadow, plus
copy the JDBC driver in the flink lib folder, plus make an
entry in config for
|classloader.parent-first-patterns-additional|
<https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#classloader-parent-first-patterns-additional>
On Thu, Apr 28, 2022 at 10:17 AM Chesnay Schepler
<ches...@apache.org> wrote:
I think what I meant was "either add it to /lib, or [if it
is already in /lib but also bundled in the jar] add it to
the parent-first patterns."
On 28/04/2022 15:56, Chesnay Schepler wrote:
Pretty sure, even though I seemingly documented it
incorrectly :)
On 28/04/2022 15:49, John Smith wrote:
You sure?
*
/JDBC/: JDBC drivers leak references outside the
user code classloader. To ensure that these classes
are only loaded once you should either add the
driver jars to Flink’s |lib/| folder, or add the
driver classes to the list of parent-first loaded
class via
|classloader.parent-first-patterns-additional|
<https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#classloader-parent-first-patterns-additional>.
It says either or
On Wed, Apr 27, 2022 at 3:44 AM Chesnay Schepler
<ches...@apache.org> wrote:
You're misinterpreting the docs.
The parent/child-first classloading controls where
Flink looks for a class /first/, specifically
whether we first load from /lib or the user-jar.
It does not allow you to load something from the
user-jar in the parent classloader. That's just not
how it works.
It must be in /lib.
On 27/04/2022 04:59, John Smith wrote:
Hi Chesnay as per the docs...
https://nightlies.apache.org/flink/flink-docs-master/docs/ops/debugging/debugging_classloading/
You can either put the jars in task manager lib
folder or use
|classloader.parent-first-patterns-additional|
<https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#classloader-parent-first-patterns-additional>
I prefer the latter like this: the dependency stays
with the user-jar and not on the task manager.
On Tue, Apr 26, 2022 at 9:52 PM John Smith
<java.dev....@gmail.com> wrote:
Ok so I should put the Apache ignite and my
Microsoft drivers in the lib folders of my task
managers?
And then in my job jar only include them as
compile time dependencies?
On Tue, Apr 26, 2022 at 10:42 AM Chesnay
Schepler <ches...@apache.org> wrote:
JDBC drivers are well-known for leaking
classloaders unfortunately.
You have correctly identified your
alternatives.
You must put the jdbc driver into /lib
instead. Setting only the parent-first
pattern shouldn't affect anything.
That is only relevant if something is in
both in /lib and the user-jar, telling
Flink to prioritize what is in lib.
On 26/04/2022 15:35, John Smith wrote:
So I
put classloader.parent-first-patterns.additional:
"org.apache.ignite." in the task config
and so far I don't think I'm getting
"java.lang.OutOfMemoryError: Metaspace"
any more.
Or it's too early to tell.
Though now, the task managers are shutting
down due to some other failures.
So maybe because tasks were failing and
reloading often the task manager was
running out of Metspace. But now maybe
it's just cleanly shutting down.
On Wed, Apr 20, 2022 at 11:35 AM John
Smith <java.dev....@gmail.com> wrote:
Or I can put in the config to treat
org.apache.ignite. classes as first class?
On Tue, Apr 19, 2022 at 10:18 PM John
Smith <java.dev....@gmail.com> wrote:
Ok, so I loaded the dump into
Eclipse Mat and followed:
https://cwiki.apache.org/confluence/display/FLINK/Debugging+ClassLoader+leaks
- On the Histogram, I got over 30
entries for: ChildFirstClassLoader
- Then I clicked on one of them
"Merge Shortest Path..." and
picked "Exclude all
phantom/weak/soft references"
- Which then gave me:
SqlDriverManager > Apache Ignite
JdbcThin Driver
So i'm guessing anything JDBC
based. I should copy into the task
manager libs folder and my jobs
make the dependencies as compile only?
On Tue, Apr 19, 2022 at 12:18 PM
Yaroslav Tkachenko
<yaros...@goldsky.io> wrote:
Also
https://shopify.engineering/optimizing-apache-flink-applications-tips
might be helpful (has a
section on profiling, as well
as classloading).
On Tue, Apr 19, 2022 at 4:35
AM Chesnay Schepler
<ches...@apache.org> wrote:
We have a very rough
"guide" in the wiki (it's
just the specific steps I
took to debug another leak):
https://cwiki.apache.org/confluence/display/FLINK/Debugging+ClassLoader+leaks
On 19/04/2022 12:01,
huweihua wrote:
Hi, John
Sorry for the late reply.
You can use MAT[1] to
analyze the dump file.
Check whether have too
many loaded classes.
[1]
https://www.eclipse.org/mat/
2022年4月18日 下午9:55，John
Smith
<java.dev....@gmail.com>
写道：
Hi, can anyone help with
this? I never looked at
a dump file before.
On Thu, Apr 14, 2022 at
11:59 AM John Smith
<java.dev....@gmail.com>
wrote:
Hi, so I have a dump
file. What do I look
for?
On Thu, Mar 31, 2022
at 3:28 PM John
Smith
<java.dev....@gmail.com>
wrote:
Ok so if there's
a leak, if I
manually stop
the job and
restart it from
the UI multiple
times, I won't
see the issue
because because
the classes are
unloaded correctly?
On Thu, Mar 31,
2022 at 9:20 AM
huweihua
<huweihua....@gmail.com>
wrote:
The
difference
is that
manually
canceling
the job
stops the
JobMaster,
but
automatic
failover
keeps the
JobMaster
running. But
looking on
TaskManager,
it doesn't
make much
difference
2022年3月31日
上午4:01，John
Smith
<java.dev....@gmail.com>
写道：
Also if I
manually
cancel and
restart the
same job
over and
over is it
the same as
if flink
was
restarting
a job due
to failure?
I.e: When I
click
"Cancel
Job" on the
UI is the
job
completely
unloaded vs
when the
job
scheduler
restarts a
job because
if whatever
reason?
Lile this
I'll stop
and restart
the job a
few times
or maybe I
can trick
my job to
fail and
have the
scheduler
restart it.
Ok let me
think about
this...
On Wed, Mar
30, 2022 at
10:24 AM
胡伟华
<huweihua....@gmail.com>
wrote:
So if
I run
the
same
jobs
in my
dev
env
will I
still
be
able
to see
the
similar
dump?
I think
running
the
same
job in
dev
should
be
reproducible,
maybe
you can
have a try.
If
not I
would
have
to
wait
at a
low
volume
time
to do
it on
production.
Aldo
if I
recall
the
dump
is as
big as
the
JVM
memory
right
so if
I have
10GB
configed
for
the
JVM
the
dump
will
be
10GB file?
Yes,
JMAP
will
pause
the
JVM,
the
time of
pause
depends
on the
size to
dump.
you can
use
"jmap
-dump:live"
to dump
only
the
reachable
objects,
this
will
take a
brief pause
2022年3月30日
下午9:47，John
Smith
<java.dev....@gmail.com>
写道：
I have
3 task
managers
(see
config
below).
There
is
total
of 10
jobs
with
25
slots
being
used.
The
jobs
are
100%
ETL
I.e;
They
load
Json,
transform
it and
push
it to
JDBC,
only 1
job of
the 10
is
pushing
to
Apache
Ignite
cluster.
FOR
JMAP.
I know
that
it
will
pause
the
task
manager.
So if
I run
the
same
jobs
in my
dev
env
will I
still
be
able
to see
the
similar
dump?
I I
assume
so. If
not I
would
have
to
wait
at a
low
volume
time
to do
it on
production.
Aldo
if I
recall
the
dump
is as
big as
the
JVM
memory
right
so if
I have
10GB
configed
for
the
JVM
the
dump
will
be
10GB file?
#
Operating
system
has
16GB
total.
env.ssh.opts:
-l
flink
-oStrictHostKeyChecking=no
cluster.evenly-spread-out-slots:
true
taskmanager.memory.flink.size:
10240m
taskmanager.memory.jvm-metaspace.size:
2048m
taskmanager.numberOfTaskSlots:
16
parallelism.default:
1
high-availability:
zookeeper
high-availability.storageDir:
file:///mnt/flink/ha/flink_1_14/
high-availability.zookeeper.quorum:
...
high-availability.zookeeper.path.root:
/flink_1_14
high-availability.cluster-id:
/flink_1_14_cluster_0001
web.upload.dir:
/mnt/flink/uploads/flink_1_14
state.backend:
rocksdb
state.backend.incremental:
true
state.checkpoints.dir:
file:///mnt/flink/checkpoints/flink_1_14
state.savepoints.dir:
file:///mnt/flink/savepoints/flink_1_14
On
Wed,
Mar
30,
2022
at
2:16
AM 胡伟华
<huweihua....@gmail.com>
wrote:
Hi,
John
Could
you
tell
us
you
application
scenario?
Is
it
a
flink
session
cluster
with
a
lot
of
jobs?
Maybe
you
can
try
to
dump
the
memory
with
jmap
and
use
tools
such
as
MAT
to
analyze
whether
there
are
abnormal
classes
and
classloaders
>
2022年3月30日
上午6:09，John
Smith
<java.dev....@gmail.com>
写道：
>
>
Hi
running
1.14.4
>
>
My
tasks
manager
still
fails
with
java.lang.OutOfMemoryError:
Metaspace.
The
metaspace
out-of-memory
error
has
occurred.
This
can
mean
two
things:
either
the
job
requires
a
larger
size
of
JVM
metaspace
to
load
classes
or
there
is
a
class
loading
leak.
>
>
I
have
2GB
of
metaspace
configed
taskmanager.memory.jvm-metaspace.size:
2048m
>
>
But
the
task
nodes
still
fail.
>
>
When
looking
at
the
UI
metrics,
the
metaspace
starts
low.
Now
I
see
85%
usage.
It
seems
to
be
a
class
loading
leak
at
this
point,
how
can
we
debug
this
issue?
