I think what I meant was "either add it to /lib, or [if it is already in /lib but also bundled in the jar] add it to the parent-first patterns."

On 28/04/2022 15:56, Chesnay Schepler wrote:
Pretty sure, even though I seemingly documented it incorrectly :)

On 28/04/2022 15:49, John Smith wrote:
You sure?

 *

    /JDBC/: JDBC drivers leak references outside the user code
    classloader. To ensure that these classes are only loaded once
    you should either add the driver jars to Flink’s |lib/| folder,
    or add the driver classes to the list of parent-first loaded
    class via |classloader.parent-first-patterns-additional|
    
<https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#classloader-parent-first-patterns-additional>.

    It says either or


On Wed, Apr 27, 2022 at 3:44 AM Chesnay Schepler <ches...@apache.org> wrote:

    You're misinterpreting the docs.

    The parent/child-first classloading controls where Flink looks
    for a class /first/, specifically whether we first load from /lib
    or the user-jar.
    It does not allow you to load something from the user-jar in the
    parent classloader. That's just not how it works.

    It must be in /lib.

    On 27/04/2022 04:59, John Smith wrote:
    Hi Chesnay as per the docs...
    
https://nightlies.apache.org/flink/flink-docs-master/docs/ops/debugging/debugging_classloading/

    You can either put the jars in task manager lib folder or use
    |classloader.parent-first-patterns-additional|
    
<https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#classloader-parent-first-patterns-additional>

    I prefer the latter like this: the dependency stays with the
    user-jar and not on the task manager.

    On Tue, Apr 26, 2022 at 9:52 PM John Smith
    <java.dev....@gmail.com> wrote:

        Ok so I should put the Apache ignite and my Microsoft
        drivers in the lib folders of my task managers?

        And then in my job jar only include them as compile time
        dependencies?


        On Tue, Apr 26, 2022 at 10:42 AM Chesnay Schepler
        <ches...@apache.org> wrote:

            JDBC drivers are well-known for leaking classloaders
            unfortunately.

            You have correctly identified your alternatives.

            You must put the jdbc driver into /lib instead. Setting
            only the parent-first pattern shouldn't affect anything.
            That is only relevant if something is in both in /lib
            and the user-jar, telling Flink to prioritize what is in
            lib.



            On 26/04/2022 15:35, John Smith wrote:
            So I put classloader.parent-first-patterns.additional:
            "org.apache.ignite." in the task config and so far I
            don't think I'm getting "java.lang.OutOfMemoryError:
            Metaspace" any more.

            Or it's too early to tell.

            Though now, the task managers are shutting down due to
            some other failures.

            So maybe because tasks were failing and reloading often
            the task manager was running out of Metspace. But now
            maybe it's just cleanly shutting down.

            On Wed, Apr 20, 2022 at 11:35 AM John Smith
            <java.dev....@gmail.com> wrote:

                Or I can put in the config to treat
                org.apache.ignite. classes as first class?

                On Tue, Apr 19, 2022 at 10:18 PM John Smith
                <java.dev....@gmail.com> wrote:

                    Ok, so I loaded the dump into Eclipse Mat and
                    followed:
                    
https://cwiki.apache.org/confluence/display/FLINK/Debugging+ClassLoader+leaks

                    - On the Histogram, I got over 30 entries for:
                    ChildFirstClassLoader
                    - Then I clicked on one of them "Merge Shortest
                    Path..." and picked "Exclude all
                    phantom/weak/soft references"
                    - Which then gave me: SqlDriverManager > Apache
                    Ignite JdbcThin Driver

                    So i'm guessing anything JDBC based. I should
                    copy into the task manager libs folder and my
                    jobs make the dependencies as compile only?

                    On Tue, Apr 19, 2022 at 12:18 PM Yaroslav
                    Tkachenko <yaros...@goldsky.io> wrote:

                        Also
                        
https://shopify.engineering/optimizing-apache-flink-applications-tips
                        might be helpful (has a section on
                        profiling, as well as classloading).

                        On Tue, Apr 19, 2022 at 4:35 AM Chesnay
                        Schepler <ches...@apache.org> wrote:

                            We have a very rough "guide" in the
                            wiki (it's just the specific steps I
                            took to debug another leak):
                            
https://cwiki.apache.org/confluence/display/FLINK/Debugging+ClassLoader+leaks

                            On 19/04/2022 12:01, huweihua wrote:
                            Hi, John

                            Sorry for the late reply. You can use
                            MAT[1] to analyze the dump file. Check
                            whether have too many loaded classes.

                            [1] https://www.eclipse.org/mat/

                            2022年4月18日 下午9:55,John Smith
                            <java.dev....@gmail.com> 写道:

                            Hi, can anyone help with this? I
                            never looked at a dump file before.

                            On Thu, Apr 14, 2022 at 11:59 AM John
                            Smith <java.dev....@gmail.com> wrote:

                                Hi, so I have a dump file. What
                                do I look for?

                                On Thu, Mar 31, 2022 at 3:28 PM
                                John Smith
                                <java.dev....@gmail.com> wrote:

                                    Ok so if there's a leak, if I
                                    manually stop the job and
                                    restart it from the UI
                                    multiple times, I won't see
                                    the issue because because the
                                    classes are unloaded correctly?


                                    On Thu, Mar 31, 2022 at 9:20
                                    AM huweihua
                                    <huweihua....@gmail.com> wrote:


                                        The difference is that
                                        manually canceling the
                                        job stops the JobMaster,
                                        but automatic failover
                                        keeps the JobMaster
                                        running. But looking on
                                        TaskManager, it doesn't
                                        make much difference


                                        2022年3月31日 上午4:01,John
                                        Smith
                                        <java.dev....@gmail.com>
                                        写道:

                                        Also if I manually
                                        cancel and restart the
                                        same job over and over
                                        is it the same as if
                                        flink was restarting a
                                        job due to failure?

                                        I.e: When I click
                                        "Cancel Job" on the UI
                                        is the job completely
                                        unloaded vs when the job
                                        scheduler restarts a job
                                        because if whatever reason?

                                        Lile this I'll stop and
                                        restart the job a few
                                        times or maybe I can
                                        trick my job to fail and
                                        have the scheduler
                                        restart it. Ok let me
                                        think about this...

                                        On Wed, Mar 30, 2022 at
                                        10:24 AM 胡伟华
                                        <huweihua....@gmail.com>
                                        wrote:

                                            So if I run the
                                            same jobs in my dev
                                            env will I still be
                                            able to see the
                                            similar dump?
                                            I think running the
                                            same job in dev
                                            should be
                                            reproducible, maybe
                                            you can have a try.

                                             If not I would
                                            have to wait at a
                                            low volume time to
                                            do it on
                                            production. Aldo if
                                            I recall the dump
                                            is as big as the
                                            JVM memory right so
                                            if I have 10GB
                                            configed for the
                                            JVM the dump will
                                            be 10GB file?
                                            Yes, JMAP will pause
                                            the JVM, the time of
                                            pause depends on the
                                            size to dump. you
                                            can use "jmap
                                            -dump:live" to dump
                                            only the reachable
                                            objects, this will
                                            take a brief pause



                                            2022年3月30日
                                            下午9:47,John Smith
                                            <java.dev....@gmail.com>
                                            写道:

                                            I have 3 task
                                            managers (see
                                            config below).
                                            There is total of
                                            10 jobs with 25
                                            slots being used.
                                            The jobs are 100%
                                            ETL I.e; They load
                                            Json, transform it
                                            and push it to
                                            JDBC, only 1 job of
                                            the 10 is pushing
                                            to Apache Ignite
                                            cluster.

                                            FOR JMAP. I know
                                            that it will pause
                                            the task manager.
                                            So if I run the
                                            same jobs in my dev
                                            env will I still be
                                            able to see the
                                            similar dump? I I
                                            assume so. If not I
                                            would have to wait
                                            at a low volume
                                            time to do it on
                                            production. Aldo if
                                            I recall the dump
                                            is as big as the
                                            JVM memory right so
                                            if I have 10GB
                                            configed for the
                                            JVM the dump will
                                            be 10GB file?


                                            # Operating system
                                            has 16GB total.
                                            env.ssh.opts: -l
                                            flink
                                            -oStrictHostKeyChecking=no

                                            cluster.evenly-spread-out-slots:
                                            true

                                            taskmanager.memory.flink.size:
                                            10240m
                                            
taskmanager.memory.jvm-metaspace.size:
                                            2048m
                                            taskmanager.numberOfTaskSlots:
                                            16
                                            parallelism.default: 1

                                            high-availability:
                                            zookeeper
                                            high-availability.storageDir:
                                            file:///mnt/flink/ha/flink_1_14/
                                            high-availability.zookeeper.quorum:
                                            ...
                                            
high-availability.zookeeper.path.root:
                                            /flink_1_14
                                            high-availability.cluster-id:
                                            /flink_1_14_cluster_0001

                                            web.upload.dir:
                                            /mnt/flink/uploads/flink_1_14

                                            state.backend: rocksdb
                                            state.backend.incremental:
                                            true
                                            state.checkpoints.dir:
                                            
file:///mnt/flink/checkpoints/flink_1_14
                                            state.savepoints.dir:
                                            
file:///mnt/flink/savepoints/flink_1_14

                                            On Wed, Mar 30,
                                            2022 at 2:16 AM 胡伟华
                                            <huweihua....@gmail.com>
                                            wrote:

                                                Hi, John

                                                Could you tell
                                                us you
                                                application
                                                scenario? Is it
                                                a flink session
                                                cluster with a
                                                lot of jobs?

                                                Maybe you can
                                                try to dump the
                                                memory with
                                                jmap and use
                                                tools such as
                                                MAT to analyze
                                                whether there
                                                are abnormal
                                                classes and
                                                classloaders


                                                > 2022年3月30日
                                                上午6:09,John
                                                Smith
                                                <java.dev....@gmail.com>
                                                写道:
                                                >
                                                > Hi running 1.14.4
                                                >
                                                > My tasks
                                                manager still
                                                fails with
                                                java.lang.OutOfMemoryError:
                                                Metaspace. The
                                                metaspace
                                                out-of-memory
                                                error has
                                                occurred. This
                                                can mean two
                                                things: either
                                                the job
                                                requires a
                                                larger size of
                                                JVM metaspace
                                                to load classes
                                                or there is a
                                                class loading leak.
                                                >
                                                > I have 2GB of
                                                metaspace
                                                configed
                                                
taskmanager.memory.jvm-metaspace.size:
                                                2048m
                                                >
                                                > But the task
                                                nodes still fail.
                                                >
                                                > When looking
                                                at the UI
                                                metrics, the
                                                metaspace
                                                starts low. Now
                                                I see 85%
                                                usage. It seems
                                                to be a class
                                                loading leak at
                                                this point, how
                                                can we debug
                                                this issue?








Reply via email to