I assume you will take action on your side to track and fix the doc? :) On Thu, Apr 28, 2022 at 11:12 AM John Smith <java.dev....@gmail.com> wrote:
> Ok so to summarize... > > - Build my job jar and have the JDBC driver as a compile only > dependency and copy the JDBC driver to flink lib folder. > > Or > > - Build my job jar and include JDBC driver in the shadow, plus copy the > JDBC driver in the flink lib folder, plus make an entry in config for > classloader.parent-first-patterns-additional > <https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#classloader-parent-first-patterns-additional> > > > On Thu, Apr 28, 2022 at 10:17 AM Chesnay Schepler <ches...@apache.org> > wrote: > >> I think what I meant was "either add it to /lib, or [if it is already in >> /lib but also bundled in the jar] add it to the parent-first patterns." >> >> On 28/04/2022 15:56, Chesnay Schepler wrote: >> >> Pretty sure, even though I seemingly documented it incorrectly :) >> >> On 28/04/2022 15:49, John Smith wrote: >> >> You sure? >> >> - >> >> *JDBC*: JDBC drivers leak references outside the user code >> classloader. To ensure that these classes are only loaded once you should >> either add the driver jars to Flink’s lib/ folder, or add the driver >> classes to the list of parent-first loaded class via >> classloader.parent-first-patterns-additional >> >> <https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#classloader-parent-first-patterns-additional> >> . >> >> It says either or >> >> >> On Wed, Apr 27, 2022 at 3:44 AM Chesnay Schepler <ches...@apache.org> >> wrote: >> >>> You're misinterpreting the docs. >>> >>> The parent/child-first classloading controls where Flink looks for a >>> class *first*, specifically whether we first load from /lib or the >>> user-jar. >>> It does not allow you to load something from the user-jar in the parent >>> classloader. That's just not how it works. >>> >>> It must be in /lib. >>> >>> On 27/04/2022 04:59, John Smith wrote: >>> >>> Hi Chesnay as per the docs... >>> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/debugging/debugging_classloading/ >>> >>> You can either put the jars in task manager lib folder or use >>> classloader.parent-first-patterns-additional >>> <https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#classloader-parent-first-patterns-additional> >>> >>> I prefer the latter like this: the dependency stays with the user-jar >>> and not on the task manager. >>> >>> On Tue, Apr 26, 2022 at 9:52 PM John Smith <java.dev....@gmail.com> >>> wrote: >>> >>>> Ok so I should put the Apache ignite and my Microsoft drivers in the >>>> lib folders of my task managers? >>>> >>>> And then in my job jar only include them as compile time dependencies? >>>> >>>> >>>> On Tue, Apr 26, 2022 at 10:42 AM Chesnay Schepler <ches...@apache.org> >>>> wrote: >>>> >>>>> JDBC drivers are well-known for leaking classloaders unfortunately. >>>>> >>>>> You have correctly identified your alternatives. >>>>> >>>>> You must put the jdbc driver into /lib instead. Setting only the >>>>> parent-first pattern shouldn't affect anything. >>>>> That is only relevant if something is in both in /lib and the >>>>> user-jar, telling Flink to prioritize what is in lib. >>>>> >>>>> >>>>> >>>>> On 26/04/2022 15:35, John Smith wrote: >>>>> >>>>> So I put classloader.parent-first-patterns.additional: >>>>> "org.apache.ignite." in the task config and so far I don't think I'm >>>>> getting "java.lang.OutOfMemoryError: Metaspace" any more. >>>>> >>>>> Or it's too early to tell. >>>>> >>>>> Though now, the task managers are shutting down due to some >>>>> other failures. >>>>> >>>>> So maybe because tasks were failing and reloading often the task >>>>> manager was running out of Metspace. But now maybe it's just >>>>> cleanly shutting down. >>>>> >>>>> On Wed, Apr 20, 2022 at 11:35 AM John Smith <java.dev....@gmail.com> >>>>> wrote: >>>>> >>>>>> Or I can put in the config to treat org.apache.ignite. classes as >>>>>> first class? >>>>>> >>>>>> On Tue, Apr 19, 2022 at 10:18 PM John Smith <java.dev....@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Ok, so I loaded the dump into Eclipse Mat and followed: >>>>>>> https://cwiki.apache.org/confluence/display/FLINK/Debugging+ClassLoader+leaks >>>>>>> >>>>>>> - On the Histogram, I got over 30 entries for: ChildFirstClassLoader >>>>>>> - Then I clicked on one of them "Merge Shortest Path..." and picked >>>>>>> "Exclude all phantom/weak/soft references" >>>>>>> - Which then gave me: SqlDriverManager > Apache Ignite JdbcThin >>>>>>> Driver >>>>>>> >>>>>>> So i'm guessing anything JDBC based. I should copy into the task >>>>>>> manager libs folder and my jobs make the dependencies as compile only? >>>>>>> >>>>>>> On Tue, Apr 19, 2022 at 12:18 PM Yaroslav Tkachenko < >>>>>>> yaros...@goldsky.io> wrote: >>>>>>> >>>>>>>> Also >>>>>>>> https://shopify.engineering/optimizing-apache-flink-applications-tips >>>>>>>> might be helpful (has a section on profiling, as well as classloading). >>>>>>>> >>>>>>>> On Tue, Apr 19, 2022 at 4:35 AM Chesnay Schepler < >>>>>>>> ches...@apache.org> wrote: >>>>>>>> >>>>>>>>> We have a very rough "guide" in the wiki (it's just the specific >>>>>>>>> steps I took to debug another leak): >>>>>>>>> >>>>>>>>> https://cwiki.apache.org/confluence/display/FLINK/Debugging+ClassLoader+leaks >>>>>>>>> >>>>>>>>> On 19/04/2022 12:01, huweihua wrote: >>>>>>>>> >>>>>>>>> Hi, John >>>>>>>>> >>>>>>>>> Sorry for the late reply. You can use MAT[1] to analyze the dump >>>>>>>>> file. Check whether have too many loaded classes. >>>>>>>>> >>>>>>>>> [1] https://www.eclipse.org/mat/ >>>>>>>>> >>>>>>>>> 2022年4月18日 下午9:55，John Smith <java.dev....@gmail.com> 写道： >>>>>>>>> >>>>>>>>> Hi, can anyone help with this? I never looked at a dump file >>>>>>>>> before. >>>>>>>>> >>>>>>>>> On Thu, Apr 14, 2022 at 11:59 AM John Smith < >>>>>>>>> java.dev....@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Hi, so I have a dump file. What do I look for? >>>>>>>>>> >>>>>>>>>> On Thu, Mar 31, 2022 at 3:28 PM John Smith < >>>>>>>>>> java.dev....@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Ok so if there's a leak, if I manually stop the job and restart >>>>>>>>>>> it from the UI multiple times, I won't see the issue because >>>>>>>>>>> because the >>>>>>>>>>> classes are unloaded correctly? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Thu, Mar 31, 2022 at 9:20 AM huweihua <huweihua....@gmail.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> The difference is that manually canceling the job stops the >>>>>>>>>>>> JobMaster, but automatic failover keeps the JobMaster running. But >>>>>>>>>>>> looking >>>>>>>>>>>> on TaskManager, it doesn't make much difference >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 2022年3月31日 上午4:01，John Smith <java.dev....@gmail.com> 写道： >>>>>>>>>>>> >>>>>>>>>>>> Also if I manually cancel and restart the same job over and >>>>>>>>>>>> over is it the same as if flink was restarting a job due to >>>>>>>>>>>> failure? >>>>>>>>>>>> >>>>>>>>>>>> I.e: When I click "Cancel Job" on the UI is the job completely >>>>>>>>>>>> unloaded vs when the job scheduler restarts a job because if >>>>>>>>>>>> whatever >>>>>>>>>>>> reason? >>>>>>>>>>>> >>>>>>>>>>>> Lile this I'll stop and restart the job a few times or maybe I >>>>>>>>>>>> can trick my job to fail and have the scheduler restart it. Ok let >>>>>>>>>>>> me think >>>>>>>>>>>> about this... >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Mar 30, 2022 at 10:24 AM 胡伟华 <huweihua....@gmail.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> So if I run the same jobs in my dev env will I still be able >>>>>>>>>>>>> to see the similar dump? >>>>>>>>>>>>> >>>>>>>>>>>>> I think running the same job in dev should be reproducible, >>>>>>>>>>>>> maybe you can have a try. >>>>>>>>>>>>> >>>>>>>>>>>>> If not I would have to wait at a low volume time to do it on >>>>>>>>>>>>> production. Aldo if I recall the dump is as big as the JVM memory >>>>>>>>>>>>> right so >>>>>>>>>>>>> if I have 10GB configed for the JVM the dump will be 10GB file? >>>>>>>>>>>>> >>>>>>>>>>>>> Yes, JMAP will pause the JVM, the time of pause depends on the >>>>>>>>>>>>> size to dump. you can use "jmap -dump:live" to dump only the >>>>>>>>>>>>> reachable >>>>>>>>>>>>> objects, this will take a brief pause >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 2022年3月30日 下午9:47，John Smith <java.dev....@gmail.com> 写道： >>>>>>>>>>>>> >>>>>>>>>>>>> I have 3 task managers (see config below). There is total of >>>>>>>>>>>>> 10 jobs with 25 slots being used. >>>>>>>>>>>>> The jobs are 100% ETL I.e; They load Json, transform it and >>>>>>>>>>>>> push it to JDBC, only 1 job of the 10 is pushing to Apache Ignite >>>>>>>>>>>>> cluster. >>>>>>>>>>>>> >>>>>>>>>>>>> FOR JMAP. I know that it will pause the task manager. So if I >>>>>>>>>>>>> run the same jobs in my dev env will I still be able to see the >>>>>>>>>>>>> similar >>>>>>>>>>>>> dump? I I assume so. If not I would have to wait at a low volume >>>>>>>>>>>>> time to do >>>>>>>>>>>>> it on production. Aldo if I recall the dump is as big as the JVM >>>>>>>>>>>>> memory >>>>>>>>>>>>> right so if I have 10GB configed for the JVM the dump will be >>>>>>>>>>>>> 10GB file? >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> # Operating system has 16GB total. >>>>>>>>>>>>> env.ssh.opts: -l flink -oStrictHostKeyChecking=no >>>>>>>>>>>>> >>>>>>>>>>>>> cluster.evenly-spread-out-slots: true >>>>>>>>>>>>> >>>>>>>>>>>>> taskmanager.memory.flink.size: 10240m >>>>>>>>>>>>> taskmanager.memory.jvm-metaspace.size: 2048m >>>>>>>>>>>>> taskmanager.numberOfTaskSlots: 16 >>>>>>>>>>>>> parallelism.default: 1 >>>>>>>>>>>>> >>>>>>>>>>>>> high-availability: zookeeper >>>>>>>>>>>>> high-availability.storageDir: file:///mnt/flink/ha/flink_1_14/ >>>>>>>>>>>>> high-availability.zookeeper.quorum: ... >>>>>>>>>>>>> high-availability.zookeeper.path.root: /flink_1_14 >>>>>>>>>>>>> high-availability.cluster-id: /flink_1_14_cluster_0001 >>>>>>>>>>>>> >>>>>>>>>>>>> web.upload.dir: /mnt/flink/uploads/flink_1_14 >>>>>>>>>>>>> >>>>>>>>>>>>> state.backend: rocksdb >>>>>>>>>>>>> state.backend.incremental: true >>>>>>>>>>>>> state.checkpoints.dir: >>>>>>>>>>>>> file:///mnt/flink/checkpoints/flink_1_14 >>>>>>>>>>>>> state.savepoints.dir: file:///mnt/flink/savepoints/flink_1_14 >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Mar 30, 2022 at 2:16 AM 胡伟华 <huweihua....@gmail.com> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, John >>>>>>>>>>>>>> >>>>>>>>>>>>>> Could you tell us you application scenario? Is it a flink >>>>>>>>>>>>>> session cluster with a lot of jobs? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Maybe you can try to dump the memory with jmap and use tools >>>>>>>>>>>>>> such as MAT to analyze whether there are abnormal classes and >>>>>>>>>>>>>> classloaders >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> > 2022年3月30日 上午6:09，John Smith <java.dev....@gmail.com> 写道： >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > Hi running 1.14.4 >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > My tasks manager still fails with >>>>>>>>>>>>>> java.lang.OutOfMemoryError: Metaspace. The metaspace >>>>>>>>>>>>>> out-of-memory error >>>>>>>>>>>>>> has occurred. This can mean two things: either the job requires >>>>>>>>>>>>>> a larger >>>>>>>>>>>>>> size of JVM metaspace to load classes or there is a class >>>>>>>>>>>>>> loading leak. >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > I have 2GB of metaspace configed >>>>>>>>>>>>>> taskmanager.memory.jvm-metaspace.size: 2048m >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > But the task nodes still fail. >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > When looking at the UI metrics, the metaspace starts low. >>>>>>>>>>>>>> Now I see 85% usage. It seems to be a class loading leak at this >>>>>>>>>>>>>> point, how >>>>>>>>>>>>>> can we debug this issue? >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> >>> >> >>
