Just to close this thread I found the cause of the problem: looking into the code of the mysql connector the value of PropertyDefinitions.SYSP_disableAbandonedConnectionCleanup is "com.mysql.cj.disableAbandonedConnectionCleanup" and not "com.mysql.disableAbandonedConnectionCleanup" as stated in [1]...I'll try to inform the guys of the mysql site.. Thanks anyway for the support.
[1] https://dev.mysql.com/doc/relnotes/connector-j/8.0/en/news-8-0-22.html On Fri, Nov 20, 2020 at 1:13 PM Arvid Heise <ar...@ververica.com> wrote: > No magic for JVM properties afaik (and I just looked in the code base for > the most obvious candidates). There is also nothing to gain from > overwriting properties. > I'm also certain that it should work as it's used in most secured setups > to inject keys/keytabs. > > What happens if you execute the Flink program in a local executor or mini > cluster? Could you set a breakpoint to the static initializer of > AbandonedConnectionCleanupThread and check what's going on there? > > > > On Fri, Nov 20, 2020 at 12:58 PM Flavio Pompermaier <pomperma...@okkam.it> > wrote: > >> Yes, that's what is surprising..I already did a remote debug on the TM >> and that property is not read..but that's really weird..could it be that >> the JVM properties gets cleared before invoking the tasks? >> >> Il ven 20 nov 2020, 12:50 Arvid Heise <ar...@ververica.com> ha scritto: >> >>> All looks good and as it should be. >>> >>> Can you do a remote debugging session to the tm once more and check >>> Boolean.getBoolean("com.mysql.disableAbandonedConnectionCleanup") >>> >>> There is no magic involved in System properties in Flink. If the >>> property is set on the process, the configuration works. If it's in the >>> project, mysql can access it. >>> >>> On Fri, Nov 20, 2020 at 10:46 AM Flavio Pompermaier < >>> pomperma...@okkam.it> wrote: >>> >>>> I've just tested the following code in a java class and the property >>>> (-Dcom.mysql.cj.disableAbandonedConnectionCleanup=true) is read correctly >>>> and the abandonedConnectionCleanupDisabled does not initialize >>>> the cleanupThreadExecutorService (that in my other test was causing a >>>> dynamic classloading memory leak): >>>> >>>> try { >>>> Class.forName(drivername); >>>> Connection dbConn = DriverManager.getConnection(dbURL, username, >>>> password); >>>> System.out.println("OK"); >>>> } catch (SQLException se) { >>>> throw new IllegalArgumentException("open() failed." + >>>> se.getMessage(), se); >>>> } catch (ClassNotFoundException cnfe) { >>>> throw new IllegalArgumentException("JDBC-Class not found. - " + >>>> cnfe.getMessage(), cnfe); >>>> } >>>> >>>> This is the output of ps aux command (I can also notice that the >>>> taskmanager parameters are created using a space after the -D option that I >>>> don't know if it intended or not): >>>> >>>> flink 23904 59.5 1.1 5488264 380396 pts/5 Sl 10:41 0:08 java >>>> -Xmx536870902 -Xms536870902 -XX:MaxDirectMemorySize=268435458 >>>> -XX:MaxMetaspaceSize=268435456 >>>> -Dcom.mysql.disableAbandonedConnectionCleanup=true >>>> -Dlog.file=/opt/flink/flink-1.11.0/log/flink-flink-taskexecutor-0-flavio-ThinkPad-P50.log >>>> -Dlog4j.configuration=file:/opt/flink/flink-1.11.0/conf/log4j.properties >>>> -Dlog4j.configurationFile=file:/opt/flink/flink-1.11.0/conf/log4j.properties >>>> -Dlogback.configurationFile=file:/opt/flink/flink-1.11.0/conf/logback.xml >>>> -classpath >>>> /opt/flink/flink-1.11.0/lib/flink-csv-1.11.0.jar:/opt/flink/flink-1.11.0/lib/flink-json-1.11.0.jar:/opt/flink/flink-1.11.0/lib/flink-shaded-zookeeper-3.4.14.jar:/opt/flink/flink-1.11.0/lib/flink-table_2.12-1.11.0.jar:/opt/flink/flink-1.11.0/lib/flink-table-blink_2.12-1.11.0.jar:/opt/flink/flink-1.11.0/lib/log4j-1.2-api-2.12.1.jar:/opt/flink/flink-1.11.0/lib/log4j-api-2.12.1.jar:/opt/flink/flink-1.11.0/lib/log4j-core-2.12.1.jar:/opt/flink/flink-1.11.0/lib/log4j-slf4j-impl-2.12.1.jar:/opt/flink/flink-1.11.0/lib/mariadb-java-client-2.6.0.jar:/opt/flink/flink-1.11.0/lib/mssql-jdbc-7.4.1.jre11.jar:/opt/flink/flink-1.11.0/lib/mysql-connector-java-8.0.22.jar:/opt/flink/flink-1.11.0/lib/flink-dist_2.12-1.11.0.jar::: >>>> org.apache.flink.runtime.taskexecutor.TaskManagerRunner --configDir >>>> /opt/flink/flink-1.11.0/conf -D >>>> taskmanager.memory.framework.off-heap.size=134217728b -D >>>> taskmanager.memory.network.max=134217730b -D >>>> taskmanager.memory.network.min=134217730b -D >>>> taskmanager.memory.framework.heap.size=134217728b -D >>>> taskmanager.memory.managed.size=536870920b -D taskmanager.cpu.cores=1.0 -D >>>> taskmanager.memory.task.heap.size=402653174b -D >>>> taskmanager.memory.task.off-heap.size=0b >>>> >>>> The mysql dependency in the maven project is: >>>> >>>> <dependency> >>>> <groupId>mysql</groupId> >>>> <artifactId>mysql-connector-java</artifactId> >>>> <version>8.0.22</version> >>>> <scope>provided</scope> >>>> </dependency> >>>> >>>> On Fri, Nov 20, 2020 at 10:07 AM Flavio Pompermaier < >>>> pomperma...@okkam.it> wrote: >>>> >>>>> no no I didn't relocate any class related to jdbc >>>>> >>>>> Il ven 20 nov 2020, 10:02 Arvid Heise <ar...@ververica.com> ha >>>>> scritto: >>>>> >>>>>> I was particularly asking if you relocate classes. Since the property >>>>>> name looks like a class name, it could have been changed as well. Could >>>>>> you >>>>>> check the value of >>>>>> PropertyDefinitions.SYSP_disableAbandonedConnectionCleanup in your final >>>>>> jar? >>>>>> >>>>>> On Fri, Nov 20, 2020 at 9:35 AM Flavio Pompermaier < >>>>>> pomperma...@okkam.it> wrote: >>>>>> >>>>>>> the mysql connector is put in the client classpath and in the Flink >>>>>>> lib dir. When i debugged remotely the AbandonedConnectionCleanupThread >>>>>>> was initialized at the first run of the job by the taskmamager. Today >>>>>>> I'll >>>>>>> try to run the mysql connector in a standalone java app to see if the >>>>>>> property is read correctly or not. >>>>>>> >>>>>>> Il ven 20 nov 2020, 07:52 Arvid Heise <ar...@ververica.com> ha >>>>>>> scritto: >>>>>>> >>>>>>>> Hi Flavio, >>>>>>>> >>>>>>>> if it arrives in the java process then you are doing everything >>>>>>>> right already (or almost). >>>>>>>> >>>>>>>> Are you shading the mysql connector? I'm suspecting that the >>>>>>>> property also get shaded then. You could decompile your jar to be sure. >>>>>>>> Have you verified that this is working as intended without Flink? >>>>>>>> >>>>>>>> On Thu, Nov 19, 2020 at 9:19 PM Flavio Pompermaier < >>>>>>>> pomperma...@okkam.it> wrote: >>>>>>>> >>>>>>>>> the properties arrives to the task manager because I can see them >>>>>>>>> in the java process (using ps aux)..or donyoubmean some special line >>>>>>>>> of >>>>>>>>> code? >>>>>>>>> >>>>>>>>> Il gio 19 nov 2020, 20:53 Arvid Heise <ar...@ververica.com> ha >>>>>>>>> scritto: >>>>>>>>> >>>>>>>>>> Hi Flavio, >>>>>>>>>> >>>>>>>>>> you are right, all looks good. >>>>>>>>>> >>>>>>>>>> Can you please verify if the properties arrived at the task >>>>>>>>>> manager in the remote debugger session? For example, you could check >>>>>>>>>> the >>>>>>>>>> JVisualVM Overview tab. >>>>>>>>>> >>>>>>>>>> On Thu, Nov 19, 2020 at 8:38 PM Flavio Pompermaier < >>>>>>>>>> pomperma...@okkam.it> wrote: >>>>>>>>>> >>>>>>>>>>> At the moment I use a standalone cluster, isn't using >>>>>>>>>>> env.java.opts the right way to do it? >>>>>>>>>>> >>>>>>>>>>> Il gio 19 nov 2020, 20:11 Arvid Heise <ar...@ververica.com> ha >>>>>>>>>>> scritto: >>>>>>>>>>> >>>>>>>>>>>> Hi Flavio, >>>>>>>>>>>> >>>>>>>>>>>> -D afaik passes only the system property to the entry point >>>>>>>>>>>> (client or jobmanager depending on setup), while you probably want >>>>>>>>>>>> to have >>>>>>>>>>>> it on the task managers. >>>>>>>>>>>> >>>>>>>>>>>> The specific options to pass it to the task managers depend on >>>>>>>>>>>> the way you deploy. -yD for yarn for example. For docker or k8s, >>>>>>>>>>>> you would >>>>>>>>>>>> use env. >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Nov 18, 2020 at 10:20 PM Flavio Pompermaier < >>>>>>>>>>>> pomperma...@okkam.it> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi to all, >>>>>>>>>>>>> while trying to solve a leak with dynamic class loading I >>>>>>>>>>>>> found out >>>>>>>>>>>>> that mysql connector creates an >>>>>>>>>>>>> AbandonedConnectionCleanupThread that >>>>>>>>>>>>> is retained in the ChildFirstClassLoader..from version 8.0.22 >>>>>>>>>>>>> there's >>>>>>>>>>>>> the possibility to inhibit this thread passing the system >>>>>>>>>>>>> property >>>>>>>>>>>>> com.mysql.disableAbandonedConnectionCleanup=true [1] (I put >>>>>>>>>>>>> the mysql >>>>>>>>>>>>> jar in the lib folder). >>>>>>>>>>>>> >>>>>>>>>>>>> I tried to set in the flink-conf.yml >>>>>>>>>>>>> env.java.opts: >>>>>>>>>>>>> "-Dcom.mysql.disableAbandonedConnectionCleanup=true" >>>>>>>>>>>>> >>>>>>>>>>>>> but the property does not produce the desired effect in the >>>>>>>>>>>>> static >>>>>>>>>>>>> section of such a thread [2] (I verified that attaching the >>>>>>>>>>>>> remote >>>>>>>>>>>>> debugger to the task manager). >>>>>>>>>>>>> >>>>>>>>>>>>> How can I fix this problem? >>>>>>>>>>>>> >>>>>>>>>>>>> [1] >>>>>>>>>>>>> https://dev.mysql.com/doc/relnotes/connector-j/8.0/en/news-8-0-22.html >>>>>>>>>>>>> [2] >>>>>>>>>>>>> public class AbandonedConnectionCleanupThread implements >>>>>>>>>>>>> Runnable {8898 7324 0768 >>>>>>>>>>>>> private static boolean abandonedConnectionCleanupDisabled = >>>>>>>>>>>>> >>>>>>>>>>>>> Boolean.getBoolean(PropertyDefinitions.SYSP_disableAbandonedConnectionCleanup); >>>>>>>>>>>>> >>>>>>>>>>>>> static { >>>>>>>>>>>>> if (abandonedConnectionCleanupDisabled) { >>>>>>>>>>>>> cleanupThreadExecutorService = null; >>>>>>>>>>>>> } else { >>>>>>>>>>>>> cleanupThreadExecutorService = >>>>>>>>>>>>> Executors.newSingleThreadExecutor(r -> {....} >>>>>>>>>>>>> } >>>>>>>>>>>>> } >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> Arvid Heise | Senior Java Developer >>>>>>>>>>>> >>>>>>>>>>>> <https://www.ververica.com/> >>>>>>>>>>>> >>>>>>>>>>>> Follow us @VervericaData >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> Join Flink Forward <https://flink-forward.org/> - The Apache >>>>>>>>>>>> Flink Conference >>>>>>>>>>>> >>>>>>>>>>>> Stream Processing | Event Driven | Real Time >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Ververica GmbH >>>>>>>>>>>> Registered at Amtsgericht Charlottenburg: HRB 158244 B >>>>>>>>>>>> Managing Directors: Timothy Alexander Steinert, Yip Park Tung >>>>>>>>>>>> Jason, Ji (Toni) Cheng >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> Arvid Heise | Senior Java Developer >>>>>>>>>> >>>>>>>>>> <https://www.ververica.com/> >>>>>>>>>> >>>>>>>>>> Follow us @VervericaData >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> Join Flink Forward <https://flink-forward.org/> - The Apache >>>>>>>>>> Flink Conference >>>>>>>>>> >>>>>>>>>> Stream Processing | Event Driven | Real Time >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Ververica GmbH >>>>>>>>>> Registered at Amtsgericht Charlottenburg: HRB 158244 B >>>>>>>>>> Managing Directors: Timothy Alexander Steinert, Yip Park Tung >>>>>>>>>> Jason, Ji (Toni) Cheng >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Arvid Heise | Senior Java Developer >>>>>>>> >>>>>>>> <https://www.ververica.com/> >>>>>>>> >>>>>>>> Follow us @VervericaData >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink >>>>>>>> Conference >>>>>>>> >>>>>>>> Stream Processing | Event Driven | Real Time >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany >>>>>>>> >>>>>>>> -- >>>>>>>> Ververica GmbH >>>>>>>> Registered at Amtsgericht Charlottenburg: HRB 158244 B >>>>>>>> Managing Directors: Timothy Alexander Steinert, Yip Park Tung >>>>>>>> Jason, Ji (Toni) Cheng >>>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Arvid Heise | Senior Java Developer >>>>>> >>>>>> <https://www.ververica.com/> >>>>>> >>>>>> Follow us @VervericaData >>>>>> >>>>>> -- >>>>>> >>>>>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink >>>>>> Conference >>>>>> >>>>>> Stream Processing | Event Driven | Real Time >>>>>> >>>>>> -- >>>>>> >>>>>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany >>>>>> >>>>>> -- >>>>>> Ververica GmbH >>>>>> Registered at Amtsgericht Charlottenburg: HRB 158244 B >>>>>> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, >>>>>> Ji (Toni) Cheng >>>>>> >>>>> >>> >>> -- >>> >>> Arvid Heise | Senior Java Developer >>> >>> <https://www.ververica.com/> >>> >>> Follow us @VervericaData >>> >>> -- >>> >>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink >>> Conference >>> >>> Stream Processing | Event Driven | Real Time >>> >>> -- >>> >>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany >>> >>> -- >>> Ververica GmbH >>> Registered at Amtsgericht Charlottenburg: HRB 158244 B >>> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji >>> (Toni) Cheng >>> >> > > -- > > Arvid Heise | Senior Java Developer > > <https://www.ververica.com/> > > Follow us @VervericaData > > -- > > Join Flink Forward <https://flink-forward.org/> - The Apache Flink > Conference > > Stream Processing | Event Driven | Real Time > > -- > > Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany > > -- > Ververica GmbH > Registered at Amtsgericht Charlottenburg: HRB 158244 B > Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji > (Toni) Cheng >
