I fired off profiler. After couple of uploads of WAR file: https://imgur.com/a/1BNM1fr
One of the threads that shows as GC root for stale web app class loader: https://imgur.com/a/U9s4Y0V ... now i hope i have something useful. Regards On Fri, Jun 26, 2020 at 3:36 PM Jan Bartel <j...@webtide.com> wrote: > I wonder if this is the same issue that was reported a few years ago, that > we could never get to the bottom of: > https://github.com/eclipse/jetty.project/issues/1425 > > Jan > > On Fri, 26 Jun 2020 at 15:06, Joakim Erdfelt <joa...@webtide.com> wrote: > >> Hot reload on Java has many pitfalls. >> >> Most of them come from WebApp ClassLoader Pinning. >> Where a class being loaded from the WebApp is keeping a reference to the >> WebApp ClassLoader, preventing it from being recycled. >> >> We have many LeakPreventers implemented that prevent the WebApp >> ClassLoader Pinning, for the common JVM and Servlet/Java EE classes, but >> there are still more in various other 3rd party libraries. >> >> See >> https://www.eclipse.org/jetty/documentation/current/preventing-memory-leaks.html >> >> Give them a try, if it works out for you GREAT. >> But if you still have this issue, then you'll have to figure out which >> class is holding the reference to the WebAppClassLoader and perhaps >> implement a LeakPreventere for it to let you have hot reload success. >> >> Joakim Erdfelt / joa...@webtide.com >> >> >> On Fri, Jun 26, 2020 at 7:54 AM Kiao Desouzza <kiao.desou...@gmail.com> >> wrote: >> >>> Hi, >>> >>> I am having an issue with file system quota tools not reporting >>> correctly on the directory where jetty is deployed. After some >>> investigation I think it's down to left over file descriptors like these >>> (output from lsof): >>> >>> java 39467 sclient 5r REG >>> 8,1 134017 101365281 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/HikariCP-java7-2.4.13.jar >>> (deleted) >>> java 39467 sclient 5r REG >>> 8,1 134017 101365281 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/HikariCP-java7-2.4.13.jar >>> (deleted) >>> java 39467 sclient 45r REG >>> 8,1 69409 101365257 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/activation-1.1.1.jar >>> (deleted) >>> java 39467 sclient 46r REG >>> 8,1 267827 101365259 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/admin_2.13.jar >>> (deleted) >>> java 39467 sclient 47r REG >>> 8,1 499844 101365280 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/c3p0-0.9.5.4.jar >>> (deleted) >>> java 39467 sclient 48r REG >>> 8,1 246918 101365265 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/commons-beanutils-1.9.4.jar >>> (deleted) >>> java 39467 sclient 49r REG >>> 8,1 347669 101365260 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/commons-codec-1.14.jar >>> (deleted) >>> java 39467 sclient 50r REG >>> 8,1 588337 101365276 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/commons-collections-3.2.2.jar >>> (deleted) >>> java 39467 sclient 51r REG >>> 8,1 208475 101365250 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/commons-dbcp2-2.7.0.jar >>> (deleted) >>> java 39467 sclient 52r REG >>> 8,1 146108 101365282 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/commons-digester-1.8.1.jar >>> (deleted) >>> java 39467 sclient 53r REG >>> 8,1 214788 101365256 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/commons-io-2.6.jar >>> (deleted) >>> java 39467 sclient 54r REG >>> 8,1 61829 101365278 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/commons-logging-1.2.jar >>> (deleted) >>> java 39467 sclient 55r REG >>> 8,1 129592 101365268 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/commons-pool2-2.7.0.jar >>> (deleted) >>> java 39467 sclient 56r REG >>> 8,1 186077 102297951 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/commons-validator-1.6.jar >>> (deleted) >>> java 39467 sclient 57r REG >>> 8,1 580870 101365271 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/flyway-core-6.3.3.jar >>> (deleted) >>> java 39467 sclient 58r REG >>> 8,1 1586948 101365273 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/freemarker-2.3.29.jar >>> (deleted) >>> java 39467 sclient 59r REG >>> 8,1 80511 101365272 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/geny_2.13-0.6.0.jar >>> (deleted) >>> java 39467 sclient 60r REG >>> 8,1 219146 101365253 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/javax.mail-api-1.6.2.jar >>> (deleted) >>> java 39467 sclient 61r REG >>> 8,1 25464 101365283 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/jetty-continuation-9.4.28.v20200408.jar >>> (deleted) >>> java 39467 sclient 62r REG >>> 8,1 624036 101365275 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/mchange-commons-java-0.2.15.jar >>> (deleted) >>> java 39467 sclient 63r REG >>> 8,1 927447 101365252 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/postgresql-42.2.10.jar >>> (deleted) >>> java 39467 sclient 64r REG >>> 8,1 134017 1352185 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-17785782056536538882.dir/webapp/WEB-INF/lib/HikariCP-java7-2.4.13.jar >>> (deleted) >>> java 39467 sclient 65r REG >>> 8,1 4250 101365255 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/scala-collection-compat_2.13-2.1.6.jar >>> (deleted) >>> java 39467 sclient 66r REG >>> 8,1 5865349 101365264 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/scala-library-2.13.2.jar >>> (deleted) >>> java 39467 sclient 67r REG >>> 8,1 225338 101365277 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/scala-parser-combinators_2.13-1.1.2.jar >>> (deleted) >>> java 39467 sclient 68r REG >>> 8,1 129113 101365279 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/scalaj-http_2.13-2.4.2.jar >>> (deleted) >>> java 39467 sclient 69r REG >>> 8,1 3625052 101365254 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/scala-reflect-2.13.2.jar >>> (deleted) >>> java 39467 sclient 70r REG >>> 8,1 1801315 101365266 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/scalikejdbc-core_2.13-3.4.2.jar >>> (deleted) >>> java 39467 sclient 71r REG >>> 8,1 6504 102297938 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/scalikejdbc-interpolation-macro_2.13-3.4.2.jar >>> (deleted) >>> java 39467 sclient 72r REG >>> 8,1 201322 101365263 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/scalikejdbc-interpolation_2.13-3.4.2.jar >>> (deleted) >>> java 39467 sclient 73r REG >>> 8,1 19036 101365261 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/scalikejdbc_2.13-3.4.2.jar >>> (deleted) >>> java 39467 sclient 74r REG >>> 8,1 41472 101365274 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/slf4j-api-1.7.30.jar >>> (deleted) >>> java 39467 sclient 75r REG >>> 8,1 35063 101365284 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/stats_collector_2.13-0.1.0-SNAPSHOT.jar >>> (deleted) >>> java 39467 sclient 76r REG >>> 8,1 187761 101365270 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/ujson_2.13-1.1.0.jar >>> (deleted) >>> java 39467 sclient 77r REG >>> 8,1 114284 101365269 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/upickle-core_2.13-1.1.0.jar >>> (deleted) >>> java 39467 sclient 94r REG >>> 8,1 69409 1220495 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-17785782056536538882.dir/webapp/WEB-INF/lib/activation-1.1.1.jar >>> (deleted) >>> java 39467 sclient 95r REG >>> 8,1 267827 1352165 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-17785782056536538882.dir/webapp/WEB-INF/lib/admin_2.13.jar >>> (deleted) >>> java 39467 sclient 96r REG >>> 8,1 499844 1352184 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-17785782056536538882.dir/webapp/WEB-INF/lib/c3p0-0.9.5.4.jar >>> (deleted) >>> java 39467 sclient 97r REG >>> 8,1 246918 1352170 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-17785782056536538882.dir/webapp/WEB-INF/lib/commons-beanutils-1.9.4.jar >>> (deleted) >>> java 39467 sclient 98r REG >>> 8,1 347669 1352166 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-17785782056536538882.dir/webapp/WEB-INF/lib/commons-codec-1.14.jar >>> (deleted) >>> java 39467 sclient 99r REG >>> 8,1 588337 1352180 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-17785782056536538882.dir/webapp/WEB-INF/lib/commons-collections-3.2.2.jar >>> (deleted) >>> java 39467 sclient 100r REG >>> 8,1 208475 957623 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-17785782056536538882.dir/webapp/WEB-INF/lib/commons-dbcp2-2.7.0.jar >>> (deleted) >>> java 39467 sclient 101r REG >>> 8,1 146108 1352186 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-17785782056536538882.dir/webapp/WEB-INF/lib/commons-digester-1.8.1.jar >>> (deleted) >>> java 39467 sclient 102r REG >>> 8,1 214788 1212954 >>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-17785782056536538882.dir/webapp/WEB-INF/lib/commons-io-2.6.jar >>> (deleted) >>> ... >>> >>> The file system tools(du) correctly report used disk space but file >>> system quota is not seeing space being released after each update of >>> deployed WAR file (used space grows). After I restart jetty file system >>> quota is updated correctly. It's only JAR files from what I can tell so >>> could it be related to the class loader in some way? >>> >>> FS is xfs on centos8 with pquota (project quota) >>> >>> I am using latest release 9.4.30.v20200611 >>> >>> Regards >>> _______________________________________________ >>> jetty-users mailing list >>> jetty-users@eclipse.org >>> To unsubscribe from this list, visit >>> https://www.eclipse.org/mailman/listinfo/jetty-users >>> >> _______________________________________________ >> jetty-users mailing list >> jetty-users@eclipse.org >> To unsubscribe from this list, visit >> https://www.eclipse.org/mailman/listinfo/jetty-users >> > > > -- > Jan Bartel <j...@webtide.com> > www.webtide.com > *Expert assistance from the creators of Jetty and CometD* > > _______________________________________________ > jetty-users mailing list > jetty-users@eclipse.org > To unsubscribe from this list, visit > https://www.eclipse.org/mailman/listinfo/jetty-users >
_______________________________________________ jetty-users mailing list jetty-users@eclipse.org To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-users
