Good morning David, Unfortunately, I don't believe I have the depth-of-knowledge of how TomEE's CDI is interacting with CXF and Johnzon to evaluate what the proper solution should be, in this case. In our use case, turning off Johnzon's CDI integration makes the memory leak go away, however, I'm concerned about possible side-effects of doing so -- our code is not injecting any of the JSON-B objects, and we're not using any custom adapters, so it's possible the change causes some regression there. Also, in the JOHNZON-161 bug report I linked in my original e-mail, a Johnzon maintainer expressed concern about disabling CDI integration could still lead to memory leaks in some circumstances:
> Just a side note for future readers: it is not "closed automatically", just > let to the garbage collector (so some advanced usages of the underlying > mapper - addCloseable - or other features can still leak). It is unlikely but > can happen and I don't want readers assume there is any magic there. Would someone with more expertise in this subject area be able to take a look, please? If it turns out that the proper solution *is* to turn off Johnzon's CDI integration for the JAX-RS provider, I can send you a one-liner PR for that? Thank you, David -----Original Message----- From: David Blevins <[email protected]> Sent: Wednesday, February 8, 2023 4:03 PM To: [email protected] Subject: Re: CXF JAXRSClientFactoryBean, Johnzon JSON-B provider, and a CDI memory leak on TomEE 8.x Hey David, Looks like you've done some very great investigation and you're pretty deep into the code! The best path forward would be to get a PR to the main TomEE branch and once that's looking good some help backporting it to TomEE 8 and 9. Does that sound like something you'd be able to help with? If, so here's the source in Github: - https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Ftomee&data=05%7C01%7CDEBERHAR%40nycourts.gov%7Ce70bdb3db1dd4a79b89308db0a17e04a%7C3456fe92cbd1406db5a35364bec0a833%7C0%7C0%7C638114869898134445%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ha2A7c4j%2BmsCMRXm0sIU%2B6bqu2tlT%2F4lXtmiyGDz8rA%3D&reserved=0 - https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Ftomee%2Fpulls&data=05%7C01%7CDEBERHAR%40nycourts.gov%7Ce70bdb3db1dd4a79b89308db0a17e04a%7C3456fe92cbd1406db5a35364bec0a833%7C0%7C0%7C638114869898134445%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2F4RaJOEnP%2FXbDI13OLfz%2F%2FH8U0WGxC5cOgZ%2FDdjTnhE%3D&reserved=0 If you're not familiar with git and making PRs, we're very happy to help you through it - we're always willing to help new people get into the project! -David > On Feb 1, 2023, at 7:27 AM, David A. Eberhart <[email protected]> wrote: > > Good morning, > It appears that using JAXRSClientFactoryBean to create proxy-beans for JAX-RS > webservices from within a TomEE 8.x container triggers a rather severe memory > leak via Johnzon's JSON-B provider's CDI integration in TomEE 8.x (I did not > test 9.x). > > Specifically, if JAXRSClientFactoryBean is used while TomEEJsonbProvider (or > JsonbJaxrsProvider) is on the OpenEJB JAX-RS JSON providers list (either the > hard-coded default, or via 'openejb.jaxrs.jsonProviders'), then each call to > JAXRSClientFactoryBean.create() will cause a new JohnzonJsonb object to be > created and tracked by JohnzonCdiExtension. For example, the following code > is sufficient to trigger the issue if placed within a servlet class' doGet(): > JAXRSClientFactoryBean bean = new JAXRSClientFactoryBean(); > bean.setServiceClass(IJaxrsWebservice.class); > > bean.setAddress(https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1%3A8080%2Freproducer&data=05%7C01%7CDEBERHAR%40nycourts.gov%7Ce70bdb3db1dd4a79b89308db0a17e04a%7C3456fe92cbd1406db5a35364bec0a833%7C0%7C0%7C638114869898134445%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=u%2Bz8eJ5gnvpFJZZYetZGoLLFlUT%2FKCQ2iSlznG6Q0jM%3D&reserved=0 > > <https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1%3A8080%2Freproducer&data=05%7C01%7CDEBERHAR%40nycourts.gov%7Ce70bdb3db1dd4a79b89308db0a17e04a%7C3456fe92cbd1406db5a35364bec0a833%7C0%7C0%7C638114869898134445%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=u%2Bz8eJ5gnvpFJZZYetZGoLLFlUT%2FKCQ2iSlznG6Q0jM%3D&reserved=0>); > > final IJaxrsWebservice proxy = bean.create(IJaxrsWebservice.class); > > I have a sample project that includes this code and a mock JAX-RS webservice, > reproducing the issue; a copy of this is available in this GitHub repo: > https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdeberhar%2FTomeeJohnzonBugReproducer&data=05%7C01%7CDEBERHAR%40nycourts.gov%7Ce70bdb3db1dd4a79b89308db0a17e04a%7C3456fe92cbd1406db5a35364bec0a833%7C0%7C0%7C638114869898134445%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=pUJTBr97KqEVZ5x05fhX%2BqPQrS141Rwv0OEu2Fm9%2FvQ%3D&reserved=0 > > <https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdeberhar%2FTomeeJohnzonBugReproducer&data=05%7C01%7CDEBERHAR%40nycourts.gov%7Ce70bdb3db1dd4a79b89308db0a17e04a%7C3456fe92cbd1406db5a35364bec0a833%7C0%7C0%7C638114869898134445%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=pUJTBr97KqEVZ5x05fhX%2BqPQrS141Rwv0OEu2Fm9%2FvQ%3D&reserved=0>. > If I repeatedly call the servlet, I can exhaust a 1GB application heap in > ~4-6k GETs to the servlet -- the server reports OutOfMemoryErrors, and a heap > dump shows 924MB retained by the JohnzonCdiExtension. If it would be > helpful, I can provide a copy of this project (what's the best way to do so?). > > > > I believe the underlying cause is described in this Johnzon bug > report:https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FJOHNZON-161&data=05%7C01%7CDEBERHAR%40nycourts.gov%7Ce70bdb3db1dd4a79b89308db0a17e04a%7C3456fe92cbd1406db5a35364bec0a833%7C0%7C0%7C638114869898134445%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=cctDfaMZIa41hj%2Fkbp2uSO7XSQQiROAK1kZrNV40lAg%3D&reserved=0 > > <https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FJOHNZON-161&data=05%7C01%7CDEBERHAR%40nycourts.gov%7Ce70bdb3db1dd4a79b89308db0a17e04a%7C3456fe92cbd1406db5a35364bec0a833%7C0%7C0%7C638114869898134445%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=cctDfaMZIa41hj%2Fkbp2uSO7XSQQiROAK1kZrNV40lAg%3D&reserved=0>. > However, it seems the Johnzon developers believe their CDI integration is > behaving as expected, and that it is the responsibility of the integrating > container to cache and re-use Jsonb instances, writing, in part: > > Hi skay, first of all it is a bug in your application so best is to fix > > your coding style. Now you have a workaround: set johnzon.skip-cdi=true or > > johnzon.cdi.activated=false in the jsonbconfig properties or set property > > johnzon.factory to new SimpleJohnzonAdapterFactory(). > > Side note: in terms of performance, recreating Jsonb instances looses all > > caching and redo all the reflection so should be slow so in any case I > > recommend you to change the pattern you have (it is true for jackson too > > btw) > > It appears the Meecrowave developers made changes in their source code to > prevent this leak via Johnzon's JsonbJaxrsProvider's CDI integration when > used with CXF; > seehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FMEECROWAVE-104&data=05%7C01%7CDEBERHAR%40nycourts.gov%7Ce70bdb3db1dd4a79b89308db0a17e04a%7C3456fe92cbd1406db5a35364bec0a833%7C0%7C0%7C638114869898134445%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ZTIVakT5zOcu%2FeoagRQdKQf12m4t8cvrl%2FKcrn0EFEM%3D&reserved=0 > > <https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FMEECROWAVE-104&data=05%7C01%7CDEBERHAR%40nycourts.gov%7Ce70bdb3db1dd4a79b89308db0a17e04a%7C3456fe92cbd1406db5a35364bec0a833%7C0%7C0%7C638114869898134445%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ZTIVakT5zOcu%2FeoagRQdKQf12m4t8cvrl%2FKcrn0EFEM%3D&reserved=0> > or > https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fopenwebbeans-meecrowave%2Fcommit%2Feafd5b6eda81471c4abd61297a2455b2bca0a013&data=05%7C01%7CDEBERHAR%40nycourts.gov%7Ce70bdb3db1dd4a79b89308db0a17e04a%7C3456fe92cbd1406db5a35364bec0a833%7C0%7C0%7C638114869898134445%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=DLYd4ZKnWYZorixGA0%2FnG%2BH8tqE9zvE6tn16sjDmBd8%3D&reserved=0 > > <https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fopenwebbeans-meecrowave%2Fcommit%2Feafd5b6eda81471c4abd61297a2455b2bca0a013&data=05%7C01%7CDEBERHAR%40nycourts.gov%7Ce70bdb3db1dd4a79b89308db0a17e04a%7C3456fe92cbd1406db5a35364bec0a833%7C0%7C0%7C638114869898134445%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=DLYd4ZKnWYZorixGA0%2FnG%2BH8tqE9zvE6tn16sjDmBd8%3D&reserved=0>. > For our organization's use of TomEE, we've been able to hack around this > issue by subclassing TomEEJsonbProvider to set johnzon.cdi.activated=false, > and then overriding the default provider list via > openejb.jaxrs.jsonProviders. However, this is likely not a proper solution. > > What's the best way forward to addressing this properly? > > > > Thank you, > David Eberhart > NYS Unified Court System > Office of Court Administration > Division of Technology and Court Research Please be CAREFUL when clicking links or opening attachments from external senders.
