[
https://issues.apache.org/jira/browse/OFBIZ-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16831659#comment-16831659
]
Giulio Speri commented on OFBIZ-10592:
--------------------------------------
Hi Taher,
the OOM is for sure caused by the service "autoDeleteAutoSaveShoppingList",
because it tries to load a huge list of GenericValue of Shopping Lists records,
with the method delegator.findList; with millions of record in the ShoppingList
table, sooner or later, the Heap Space of the JVM will be full.
I have developed a solution for this, that I'll try to upload in next days :
basically I do not use the findList which returns a List<GenericValue>, but
I'll use the delegator.find which return an EntitiyListIterator and then I use
it to create a sub list of element (as I wrote on my last comment, I created a
list of 1000 records), and then I'll work on that list.
This configuration works with ofbiz default heap memory settings and 1M+
records in the Shopping List table.
This process goes on until there are no more value suitable for deletion.
I'm not sure if there are issues with the JobPoller, but in our production
environment we still have problems with scheduled services, but maybe it's a
completely different case, but I didn't have so much time to dig further.
> OutOfMemory and stucked JobPoller issue
> ---------------------------------------
>
> Key: OFBIZ-10592
> URL: https://issues.apache.org/jira/browse/OFBIZ-10592
> Project: OFBiz
> Issue Type: Bug
> Components: ALL APPLICATIONS
> Affects Versions: Release Branch 13.07
> Reporter: Giulio Speri
> Assignee: Giulio Speri
> Priority: Critical
> Attachments: Screenshot from 2019-04-20 02-32-37.png,
> alloc_tree_600k_12102018.png, jvm_ofbiz1_profi_telem.png,
> jvm_prof_ofbiz1_telem2.png, ofbiz1_jvm_profil_nojobpoller.png,
> recorder_object_600k_12102018.png, telemetry_ovrl_600k_12102018.png
>
>
>
> This installation is composed by two instances of OFBiz (v13.07.03), served
> via an Apache Tomcat webserver, along with a load balancer.
> The database server is MariaDB.
>
> We had the first problems, about 3 weeks ago, when suddenly, the front1
> (ofbiz instance 1), stopped serving web requests; front2, instead, was still
> working correctly.
>
> Obviously we checked the log files, and we saw that async services were
> failing; the failure was accompanied by this error line:
>
> *_Thread "AsyncAppender-async" java.lang.OutOfMemoryError: GC overhead limit
> exceeded_*
>
> We analyzed the situation with our system specialists, and they told us that
> the application was highly stressing machine resources (cpu always at or near
> 100%, RAM usage rapidly increasing), until the jvm run out of memory.
> This "resource-high-consumption situation", occurred only when ofbiz1
> instance was started with the JobPoller enabled; if the JobPoller was not
> enabled, ofbiz run with low resource usage.
>
> We then focused on the db, to check first of all the dimensions; the result
> was disconcerting; 45GB, mainly divided on four tables: SERVER_HIT (about 18
> GB), VISIT (about 15 GB), ENTITY_SYNC_REMOVE (about 8 GB), VISITOR (about 2
> GB).
> All the other tables had a size in the order of few MB each.
>
> The first thing we did, was to clear all those tables, reducing considerably
> the db size.
> After the cleaning, we tried to start ofbiz1 again, with the JobPoller
> component enabled; this caused a lot of old scheduled/queued jobs, to execute.
> Except than for the start-up time, the resource usage of the machine,
> stabilized around normal to low values (cpu 1-10%).
> Ofbiz seemed to work (web request was served), but we noticed that the
> JobPoller did not schedule or run jobs, anymore.
> The number of job in "Pending" state in the JobSandbox entity was small
> (about 20); no Queued, no Failed, no jobs in other states.
> In addition to this, unfortunately, after few hours, jvm run out of memory
> again.
>
> Our jvm has an heap maximum size of 20GB ( we have 32GB on the machine), so
> it's not so small, I think.
> The next step we're going to do is set-up locally the application over the
> same production db to see what happens.
>
> Now that I explained the situation, I am going to ask if, in your
> opinion/experience:
>
> Could the JobPoller component be the root (and only) cause of the OutOfMemory
> of the jvm?
>
> Could this issue be related to OFBIZ-5710?
>
> Dumping and analyzing the heap of the jvm could help in some way to
> understand what or who fills the memory or is this operation a waste of time?
>
> Is there something that we did not considered or missed during the whole
> process of problem analysis?
>
>
> I really thank you all for your attention and your help; any suggestion or
> advice would really be greatly appreciated.
>
> Kind regards,
> Giulio
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
