[jira] [Commented] (OFBIZ-10592) OutOfMemory and stucked JobPoller issue
[ https://issues.apache.org/jira/browse/OFBIZ-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16859500#comment-16859500 ] Taher Alkhateeb commented on OFBIZ-10592: - Hi Giulio, Awesome work, however, I'm having a hard time applying your patch because it's in the non-unified format. Can you re-create it with svn diff or diff -Naur instead? > OutOfMemory and stucked JobPoller issue > --- > > Key: OFBIZ-10592 > URL: https://issues.apache.org/jira/browse/OFBIZ-10592 > Project: OFBiz > Issue Type: Bug > Components: ALL APPLICATIONS >Affects Versions: Release Branch 13.07 >Reporter: Giulio Speri >Assignee: Giulio Speri >Priority: Critical > Attachments: Screenshot from 2019-04-20 02-32-37.png, > ShoppingListServices.patch, ShoppingListServices.patch, > alloc_tree_600k_12102018.png, jvm_ofbiz1_profi_telem.png, > jvm_prof_ofbiz1_telem2.png, ofbiz1_jvm_profil_nojobpoller.png, > order_properties.patch, recorder_object_600k_12102018.png, > telemetry_ovrl_600k_12102018.png > > > > This installation is composed by two instances of OFBiz (v13.07.03), served > via an Apache Tomcat webserver, along with a load balancer. > The database server is MariaDB. > > We had the first problems, about 3 weeks ago, when suddenly, the front1 > (ofbiz instance 1), stopped serving web requests; front2, instead, was still > working correctly. > > Obviously we checked the log files, and we saw that async services were > failing; the failure was accompanied by this error line: > > *_Thread "AsyncAppender-async" java.lang.OutOfMemoryError: GC overhead limit > exceeded_* > > We analyzed the situation with our system specialists, and they told us that > the application was highly stressing machine resources (cpu always at or near > 100%, RAM usage rapidly increasing), until the jvm run out of memory. > This "resource-high-consumption situation", occurred only when ofbiz1 > instance was started with the JobPoller enabled; if the JobPoller was not > enabled, ofbiz run with low resource usage. > > We then focused on the db, to check first of all the dimensions; the result > was disconcerting; 45GB, mainly divided on four tables: SERVER_HIT (about 18 > GB), VISIT (about 15 GB), ENTITY_SYNC_REMOVE (about 8 GB), VISITOR (about 2 > GB). > All the other tables had a size in the order of few MB each. > > The first thing we did, was to clear all those tables, reducing considerably > the db size. > After the cleaning, we tried to start ofbiz1 again, with the JobPoller > component enabled; this caused a lot of old scheduled/queued jobs, to execute. > Except than for the start-up time, the resource usage of the machine, > stabilized around normal to low values (cpu 1-10%). > Ofbiz seemed to work (web request was served), but we noticed that the > JobPoller did not schedule or run jobs, anymore. > The number of job in "Pending" state in the JobSandbox entity was small > (about 20); no Queued, no Failed, no jobs in other states. > In addition to this, unfortunately, after few hours, jvm run out of memory > again. > > Our jvm has an heap maximum size of 20GB ( we have 32GB on the machine), so > it's not so small, I think. > The next step we're going to do is set-up locally the application over the > same production db to see what happens. > > Now that I explained the situation, I am going to ask if, in your > opinion/experience: > > Could the JobPoller component be the root (and only) cause of the OutOfMemory > of the jvm? > > Could this issue be related to OFBIZ-5710? > > Dumping and analyzing the heap of the jvm could help in some way to > understand what or who fills the memory or is this operation a waste of time? > > Is there something that we did not considered or missed during the whole > process of problem analysis? > > > I really thank you all for your attention and your help; any suggestion or > advice would really be greatly appreciated. > > Kind regards, > Giulio -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (OFBIZ-10966) JSON entity data import and export utility
[
https://issues.apache.org/jira/browse/OFBIZ-10966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jayansh Shinde reassigned OFBIZ-10966:
--
Assignee: Jayansh Shinde
> JSON entity data import and export utility
> --
>
> Key: OFBIZ-10966
> URL: https://issues.apache.org/jira/browse/OFBIZ-10966
> Project: OFBiz
> Issue Type: New Feature
> Components: framework
>Reporter: Jayansh Shinde
>Assignee: Jayansh Shinde
>Priority: Minor
>
> Currently, we support import/export entity data in XML format.
> Nowadays JSON is widely used in industry, we can have support for JSON
> format which looks quite similar to XML support.
> Here is example of XML data and it's JSON version
> {code:java}
>
> {code}
> {code:java}
> {“Party”:
> {"partyId":"123456","partyTypeId":"PERSON","statusId":"PARTY_ENABLED”}}
> {code}
>
> *Design Proposal*
> We can write *entityImportJson* and *entityImportDirJson* services for
> importing JSON from screen and directory respectively.
> And the *entityExportAllJson* service for exporting entity data in JSON.
>
> *Import Design*
> The import service will perform following operations:
> 1.) Validate the input JSON data (I am in process of exploring the way for
> this)
> 2.) On successful validation, convert JSON to OFBiz's entity model
> (GenericValue)
> 3.) The GenericValue will be inserted in database by some handler class for
> e.g we can write JsonDataHandler, it will convert given JSON to
> List, and finally write it to database (Similar pattern is used
> in XML import).
>
> *Export Design*
> Based on existing XML pattern the writeXmlText method of GenericEntity class
> write the exported data in XML format.
> In the similar way, we can implement writeJsonText to export data in JSON
> format.
> Please free feel to share your thought.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OFBIZ-10592) OutOfMemory and stucked JobPoller issue
[ https://issues.apache.org/jira/browse/OFBIZ-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16859466#comment-16859466 ] Giulio Speri commented on OFBIZ-10592: -- Hi everyone, sorry for the latency, but I can upload the patch files that make the service avoid the OOM problems. Just few notes on the solution I have thought: * shopping list are looped and processed at 1000 at a time; potentially, on very large amount of lists the service could run for quite a lot. To avoid this, I used an internal timer, with the same value of the service transaction timeout, that is checked for expiration at the end of each block of SL processed.Doing this way, if the timer expires in the middle of a processing block, the service won't stop immediately, but it will finish to loop the block. * I noticed that if a small timer is configured in the service definition (I tested 10 secs), when the timer expires and the parent transaction is resumed, the transaction timeout will light up.Also in this case, the first block of 1000 SL is processed anyway; * The number of SL retrieved at a time is configurable in the order.properties file. I already installed this solution on our customer's servers and it is doing it's work; I just reduced the days of the validity of the SL down to 5. Let me know if you encounter problems on applying the patch files. [^ShoppingListServices.patch] [^order_properties.patch] > OutOfMemory and stucked JobPoller issue > --- > > Key: OFBIZ-10592 > URL: https://issues.apache.org/jira/browse/OFBIZ-10592 > Project: OFBiz > Issue Type: Bug > Components: ALL APPLICATIONS >Affects Versions: Release Branch 13.07 >Reporter: Giulio Speri >Assignee: Giulio Speri >Priority: Critical > Attachments: Screenshot from 2019-04-20 02-32-37.png, > ShoppingListServices.patch, ShoppingListServices.patch, > alloc_tree_600k_12102018.png, jvm_ofbiz1_profi_telem.png, > jvm_prof_ofbiz1_telem2.png, ofbiz1_jvm_profil_nojobpoller.png, > order_properties.patch, recorder_object_600k_12102018.png, > telemetry_ovrl_600k_12102018.png > > > > This installation is composed by two instances of OFBiz (v13.07.03), served > via an Apache Tomcat webserver, along with a load balancer. > The database server is MariaDB. > > We had the first problems, about 3 weeks ago, when suddenly, the front1 > (ofbiz instance 1), stopped serving web requests; front2, instead, was still > working correctly. > > Obviously we checked the log files, and we saw that async services were > failing; the failure was accompanied by this error line: > > *_Thread "AsyncAppender-async" java.lang.OutOfMemoryError: GC overhead limit > exceeded_* > > We analyzed the situation with our system specialists, and they told us that > the application was highly stressing machine resources (cpu always at or near > 100%, RAM usage rapidly increasing), until the jvm run out of memory. > This "resource-high-consumption situation", occurred only when ofbiz1 > instance was started with the JobPoller enabled; if the JobPoller was not > enabled, ofbiz run with low resource usage. > > We then focused on the db, to check first of all the dimensions; the result > was disconcerting; 45GB, mainly divided on four tables: SERVER_HIT (about 18 > GB), VISIT (about 15 GB), ENTITY_SYNC_REMOVE (about 8 GB), VISITOR (about 2 > GB). > All the other tables had a size in the order of few MB each. > > The first thing we did, was to clear all those tables, reducing considerably > the db size. > After the cleaning, we tried to start ofbiz1 again, with the JobPoller > component enabled; this caused a lot of old scheduled/queued jobs, to execute. > Except than for the start-up time, the resource usage of the machine, > stabilized around normal to low values (cpu 1-10%). > Ofbiz seemed to work (web request was served), but we noticed that the > JobPoller did not schedule or run jobs, anymore. > The number of job in "Pending" state in the JobSandbox entity was small > (about 20); no Queued, no Failed, no jobs in other states. > In addition to this, unfortunately, after few hours, jvm run out of memory > again. > > Our jvm has an heap maximum size of 20GB ( we have 32GB on the machine), so > it's not so small, I think. > The next step we're going to do is set-up locally the application over the > same production db to see what happens. > > Now that I explained the situation, I am going to ask if, in your > opinion/experience: > > Could the JobPoller component be the root (and only) cause of the OutOfMemory > of the jvm? > > Could this issue be related to OFBIZ-5710? > > Dumping and analyzing the heap of the jvm could help in some way to > understand what or who fills the memory or is this operation
[jira] [Updated] (OFBIZ-10592) OutOfMemory and stucked JobPoller issue
[ https://issues.apache.org/jira/browse/OFBIZ-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giulio Speri updated OFBIZ-10592: - Attachment: ShoppingListServices.patch > OutOfMemory and stucked JobPoller issue > --- > > Key: OFBIZ-10592 > URL: https://issues.apache.org/jira/browse/OFBIZ-10592 > Project: OFBiz > Issue Type: Bug > Components: ALL APPLICATIONS >Affects Versions: Release Branch 13.07 >Reporter: Giulio Speri >Assignee: Giulio Speri >Priority: Critical > Attachments: Screenshot from 2019-04-20 02-32-37.png, > ShoppingListServices.patch, ShoppingListServices.patch, > alloc_tree_600k_12102018.png, jvm_ofbiz1_profi_telem.png, > jvm_prof_ofbiz1_telem2.png, ofbiz1_jvm_profil_nojobpoller.png, > order_properties.patch, recorder_object_600k_12102018.png, > telemetry_ovrl_600k_12102018.png > > > > This installation is composed by two instances of OFBiz (v13.07.03), served > via an Apache Tomcat webserver, along with a load balancer. > The database server is MariaDB. > > We had the first problems, about 3 weeks ago, when suddenly, the front1 > (ofbiz instance 1), stopped serving web requests; front2, instead, was still > working correctly. > > Obviously we checked the log files, and we saw that async services were > failing; the failure was accompanied by this error line: > > *_Thread "AsyncAppender-async" java.lang.OutOfMemoryError: GC overhead limit > exceeded_* > > We analyzed the situation with our system specialists, and they told us that > the application was highly stressing machine resources (cpu always at or near > 100%, RAM usage rapidly increasing), until the jvm run out of memory. > This "resource-high-consumption situation", occurred only when ofbiz1 > instance was started with the JobPoller enabled; if the JobPoller was not > enabled, ofbiz run with low resource usage. > > We then focused on the db, to check first of all the dimensions; the result > was disconcerting; 45GB, mainly divided on four tables: SERVER_HIT (about 18 > GB), VISIT (about 15 GB), ENTITY_SYNC_REMOVE (about 8 GB), VISITOR (about 2 > GB). > All the other tables had a size in the order of few MB each. > > The first thing we did, was to clear all those tables, reducing considerably > the db size. > After the cleaning, we tried to start ofbiz1 again, with the JobPoller > component enabled; this caused a lot of old scheduled/queued jobs, to execute. > Except than for the start-up time, the resource usage of the machine, > stabilized around normal to low values (cpu 1-10%). > Ofbiz seemed to work (web request was served), but we noticed that the > JobPoller did not schedule or run jobs, anymore. > The number of job in "Pending" state in the JobSandbox entity was small > (about 20); no Queued, no Failed, no jobs in other states. > In addition to this, unfortunately, after few hours, jvm run out of memory > again. > > Our jvm has an heap maximum size of 20GB ( we have 32GB on the machine), so > it's not so small, I think. > The next step we're going to do is set-up locally the application over the > same production db to see what happens. > > Now that I explained the situation, I am going to ask if, in your > opinion/experience: > > Could the JobPoller component be the root (and only) cause of the OutOfMemory > of the jvm? > > Could this issue be related to OFBIZ-5710? > > Dumping and analyzing the heap of the jvm could help in some way to > understand what or who fills the memory or is this operation a waste of time? > > Is there something that we did not considered or missed during the whole > process of problem analysis? > > > I really thank you all for your attention and your help; any suggestion or > advice would really be greatly appreciated. > > Kind regards, > Giulio -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OFBIZ-10592) OutOfMemory and stucked JobPoller issue
[ https://issues.apache.org/jira/browse/OFBIZ-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giulio Speri updated OFBIZ-10592: - Attachment: ShoppingListServices.patch order_properties.patch > OutOfMemory and stucked JobPoller issue > --- > > Key: OFBIZ-10592 > URL: https://issues.apache.org/jira/browse/OFBIZ-10592 > Project: OFBiz > Issue Type: Bug > Components: ALL APPLICATIONS >Affects Versions: Release Branch 13.07 >Reporter: Giulio Speri >Assignee: Giulio Speri >Priority: Critical > Attachments: Screenshot from 2019-04-20 02-32-37.png, > ShoppingListServices.patch, alloc_tree_600k_12102018.png, > jvm_ofbiz1_profi_telem.png, jvm_prof_ofbiz1_telem2.png, > ofbiz1_jvm_profil_nojobpoller.png, order_properties.patch, > recorder_object_600k_12102018.png, telemetry_ovrl_600k_12102018.png > > > > This installation is composed by two instances of OFBiz (v13.07.03), served > via an Apache Tomcat webserver, along with a load balancer. > The database server is MariaDB. > > We had the first problems, about 3 weeks ago, when suddenly, the front1 > (ofbiz instance 1), stopped serving web requests; front2, instead, was still > working correctly. > > Obviously we checked the log files, and we saw that async services were > failing; the failure was accompanied by this error line: > > *_Thread "AsyncAppender-async" java.lang.OutOfMemoryError: GC overhead limit > exceeded_* > > We analyzed the situation with our system specialists, and they told us that > the application was highly stressing machine resources (cpu always at or near > 100%, RAM usage rapidly increasing), until the jvm run out of memory. > This "resource-high-consumption situation", occurred only when ofbiz1 > instance was started with the JobPoller enabled; if the JobPoller was not > enabled, ofbiz run with low resource usage. > > We then focused on the db, to check first of all the dimensions; the result > was disconcerting; 45GB, mainly divided on four tables: SERVER_HIT (about 18 > GB), VISIT (about 15 GB), ENTITY_SYNC_REMOVE (about 8 GB), VISITOR (about 2 > GB). > All the other tables had a size in the order of few MB each. > > The first thing we did, was to clear all those tables, reducing considerably > the db size. > After the cleaning, we tried to start ofbiz1 again, with the JobPoller > component enabled; this caused a lot of old scheduled/queued jobs, to execute. > Except than for the start-up time, the resource usage of the machine, > stabilized around normal to low values (cpu 1-10%). > Ofbiz seemed to work (web request was served), but we noticed that the > JobPoller did not schedule or run jobs, anymore. > The number of job in "Pending" state in the JobSandbox entity was small > (about 20); no Queued, no Failed, no jobs in other states. > In addition to this, unfortunately, after few hours, jvm run out of memory > again. > > Our jvm has an heap maximum size of 20GB ( we have 32GB on the machine), so > it's not so small, I think. > The next step we're going to do is set-up locally the application over the > same production db to see what happens. > > Now that I explained the situation, I am going to ask if, in your > opinion/experience: > > Could the JobPoller component be the root (and only) cause of the OutOfMemory > of the jvm? > > Could this issue be related to OFBIZ-5710? > > Dumping and analyzing the heap of the jvm could help in some way to > understand what or who fills the memory or is this operation a waste of time? > > Is there something that we did not considered or missed during the whole > process of problem analysis? > > > I really thank you all for your attention and your help; any suggestion or > advice would really be greatly appreciated. > > Kind regards, > Giulio -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OFBIZ-5048) Multi Part Input Parameters not Available in Groovy Event
[ https://issues.apache.org/jira/browse/OFBIZ-5048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16859401#comment-16859401 ] Jacques Le Roux commented on OFBIZ-5048: Hi Michael, Actually this line was copied as is from [ServiceEventHandler|http://svn.apache.org/viewvc/ofbiz/ofbiz-framework/trunk/framework/webapp/src/main/java/org/apache/ofbiz/webapp/event/ServiceEventHandler.java?r1=1847478&r2=1847477&pathrev=1847478]. So it's hard to tell w/o going into history. I did it for you: it's like that from pre Apache era. So it's your decision if nobody answer... > Multi Part Input Parameters not Available in Groovy Event > - > > Key: OFBIZ-5048 > URL: https://issues.apache.org/jira/browse/OFBIZ-5048 > Project: OFBiz > Issue Type: New Feature > Components: ALL COMPONENTS >Affects Versions: Trunk >Reporter: Vikramjit Singh >Assignee: Jacques Le Roux >Priority: Major > Fix For: 17.12.01 > > Attachments: GroovyEventMultipartParametes.patch, OFBIZ-5048.patch, > OFBIZ-5048V2.patch > > > If form is of type enctype="multipart/form-data" and we are handling it's > submission through Groovy Event then in the parameters Map Ofbiz does not set > the multipart input parameters from request parameters. > The same are available when multipart form submission is handled through > service. > The reason being the code that sets the multipart parameters in request > attribute is only available in ServiceEventHandler.java and in > GroovyEventHandler the multipart are never set. > So I have created a method getMultiPartParameterMap in the class > UtilHttp.java and put the common logic in that method so that when > getCombinedMap method is called from the GroovEventHandler the method also > call getMultiPartParameterMap and in the ServiceEventHandler I have written a > call for getMultiPartParameterMap method. > I am attaching the Patch Kindly Verify the same. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OFBIZ-11094) Buildbot RAT for releases branches
[ https://issues.apache.org/jira/browse/OFBIZ-11094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacques Le Roux updated OFBIZ-11094: Description: For the moment, only the trunk is verified by RAT (Apache Creadur). The initial idea was that once valided in trunk there should not be issues in releases branches, but better be safe than sorry! (was: For the moment, only the trunk is verified by RAT (Apache Creadur). The initial idea was that once valided in trunk there should not be issue in releases branches, but better be safe than sorry!) > Buildbot RAT for releases branches > -- > > Key: OFBIZ-11094 > URL: https://issues.apache.org/jira/browse/OFBIZ-11094 > Project: OFBiz > Issue Type: New Feature > Components: BuildBot >Affects Versions: Release Branch 16.11, Release Branch 17.12, Release > Branch 18.12 >Reporter: Jacques Le Roux >Assignee: Jacques Le Roux >Priority: Minor > > For the moment, only the trunk is verified by RAT (Apache Creadur). The > initial idea was that once valided in trunk there should not be issues in > releases branches, but better be safe than sorry! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OFBIZ-11014) StringUtil cleanup
[
https://issues.apache.org/jira/browse/OFBIZ-11014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16859399#comment-16859399
]
Jacques Le Roux commented on OFBIZ-11014:
-
+1
> StringUtil cleanup
> --
>
> Key: OFBIZ-11014
> URL: https://issues.apache.org/jira/browse/OFBIZ-11014
> Project: OFBiz
> Issue Type: Improvement
> Components: framework
>Affects Versions: Trunk
>Reporter: Mathieu Lirzin
>Assignee: Mathieu Lirzin
>Priority: Minor
> Attachments:
> OFBIZ-11014_0001-Remove-useless-StringUtil-join-overload.patch,
> OFBIZ-11014_0002-Rewrite-StringUtil-join-method.patch,
> OFBIZ-11014_0003-Remove-unused-StringUtil-append-method.patch,
> OFBIZ-11014_0004-Remove-unused-StringUtil-split-overload.patch,
> OFBIZ-11014_0005-Inline-StringUtil-quoteStrList-method.patch,
> OFBIZ-11014_0006-Remove-unused-StringUtil-strToMap-overload.patch,
> OFBIZ-11014_0007-Remove-unused-StringUtil-mapToStr-method.patch,
> OFBIZ-11014_0008-Remove-unused-StringUtil-convertChar-method.patch,
> OFBIZ-11014_0009-Remove-unused-StringUtil-removeNumeric-meth.patch,
> OFBIZ-11014_0010-Remove-unused-StringUtil-collapseNewlines-m.patch,
> OFBIZ-11014_0011-Remove-unused-StringUtil-collapseSpaces-met.patch,
> OFBIZ-11014_0012-Remove-unused-StringUtil-collapseCharacter.patch,
> OFBIZ-11014_0013-Remove-unused-StringUtil-appendTo-methods.patch,
> OFBIZ-11014_0014-Turn-StringUtilTests-into-a-unit-test.patch,
> OFBIZ-11014_0015-Inline-StringUtil-toMap.patch
>
>
> {{StringUtil}} contains stuff that is not useful with recent version of Java.
> For example the {{StringUtil#split}} method could be replaced by the
> {{String#split}} method. As a consequence {{StringUtil}} should be cleaned up.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
