Re: JobManager/JobPoller issues
+1 Jacques Le 25/02/2019 à 21:15, Mathieu Lirzin a écrit : Hello Scott, Scott Gray writes: I'm working on implementing the priority queue approach at the moment for a client. All things going well it will be in production in a couple of weeks and I'll report back then with a patch. Sounds great!
Re: Welcome to Mathieu Lirzin as new committer!
Hello, Taher Alkhateeb writes: > The OFBiz PMC has invited Mathieu Lirzin to become a new committer and > we are happy to announce that he has accepted this role. Thanks everybody for the congratulations and welcoming me as a committer in this project. -- Mathieu Lirzin GPG: F2A3 8D7E EB2B 6640 5761 070D 0ADE E100 9460 4D37
Re: JobManager/JobPoller issues
Hello Scott, Scott Gray writes: > I'm working on implementing the priority queue approach at the moment for a > client. All things going well it will be in production in a couple of > weeks and I'll report back then with a patch. Sounds great! -- Mathieu Lirzin GPG: F2A3 8D7E EB2B 6640 5761 070D 0ADE E100 9460 4D37
Re: JobManager/JobPoller issues
Hi Jacques, I'm working on implementing the priority queue approach at the moment for a client. All things going well it will be in production in a couple of weeks and I'll report back then with a patch. Regards Scott On Tue, 26 Feb 2019 at 03:11, Jacques Le Roux wrote: > Hi, > > I put this comment there with OFBIZ-10002 trying to document why we have 5 > as hardcoded value of /max-threads/ attribute in /thread-pool/ element > (serviceengine.xml). At this moment Scott already mentioned[1]: > > /Honestly I think the topic is generic enough that OFBiz doesn't need > to provide any information at all. Thread pool sizing is not exclusive to > OFBiz and it would be strange for anyone to modify the numbers without > first researching sources that provide far more detail than a few sentences > in our config files will ever cover./ > > I agree with Scott and Jacopo that jobs are more likely IO rather than CPU > bounded. So I agree that we should take that into account, change the > current algorithm and remove this somehow misleading comment. Scott's > suggestion in his 2nd email sounds good to me. So If I understood well we > could > use an unbounded but finally limited queue, like it was before. > > Although with all of that said, after a quick second look it appears > that > the current implementation doesn't try poll for more jobs than the > configured limit (minus already queued jobs) so we might be fine with > an > unbounded queue implementation. We'd just need to alter the call to > JobManager.poll(int limit) to not pass in > executor.getQueue().remainingCapacity() and instead pass in something > like > (threadPool.getJobs() - executor.getQueue().size()) > > I'm fine with that as it would continue to prevent hitting physical > limitations and can be tweaked by users as it's now. Note that it seems > though > uneasy to tweak as we received already several "complaints" about it. > > Now one of the advantage of a PriorityBlockingQueue is priority. And to > take advantage of that we can't rely on "/natural ordering"/ and need to > implement Comparable (which does no seem easy). Nicolas provided some > leads below and this should be discussed. The must would be to have that > parametrised, of course. > > My 2 cts > // > > [1] https://markmail.org/message/ixzluzd44rgloa2j > > Jacques > > Le 06/02/2019 à 14:24, Nicolas Malin a écrit : > > Hello Scott, > > > > On a customer project we use massively the job manager with an average > of one hundred thousand job per days. > > > > We have different cases like, huge long jobs, async persistent job, fast > regular job. The mainly problem that we detect has been (as you notified) > > the long jobs that stuck poller's thread and when we restart OFBiz (we > are on continuous delivery) we hadn't windows this without crash some jobs. > > > > To solve try with Gil to analyze if we can load some weighting on job > definition to help the job manager on what jobs on the pending queue it can > > push on queued queue. We changed own vision to create two pools, one for > system maintenance and huge long jobs managed by two ofbiz instances and an > > other to manage user activity jobs also managed by two instances. We > also added on service definition an information to indicate the > predilection pool > > > > This isn't a big deal and not resolve the stuck pool but all blocked > jobs aren't vital for daily activity. > > > > For crashed job, we introduced in trunk service lock that we set before > an update and wait a windows for the restart. > > > > At this time for all OOM detected we reanalyse the origin job and tried > to decompose it by persistent async service to help loading repartition. > > > > If I had more time, I would be oriented job improvement to : > > > > * Define an execution plan rule to link services and poller without > touch any service definition > > > > * Define configuration by instance for the job vacuum to refine by > service volumetric > > > > This feedback is a little confused Scott, maybe you found interesting > things > > > > Nicolas > > > > On 30/01/2019 20:47, Scott Gray wrote: > >> Hi folks, > >> > >> Just jotting down some issues with the JobManager over noticed over the > >> last few days: > >> 1. min-threads in serviceengine.xml is never exceeded unless the job > count > >> in the queue exceeds 5000 (or whatever is configured). Is this not > obvious > >> to anyone else? I don't think this was the behavior prior to a > refactoring > >> a few years ago. > >> 2. The advice on the number of threads to use doesn't seem good to me, > it > >> assumes your jobs are CPU bound when in my experience they are more > likely > >> to be I/O bound while making db or external API calls, sending emails > etc. > >> With the default setup, it only takes two long running jobs to > effectively > >> block the processing of any others until the queue hits 5000 and the > other > >> threads are finally opened up. If you're
Re: JobManager/JobPoller issues
Hi, I put this comment there with OFBIZ-10002 trying to document why we have 5 as hardcoded value of /max-threads/ attribute in /thread-pool/ element (serviceengine.xml). At this moment Scott already mentioned[1]: /Honestly I think the topic is generic enough that OFBiz doesn't need to provide any information at all. Thread pool sizing is not exclusive to OFBiz and it would be strange for anyone to modify the numbers without first researching sources that provide far more detail than a few sentences in our config files will ever cover./ I agree with Scott and Jacopo that jobs are more likely IO rather than CPU bounded. So I agree that we should take that into account, change the current algorithm and remove this somehow misleading comment. Scott's suggestion in his 2nd email sounds good to me. So If I understood well we could use an unbounded but finally limited queue, like it was before. Although with all of that said, after a quick second look it appears that the current implementation doesn't try poll for more jobs than the configured limit (minus already queued jobs) so we might be fine with an unbounded queue implementation. We'd just need to alter the call to JobManager.poll(int limit) to not pass in executor.getQueue().remainingCapacity() and instead pass in something like (threadPool.getJobs() - executor.getQueue().size()) I'm fine with that as it would continue to prevent hitting physical limitations and can be tweaked by users as it's now. Note that it seems though uneasy to tweak as we received already several "complaints" about it. Now one of the advantage of a PriorityBlockingQueue is priority. And to take advantage of that we can't rely on "/natural ordering"/ and need to implement Comparable (which does no seem easy). Nicolas provided some leads below and this should be discussed. The must would be to have that parametrised, of course. My 2 cts // [1] https://markmail.org/message/ixzluzd44rgloa2j Jacques Le 06/02/2019 à 14:24, Nicolas Malin a écrit : Hello Scott, On a customer project we use massively the job manager with an average of one hundred thousand job per days. We have different cases like, huge long jobs, async persistent job, fast regular job. The mainly problem that we detect has been (as you notified) the long jobs that stuck poller's thread and when we restart OFBiz (we are on continuous delivery) we hadn't windows this without crash some jobs. To solve try with Gil to analyze if we can load some weighting on job definition to help the job manager on what jobs on the pending queue it can push on queued queue. We changed own vision to create two pools, one for system maintenance and huge long jobs managed by two ofbiz instances and an other to manage user activity jobs also managed by two instances. We also added on service definition an information to indicate the predilection pool This isn't a big deal and not resolve the stuck pool but all blocked jobs aren't vital for daily activity. For crashed job, we introduced in trunk service lock that we set before an update and wait a windows for the restart. At this time for all OOM detected we reanalyse the origin job and tried to decompose it by persistent async service to help loading repartition. If I had more time, I would be oriented job improvement to : * Define an execution plan rule to link services and poller without touch any service definition * Define configuration by instance for the job vacuum to refine by service volumetric This feedback is a little confused Scott, maybe you found interesting things Nicolas On 30/01/2019 20:47, Scott Gray wrote: Hi folks, Just jotting down some issues with the JobManager over noticed over the last few days: 1. min-threads in serviceengine.xml is never exceeded unless the job count in the queue exceeds 5000 (or whatever is configured). Is this not obvious to anyone else? I don't think this was the behavior prior to a refactoring a few years ago. 2. The advice on the number of threads to use doesn't seem good to me, it assumes your jobs are CPU bound when in my experience they are more likely to be I/O bound while making db or external API calls, sending emails etc. With the default setup, it only takes two long running jobs to effectively block the processing of any others until the queue hits 5000 and the other threads are finally opened up. If you're not quickly maxing out the queue then any other jobs are stuck until the slow jobs finally complete. 3. Purging old jobs doesn't seem to be well implemented to me, from what I've seen the system is only capable of clearing a few hundred per minute and if you've filled the queue with them then regular jobs have to queue behind them and can take many minutes to finally be executed. I'm wondering if anyone has experimented with reducing the queue the size? I'm considering reducing it to say 100 jobs per thread (along with increasing the thread count). In
Re: Oracle Java release model changes and consequences for the project
Hi All, With https://issues.apache.org/jira/browse/OFBIZ-10757 we have getting closer to use https://adoptopenjdk.net/releases.html which seems stable so far. At OFBIZ-10757 we discussed about moving not only trunk but also not yet released R17 and R18 to Java 11. Note that current changes don't need to switch yet to Java 11 (it works also with Java 8), but we think that it's time to move on not only to Java 11 but also to Adoptopenjdk What do you think? Thanks Jacques Le 17/02/2019 à 13:28, Jacques Le Roux a écrit : +1 Jacques Le 13/02/2019 à 12:51, Jacopo Cappellato a écrit : Hi Michael, yes AdoptOpenJDK is definitely a good fit. Jacopo On Wed, Feb 13, 2019 at 12:39 PM Michael Brohl wrote: Hi Jacopo, an alternative would be https://adoptopenjdk.net/ which provides prebuild packages. The scripts for package building are Apache 2.0 licensed and they are providing Java 8 and 11 LTS versions. Seems a good fit to me. Since Java 8 is LTS there, we do not necessarily have to upgrade OFBiz for the use of Java 11. Best regards, Michael Am 13.02.19 um 11:06 schrieb Jacopo Cappellato: Considering that now Oracle JDKs are no more free for commercial use, I think that as a community we should make it a priority to suggest a different Java build in the README and other public documents. The simplest alternative (because it is the closest to Oracle JDK) is the Open JDK 11 maintained by Oracle and distributed from: https://jdk.java.net/11/ In my opinion our README should point to it rather than: http://www.oracle.com/technetwork/java/javase/downloads/index.html as it is now. However, before we can do it, we have to resolve: https://issues.apache.org/jira/browse/OFBIZ-10757 which should not be too difficult to achieve. Just my two cents, Jacopo On Wed, Oct 24, 2018 at 2:21 PM James Yong wrote: Answering my last question. From the download page for Oracle JDK 11, demo purpose is allowed. On 2018/10/24 07:38:19, James Yong wrote: Hi all, Will the release model and licensing changes impact our demos hosted with Apache Software Foundation? Regards, James On 2018/10/24 06:54:05, James Yong wrote: Hi all, OFBiz can be used as an application framework and not all business use-case justify the yearly price-tag of Oracle JDK. Given that more products(1) are moving to support OpenJDK, should OFBiz follow? Regards, James (1) See plan of Atlasians product to support OpenJDK https://community.atlassian.com/t5/Jira-discussions/Java-11-and-OpenJDK-support-for-Atlassian-Server-amp-Data-Center/m-p/872998#M4575 On 2018/07/31 06:35:46, Jacques Le Roux wrote: Hi Michael, How (by which mean) do you envision to "actively inform users about our roadmap", blog, wiki or embedded documentation? It seems the blog is not reaching all our users (needs attention). Maybe an initial statement could be used there though. The wiki is slowly deprecating in favour of the embedded documentation. So I guess we will use the embedded documentation for lasting information, right? BTW All, I want to close OFBIZ-9226 "Check that OFBiz runs and compile with Oracle JDK 9 (Java 9)" as unresolved and create a new similar issue for Java 11, what do you think? Jacques Le 28/07/2018 à 13:29, Michael Brohl a écrit : Hi Mathieu, my goal is to actively inform users about our roadmap and provide information on how the project will deal with the new Java release model. Users testing OFBiz for their needs in a professional environment also check if a project has answers to these questions so I am wrapping my mind around it. This is just to make clear that I am not eager to switch to newer Java versions just for the sake of it. Am 28.07.18 um 12:54 schrieb Mathieu Lirzin: I wonder if we should base the OFBiz 17.12 release on Java 8 or Java 11. We have no fixed release date yet so we might have time to do it. Another way would be to make a new branch which will support Java 11. What do people think? I think OFBiz should be conservative in its choices. I agree! Given the fact Java 11 is not release yet or is about to be released, Java 11 will be released as GA in Sept 18. At the same time, non-subscribed users will get no updates for Java 8 any more. OFBiz should keep compatibity with the previous LTS release meaning java 8. Of course Yes, you are right. If you focus on subscribed users, they will get Java 8 support until September 2023 (2026 for extended subscription). So following my thoughts to assume that users will subscribe, we can stay with Java 8 for a while. On the other hand, if we test Java 11 and find that we will have few issues we can easily handle, it could be a good idea to make the switch with release 17.12. I am open to both (or other) models and would like to hear more opinions about that. This does not mean that OFBiz should not be tested with more recent Java releases too. Having an extra branch has a maintenance burden that should be
Re: Enabling HTTP/2 in the embedded Tomcat connectors
I had a look, I don't think we need to worry about those vars.
If/When needed we can change defaults.
Jacques
Le 19/02/2019 à 19:05, Jacques Le Roux a écrit :
Good question, I guess most are OK, but what are Trailer Headers for instance?
BTW https://tomcat.apache.org/tomcat-9.0-doc/config/http2.html is our reference
for the trunk
Jacques
Le 19/02/2019 à 10:24, Taher Alkhateeb a écrit :
clean and simple implementation, +1
on a side note, I wonder if we need to set any of the http2 attributes
listed in [1] or whether the defaults are okay.
[1] https://tomcat.apache.org/tomcat-8.5-doc/config/http2.html
On Mon, Feb 18, 2019 at 5:58 PM Jacques Le Roux
wrote:
Hi Jacopo,
Sounds good to me, we can always easily revert in case of unexpected issue
anyway
Thanks
Jacques
Le 18/02/2019 à 11:43, Jacopo Cappellato a écrit :
Hi all,
I think it is time to enable the instance of Tomcat that is embedded in
OFBiz to communicate using the HTTP/2 protocol, when the client supports it.
For your review, before I commit, I am pasting here the patch that will
enable it (it is quite simple) .
In it I have enabled HTTP/2 by default, by setting upgradeProtocol=true, in
the http and https connectors (but they will continue to support also
HTTP/1.1); if the new property "upgradeProtocol", that I have introduced
for this specific purpose, is not set (as it would be the case in custom
configuration files) then the new protocol will not be enabled. Does the
approach look good to you?
Thanks,
Jacopo
PS: you can test it, for example, using curl:
curl -vso /dev/null --http2 http://localhost:8080
Index: framework/catalina/ofbiz-component.xml
===
--- framework/catalina/ofbiz-component.xml (revision 1853787)
+++ framework/catalina/ofbiz-component.xml (working copy)
@@ -99,6 +99,7 @@
+
@@ -128,6 +129,7 @@
+
@@ -183,6 +185,7 @@
+
@@ -194,6 +197,7 @@
+
Index:
framework/catalina/src/main/java/org/apache/ofbiz/catalina/container/CatalinaContainer.java
===
---
framework/catalina/src/main/java/org/apache/ofbiz/catalina/container/CatalinaContainer.java
(revision
1853787)
+++
framework/catalina/src/main/java/org/apache/ofbiz/catalina/container/CatalinaContainer.java
(working
copy)
@@ -63,6 +63,7 @@
import org.apache.catalina.util.ServerInfo;
import org.apache.catalina.valves.AccessLogValve;
import org.apache.catalina.webresources.StandardRoot;
+import org.apache.coyote.http2.Http2Protocol;
import org.apache.ofbiz.base.component.ComponentConfig;
import org.apache.ofbiz.base.concurrent.ExecutionPool;
import org.apache.ofbiz.base.container.Container;
@@ -417,9 +418,12 @@
private Connector prepareConnector(Property connectorProp) {
Connector connector = new
Connector(ContainerConfig.getPropertyValue(connectorProp, "protocol",
"HTTP/1.1"));
connector.setPort(ContainerConfig.getPropertyValue(connectorProp,
"port", 0) + Start.getInstance().getConfig().portOffset);
-
+ if ("true".equals(ContainerConfig.getPropertyValue(connectorProp,
"upgradeProtocol", "false"))) {
+ connector.addUpgradeProtocol(new Http2Protocol());
+ Debug.logInfo("Tomcat " + connector + ": enabled HTTP/2",
module);
+ }
connectorProp.properties.values().stream()
- .filter(prop -> !"protocol".equals(prop.name) &&
!"port".equals(prop.name))
+ .filter(prop -> !"protocol".equals(prop.name) &&
!"upgradeProtocol".equals(prop.name) && !"port".equals(prop.name))
.forEach(prop -> {
if (IntrospectionUtils.setProperty(connector, prop.name,
prop.value)) {
if (prop.name.indexOf("Pass") != -1) {
