I wanted to discuss this again.
Some time ago I modified my local copy of OFBiz to execute the the ant
run-install task faster by using multi-threaded entity creation and data
loading. I don't remember what i was working on at the time, but I
needed the process to run much faster.
Adam and I discussed my design and he said it sounded like it was
similar to SEDA (http://www.eecs.harvard.edu/~mdw/proj/seda/). Actually,
my approach was not that sophisticated, I just used a consumer-provider
design pattern based on a FIFO queue.
Anyway, based on that conversation, Adam committed the multi-threaded
entity creation/data loading code
(http://mail-archives.apache.org/mod_mbox/ofbiz-commits/201106.mbox/%3c20110626025049.f302e2388...@eris.apache.org%3E).
The work Adam committed greatly improved the database creation/data
loading time.
In this Jira issue, Adam mentions how transactions are tied to a single
thread. This is due to a fundamental weakness in OFBiz - the use of
ThreadLocal variables. In order to truly remove the bottlenecks in
OFBiz, we need to avoid the use of ThreadLocal variables - because they
prohibit handing off tasks to other threads. The Execution Context
proposed by David Jones some time ago is a step in the right direction,
but it uses ThreadLocal variables too.
So, we really need an object that is passed around the framework that
represents a TASK state, not a THREAD state - so that tasks can be
handed off to multiple threads. Transactions are tasks, so transaction
state would be contained in the object, not in ThreadLocal variables. I
believe that approach would solve the issue here.
-Adrian
On 5/10/2012 11:48 PM, Adam Heath (JIRA) wrote:
[
https://issues.apache.org/jira/browse/OFBIZ-4870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272862#comment-13272862
]
Adam Heath commented on OFBIZ-4870:
-----------------------------------
There is multi-threading per-datasource during startup. Multiple threads
creating the tables/keys/etc. Seed data(from xml) is still single-threaded.
I have a series of changes to fix this locally, but it's *really* complex.
Since an xml import may already be part of an *existing* transaction, I
absolutely *must* manipulate the database in the current thread. The parsing,
however, can be done in a background thread, in parallel. Unfortunately, the
xml parser in use is event based, in the wrong direction(push instead of pull),
so the bulk of the change is flipping that around. Tbh, the complexity wasn't
really worth the speedup(just a few percentage points, afaicr).
What threaded stuff are you still seeking?
Multithreading in GenericDAO / Delegator
----------------------------------------
Key: OFBIZ-4870
URL: https://issues.apache.org/jira/browse/OFBIZ-4870
Project: OFBiz
Issue Type: Improvement
Reporter: Mirko Vogelsmeier
Hey there,
some time ago there were some commits of Adam that brought in first ideas of
multi threaded delegator useage (r1139700).
Depending on how intense the data useage or data size is, there are performance
issues we cannot scale by pure hardware and/or configuration as there is just
one Delegator object per datasource.
I wanted to check on the progress of this very helpfull feature. Are there any
further thoughts to work on this?
Greetings,
Mirko
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
