Hi again,
In a more detailed investigation, I attached jdb to servicemix and see the
path executed line by line. i put breakpoints to each line in boolean
doLoadImmediate() block. i saw that code works without exception till
the debug("LOAD IMMEDIATE complete"); line:733 in finally block. Strange it
is that everything works great but breakpoint at
_processedSinceLastLoadTask.clear(); line 727 (
http://svn.apache.org/viewvc/ode/trunk/scheduler-simple/src/main/java/org/apache/ode/scheduler/simple/SimpleScheduler.java?revision=947456&view=markup)
is not reached although last line hit :707 which is AbsoluteTimeDateFormat
f = new AbsoluteTimeDateFormat();
I`ve experimented with single requests and also with JMeter (~300tps). The
code does not enter the the catch block. And the size of
concurrentHashMap _processedSinceLastLoadTask grows higher and higher since
clear method is never executed.
I do appreciate any help.
Hi,
I deployed ode 1.3.5 in servicemix 4.3.0 on Ubuntu 11.04 64bit with maxHeap
size = 1024m . I have set necessary coreThreadPool sizes to 16,
maxThreadPool sizes to unlimited. I generated a 300-400 tps load on a flow
which is in-memory enabled. After a while, short gc cycles could not achieve
to collect necessary objects and fullgc cycles try to collect with pausing
system 2,5 - 4.0 seconds which causes system to freeze and not handle
further requests. At the same time i took a heap snapshot and tried to
analyse it. As a result, i saw the biggest object was SimpleScheduler
instance containing ConcurrentHashMap<String,Long>
_processedSinceLastLoadTask instance (almost 1gb) which contains
jobId(String) type as keys and scheduling dates(Long) as values.
I have researched a bit on SimpleScheduler trunk version but seen that 1.3.5
version installed in Servicemix 4.3.0 contains an older version compared to
trunk. I decompiled the class in servicemix version and investigated that
either _processedSinceLastLoadTask.clear() does not work or execution does
not endure till the line _processedSinceLastLoadTask.clear() inside
boolean doLoadImmediate() method block.
I have tried several configuration parameters such as `process-events
generate="none"` `cleanup on="always"` in deploy.xml
or ode-jbi.threads.pool.size=100 in ode config file. None of them have
worked out.
Any idea about what may cause _processedSinceLastLoadTask map to grow?
Thanks.
