Hi Sébastien,
Thanks for your reply, it gave me an idea to do some more investigation.
What I noticed by looking at the code in AbstractSynchronize.java
was that SynchronizeTask objects were being added to the threadPool
very quickly in the case of a large number of entries and the threadPool
would keep growing exponentially, consuming all available memory on the
system.
I think that this occurs because LDAP ADD is a slow operation, so the pool
keeps on growing since it does not empty quickly enough by adding entries
to LDAP.
So what I did to solve the problem was to process 10,000 entries at a time,
wait for the threadPool to finish all thread executions and then continue
with the next 10,000 and so on. Here are the changes I made
to AbstractSynchronize.java:
------------------------------------------------------
/*
* Loop on all entries in the source and add or update them in
the
* destination
*/
int threadCount = 0;
for (Entry<String, LscDatasets> id : ids) {
threadPool.runTask(new SynchronizeTask(task, counter,
this, id));
threadCount++;
if (threadCount == 10000) {
try {
threadPool.shutdown();
threadPool.awaitTermination(900,
TimeUnit.SECONDS);
threadCount = 0;
threadPool = null;
threadPool = new
SynchronizeThreadPoolExecutor(getThreads());
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
try {
threadPool.shutdown();
threadPool.awaitTermination(timeLimit,
TimeUnit.SECONDS);
} catch (InterruptedException e) {
LOGGER.error("Tasks terminated according to time limit:
" + e.toString(), e);
LOGGER.info("If you want to avoid this message, " +
"increase the time limit by using dedicated parameter.");
}
----------------------------------------------------------
When I did this, the whole lsc process did not consume all memory on the system
(it never went above
10% on a machine with 8Gb of RAM) and the thread/process count did not keep
growing under Red Hat Linux.
I managed to sync. 680,000 entries in about 1 hour and 50 minutes and lsc
completed successfully.
Does this fix seem okay to you? This line: threadPool.runTask(new
SynchronizeTask(task, counter, this, id));
seems to be the problem when there are a large number of entries to synchronize
-- too many
SynchronizeTask objects are generated and the queue does not empty quickly
enough while more objects
keep getting added.
cheers,
Ven
________________________________________
From: Sébastien Bahloul [sebastien.bahl...@gmail.com]
Sent: Saturday, August 17, 2013 1:50 AM
To: Mahadevan, Venkat
Cc: lsc-users
Subject: RE: [lsc-users] Fatal LSC error when syncing large database to LDAP
Ven,
Your configuration seems quite standard so sorry because this is probably a
bug. Can you generate a thread dump to see where they are stuck ?
Regards
Le 17 août 2013 10:46, "Mahadevan, Venkat"
<venkm...@mail.ubc.ca<mailto:venkm...@mail.ubc.ca>> a écrit :
Hi Sébastien,
Here's the command line:
lsc -f /local/adm/ssd/cwlssd/lsc-2.0.2/etc -s all
where /etc contains lsc.xml, logback.xml, sql-map-config.xml, sql-map-config.d,
etc...
I'm attaching the lsc.xml file as well. Thank you for your help.
cheers,
Ven
________________________________________
From: Sébastien Bahloul
[sebastien.bahl...@gmail.com<mailto:sebastien.bahl...@gmail.com>]
Sent: Saturday, August 17, 2013 1:25 AM
To: Mahadevan, Venkat
Cc: lsc-users
Subject: Re: [lsc-users] Fatal LSC error when syncing large database to LDAP
Hi Venkat,
Can you provide the complete option and command line and also your
configuration file (lsc.xml) ? You shouldn't reach such limit.
Regards,
Le 17 août 2013 09:49, "Mahadevan, Venkat"
<venkm...@mail.ubc.ca<mailto:venkm...@mail.ubc.ca><mailto:venkm...@mail.ubc.ca<mailto:venkm...@mail.ubc.ca>>>
a écrit :
Hello everyone,
I am testing LSC 2.0.2 under Redhat Enterprise Linux 5.7 and JDK 1.7 and also
JDK 1.6 (both from Oracle, 64-bit).
I am syncing an Oracle database with about 600,000 entries to LDAP.
What happens is that the job always terminates with an out of memory exception
as follows:
Exception in thread "main" java.lang.OutOfMemoryError: unable to create new
native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:691)
at
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1336)
at
org.lsc.SynchronizeThreadPoolExecutor.runTask(SynchronizeThreadPoolExecutor.java:44)
at
org.lsc.AbstractSynchronize.synchronize2Ldap(AbstractSynchronize.java:333)
at org.lsc.SimpleSynchronize.launchTask(SimpleSynchronize.java:220)
at org.lsc.SimpleSynchronize.launch(SimpleSynchronize.java:157)
at org.lsc.Launcher.run(Launcher.java:224)
at org.lsc.Launcher.launch(Launcher.java:159)
at org.lsc.Launcher.main(Launcher.java:142)
I attempted to monitor the lsc process while it was running and noticed the
following:
watch -n 5 -d "ps -eL <pid> | wc -l"
The number of threads being spawned increased all the way from 108 to around
30,000 before the
lsc process crashed and threw the exception above. Increasing the memory to the
JVM via -Xmx had
no helpful effect.
Is there some way to tune the maximum number of threads that LSC uses or is
this a known issue
with large synchronizations? Thanks in advance for any advice.
Kind regards,
Ven
_______________________________________________________________
Ldap Synchronization Connector (LSC) - http://lsc-project.org
lsc-users mailing list
lsc-users@lists.lsc-project.org<mailto:lsc-users@lists.lsc-project.org><mailto:lsc-users@lists.lsc-project.org<mailto:lsc-users@lists.lsc-project.org>>
http://lists.lsc-project.org/listinfo/lsc-users
_______________________________________________________________
Ldap Synchronization Connector (LSC) - http://lsc-project.org
lsc-users mailing list
lsc-users@lists.lsc-project.org
http://lists.lsc-project.org/listinfo/lsc-users
