Thanks Sébastien. I will try with the new file that you sent. In the mean time
can you let me know the following?
1) I had modified your code to restrict the number of threads to be 20 and then
40 and then 60. All resulted in the partial sync and did not synced all
records. How do we arrive at the correct number of the threads? How many
minimum threads does LSC need to do the full sync? Interestingly when I
modified LSC code to use no thread at all ( ie, by calling the run method
SynchronizeTask directly. See modified code snippet below from
synchronize2Ldap() method in AbstractSynchronize. I commented your code and
added mine) syncs all records.
SynchronizeTask syncTask;
for (Entry<String, LscDatasets> id : ids) {
syncTask = new
SynchronizeTask(task, counter, this, id, true);
syncTask.run();
// threadPool.runTask(new
SynchronizeTask(task, counter, this, id, true));
}
3) Also in one of your responses you mentioned the following. Can you let me
know what is the configuration change that I need to do? If this fixes my
issue then we don't need to limit the threadpool size and change to API ( which
you just made to SyncReplSourceService) . Am I wrong?
" LSC is successfully syncing directories with millions of entries but probably
not with the same configuration. Can you generate a threaddump (jstack -l) to
help US to identify where the threads are leaking "
Regards,
Dinesh Babu.
Pitney Bowes Software
6 Hercules Way, Leavesden Park, Watford, Herts WD25 7GS
Ph: +441923 279123 www.pb.com/software<http://www.pb.com/software>
[email protected]
Every connection is a new opportunity™
[cid:[email protected]]
Please consider the environment before printing or forwarding this email. If
you do print this email, please recycle the paper.
This email message may contain confidential, proprietary and/or privileged
information. It is intended only for the use of the intended recipient(s). If
you have received it in error, please immediately advise the sender by reply
email and then delete this email message. Any disclosure, copying, distribution
or use of the information contained in this email message to or by anyone other
than the intended recipient is strictly prohibited.
From: Sébastien Bahloul [mailto:[email protected]]
Sent: 27 November 2014 13:41
To: Clément OUDOT
Cc: Dinesh Babu; [email protected]; [email protected]
Subject: Re: [lsc-users] Caution.. LSC does not seem work with large data sync
from LDAP
Hi Dinesh,
Sorry but is seems that your previous messages have been bounced by our lists
engine (messages above 200kB). So I haven't seen your thread dump until this
mail.
If you are using LSC in an embedded war, you are using the SimpleSynchronize
class. The method you should used to limit the amount of threads used is
setThreads(int threads). The sample piece of code you should look at is in the
embeddable sample :
https://lsc-project.org/svn/lsc-samples/lsc-embeddable-sample/trunk/src/main/java/org/lsc/utils/LscServlet.java
Look at line with sync.setThreads(Integer.parseInt(maxThreads);
The OutOfMemory error is due to a thread issue in MINA (which is the underlying
component of Apache LDAP API). This is under analysis and I'm trying to
reproduce it. I think that the attached SyncReplSourceService.java updated code
should fix it (because it uses a pool and handle connection releases in a
smarter way) but i'm unable to be sure that it will fix it because I don't
reproduce your issue.
Regards,
Sebastien BAHLOUL
IAM / Security specialist
Ldap Synchronization Connector : http://lsc-project.org
Blog : http://sbahloul.wordpress.com/
2014-11-27 10:32 GMT+01:00 Clément OUDOT
<[email protected]<mailto:[email protected]>>:
2014-11-27 10:23 GMT+01:00 Dinesh Babu
<[email protected]<mailto:[email protected]>>:
Just a caution/warning to LSC users..
LSC fails to sync anything above 80,000 to 100,000 records from LDAP. If you
want to sync large data, then use at your own risk.
LSC sync causes socket error consistently. This is because there are too many
threads in the thread pool and I can't figure out how to limit and LSC team has
not responded/explained how to do it.
Hi Dinesh,
we already answer you on the threads: there is an option -t for lsc launcher
that allows to configure how many threads are used by LSC.
If you encounter problems, we would be happy to help if you can bring some
steps to reproduce, logs, etc.
LSC is a free software, and you can read in the BSD licence:
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
* IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
* TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
* PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER
* OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
* EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
* PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
* LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
* NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
* SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
So use at your own risk.
Clément.
________________________________
_______________________________________________________________
Ldap Synchronization Connector (LSC) - http://lsc-project.org
lsc-users mailing list
[email protected]
http://lists.lsc-project.org/listinfo/lsc-users
