By the way compilation is failing the new file that you sent Sebastien. You are using LdapConnectionPool(getConnConfig(ldapConn), apiService, Integer.MAX_VALUE); in your SyncReplSourceService.java but only constructor available is LdapConnectionPool( PoolableLdapConnectionFactory factory ) in api-all-1.0.0-M22.jar as part of LSC2.1
Regards, Dinesh Babu. From: Dinesh Babu Sent: 27 November 2014 14:22 To: 'Sébastien Bahloul'; Clément OUDOT Cc: [email protected]; [email protected] Subject: RE: [lsc-users] Caution.. LSC does not seem work with large data sync from LDAP Thanks Sébastien. I will try with the new file that you sent. In the mean time can you let me know the following? 1) I had modified your code to restrict the number of threads to be 20 and then 40 and then 60. All resulted in the partial sync and did not synced all records. How do we arrive at the correct number of the threads? How many minimum threads does LSC need to do the full sync? Interestingly when I modified LSC code to use no thread at all ( ie, by calling the run method SynchronizeTask directly. See modified code snippet below from synchronize2Ldap() method in AbstractSynchronize. I commented your code and added mine) syncs all records. SynchronizeTask syncTask; for (Entry<String, LscDatasets> id : ids) { syncTask = new SynchronizeTask(task, counter, this, id, true); syncTask.run(); // threadPool.runTask(new SynchronizeTask(task, counter, this, id, true)); } 3) Also in one of your responses you mentioned the following. Can you let me know what is the configuration change that I need to do? If this fixes my issue then we don't need to limit the threadpool size and change to API ( which you just made to SyncReplSourceService) . Am I wrong? " LSC is successfully syncing directories with millions of entries but probably not with the same configuration. Can you generate a threaddump (jstack -l) to help US to identify where the threads are leaking " Regards, Dinesh Babu. Pitney Bowes Software 6 Hercules Way, Leavesden Park, Watford, Herts WD25 7GS Ph: +441923 279123 www.pb.com/software<http://www.pb.com/software> [email protected] Every connection is a new opportunity™ [cid:[email protected]] Please consider the environment before printing or forwarding this email. If you do print this email, please recycle the paper. This email message may contain confidential, proprietary and/or privileged information. It is intended only for the use of the intended recipient(s). If you have received it in error, please immediately advise the sender by reply email and then delete this email message. Any disclosure, copying, distribution or use of the information contained in this email message to or by anyone other than the intended recipient is strictly prohibited. From: Sébastien Bahloul [mailto:[email protected]] Sent: 27 November 2014 13:41 To: Clément OUDOT Cc: Dinesh Babu; [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]> Subject: Re: [lsc-users] Caution.. LSC does not seem work with large data sync from LDAP Hi Dinesh, Sorry but is seems that your previous messages have been bounced by our lists engine (messages above 200kB). So I haven't seen your thread dump until this mail. If you are using LSC in an embedded war, you are using the SimpleSynchronize class. The method you should used to limit the amount of threads used is setThreads(int threads). The sample piece of code you should look at is in the embeddable sample : https://lsc-project.org/svn/lsc-samples/lsc-embeddable-sample/trunk/src/main/java/org/lsc/utils/LscServlet.java Look at line with sync.setThreads(Integer.parseInt(maxThreads); The OutOfMemory error is due to a thread issue in MINA (which is the underlying component of Apache LDAP API). This is under analysis and I'm trying to reproduce it. I think that the attached SyncReplSourceService.java updated code should fix it (because it uses a pool and handle connection releases in a smarter way) but i'm unable to be sure that it will fix it because I don't reproduce your issue. Regards, Sebastien BAHLOUL IAM / Security specialist Ldap Synchronization Connector : http://lsc-project.org Blog : http://sbahloul.wordpress.com/ 2014-11-27 10:32 GMT+01:00 Clément OUDOT <[email protected]<mailto:[email protected]>>: 2014-11-27 10:23 GMT+01:00 Dinesh Babu <[email protected]<mailto:[email protected]>>: Just a caution/warning to LSC users.. LSC fails to sync anything above 80,000 to 100,000 records from LDAP. If you want to sync large data, then use at your own risk. LSC sync causes socket error consistently. This is because there are too many threads in the thread pool and I can't figure out how to limit and LSC team has not responded/explained how to do it. Hi Dinesh, we already answer you on the threads: there is an option -t for lsc launcher that allows to configure how many threads are used by LSC. If you encounter problems, we would be happy to help if you can bring some steps to reproduce, logs, etc. LSC is a free software, and you can read in the BSD licence: * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS * IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER * OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING * NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. So use at your own risk. Clément. ________________________________
_______________________________________________________________ Ldap Synchronization Connector (LSC) - http://lsc-project.org lsc-users mailing list [email protected] http://lists.lsc-project.org/listinfo/lsc-users
