Just a caution/warning to LSC users.. LSC fails to sync anything above 80,000 to 100,000 records from LDAP. If you want to sync large data, then use at your own risk.
LSC sync causes socket error consistently. This is because there are too many threads in the thread pool and I can't figure out how to limit and LSC team has not responded/explained how to do it. Regards, Dinesh Babu. Pitney Bowes Software 6 Hercules Way, Leavesden Park, Watford, Herts WD25 7GS Ph: +441923 279123 www.pb.com/software<http://www.pb.com/software> [email protected] Every connection is a new opportunity™ [cid:[email protected]] Please consider the environment before printing or forwarding this email. If you do print this email, please recycle the paper. This email message may contain confidential, proprietary and/or privileged information. It is intended only for the use of the intended recipient(s). If you have received it in error, please immediately advise the sender by reply email and then delete this email message. Any disclosure, copying, distribution or use of the information contained in this email message to or by anyone other than the intended recipient is strictly prohibited. From: Dinesh Babu Sent: 25 November 2014 16:39 To: '[email protected]'; [email protected]; [email protected] Subject: FW: [lsc-users] Socket error while syncing from LDAP Hi Sébastien, I messed up the serial numbers and their references. Here they are in the right order =========================================== Did you get a chance to go through the attached thread dump? 1) Can you let me know the configuration that we need to use to sync large LDAP data as you mentioned LSC has synced millions of records not with the same configuration. 2) If you are unable to give the info in (1) ; if I have to limit the thread pool size to one, how do I use the -t option as we are not running LSC through command line. We running it inside a war file. 3) If I limit the tead pool size usinf the -t option you mentioned , will it limit the thread pool size of my application server as well(I don't want that to happen. I just want to limit the threadpool size for LSC) ============================================= Regards, Dinesh Babu. Pitney Bowes Software 6 Hercules Way, Leavesden Park, Watford, Herts WD25 7GS Ph: +441923 279123 www.pb.com/software<http://www.pb.com/software> [email protected] Every connection is a new opportunity™ [cid:[email protected]] Please consider the environment before printing or forwarding this email. If you do print this email, please recycle the paper. This email message may contain confidential, proprietary and/or privileged information. It is intended only for the use of the intended recipient(s). If you have received it in error, please immediately advise the sender by reply email and then delete this email message. Any disclosure, copying, distribution or use of the information contained in this email message to or by anyone other than the intended recipient is strictly prohibited. From: Dinesh Babu Sent: 25 November 2014 07:49 To: '[email protected]'; [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]> Subject: FW: [lsc-users] Socket error while syncing from LDAP Hi Sébastien, Did you get a chance to go through the attached thread dump? 2) Can you let me know the configuration that we need to use to sync large LDAP data as you mentioned LSC has synced millions of records not with the same configuration. 1) If you are unable to give the info in (1) ; if I have to limit the thread pool size to one, how do I use the -t option as we are not running LSC through command line. We running it inside a war file. 2) If I limit the tead pool size usinf the -t option you mentioned , will it limit the thread pool size of my application server as well(I don't want that to happen. I just want to limit the threadpool size for LSC) Regards, Dinesh Babu. Pitney Bowes Software 6 Hercules Way, Leavesden Park, Watford, Herts WD25 7GS Ph: +441923 279123 www.pb.com/software<http://www.pb.com/software> [email protected] Every connection is a new opportunity™ [cid:[email protected]] Please consider the environment before printing or forwarding this email. If you do print this email, please recycle the paper. This email message may contain confidential, proprietary and/or privileged information. It is intended only for the use of the intended recipient(s). If you have received it in error, please immediately advise the sender by reply email and then delete this email message. Any disclosure, copying, distribution or use of the information contained in this email message to or by anyone other than the intended recipient is strictly prohibited. From: Dinesh Babu Sent: 21 November 2014 17:45 To: 'Sébastien Bahloul'; [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]> Subject: RE: [lsc-users] Socket error while syncing from LDAP Hi Sébastien, Please find the attached zip file which has two thread dumps and one server log. Regards, Dinesh Babu. Pitney Bowes Software 6 Hercules Way, Leavesden Park, Watford, Herts WD25 7GS Ph: +441923 279123 www.pb.com/software<http://www.pb.com/software> [email protected] Every connection is a new opportunity™ [cid:[email protected]] Please consider the environment before printing or forwarding this email. If you do print this email, please recycle the paper. This email message may contain confidential, proprietary and/or privileged information. It is intended only for the use of the intended recipient(s). If you have received it in error, please immediately advise the sender by reply email and then delete this email message. Any disclosure, copying, distribution or use of the information contained in this email message to or by anyone other than the intended recipient is strictly prohibited. From: Sébastien Bahloul [mailto:[email protected]] Sent: 20 November 2014 08:09 To: Dinesh Babu Cc: LSC developers list; lsc-users Subject: RE: [lsc-users] Socket error while syncing from LDAP Hi Dinesh, Look at the LSC command line options. LSC is successfully syncing directories with millions of entries but probably notre with the same configuration. Can you generate a threaddump (jstack -l) to help US to identify where the threads are leaking ? Thanks, Le 19 nov. 2014 17:02, "Dinesh Babu" <[email protected]<mailto:[email protected]>> a écrit : Thanks Sebastien, Quartz kick starts our LSC sync process. Did not understand where I can set this command line argument "-t"? Also has any of LSC users ever brought to your attention about socket connection error while syncing from an LDAP with more than 80,000 users? We are consistently getting this issue when syncing 80,000+ users. Regards, Dinesh Babu. Pitney Bowes Software 6 Hercules Way, Leavesden Park, Watford, Herts WD25 7GS Ph: +441923 279123<tel:%2B441923%20279123> www.pb.com/software<http://www.pb.com/software> [email protected]<http://[email protected]> Every connection is a new opportunity™ [cid:[email protected]] Please consider the environment before printing or forwarding this email. If you do print this email, please recycle the paper. This email message may contain confidential, proprietary and/or privileged information. It is intended only for the use of the intended recipient(s). If you have received it in error, please immediately advise the sender by reply email and then delete this email message. Any disclosure, copying, distribution or use of the information contained in this email message to or by anyone other than the intended recipient is strictly prohibited. From: Sébastien Bahloul [mailto:[email protected]<mailto:[email protected]>] Sent: 19 November 2014 15:48 To: Dinesh Babu Cc: [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]> Subject: Re: [lsc-users] Socket error while syncing from LDAP Hi Dinesh, The Thread pool size is an internal maximum size. The effective maximum size is provided by your -t command line parameter value I kindly suggest to read the following thread to understand the current implementation : http://stackoverflow.com/questions/1800317/impossible-to-make-a-cached-thread-pool-with-a-size-limit/1800583#1800583 Regards, Sebastien BAHLOUL IAM / Security specialist Ldap Synchronization Connector : http://lsc-project.org Blog : http://sbahloul.wordpress.com/ 2014-11-19 16:21 GMT+01:00 Dinesh Babu <[email protected]<mailto:[email protected]>>: In LSC code, see below, the thread pool size is set integer.MAX_VALUE which actually is billions. When we have so much threads while syncing large amount of records (we are trying to sync with 120,000+ records), that will cause socket error, won't it? Is there any particular reason you did not want to limit the tread pool size? protected SynchronizeThreadPoolExecutor(int threads) { super(threads, Integer.MAX_VALUE, keepAliveTime, TimeUnit.SECONDS, new ArrayBlockingQueue<Runnable>(Configuration.MAX_CONCURRENT_SYNCHRONIZED)); queue = getQueue(); } Regards, Dinesh Babu. Pitney Bowes Software 6 Hercules Way, Leavesden Park, Watford, Herts WD25 7GS Ph: +441923 279123<tel:%2B441923%20279123> www.pb.com/software<http://www.pb.com/software> [email protected]<http://[email protected]> Every connection is a new opportunity™ [cid:[email protected]] Please consider the environment before printing or forwarding this email. If you do print this email, please recycle the paper. This email message may contain confidential, proprietary and/or privileged information. It is intended only for the use of the intended recipient(s). If you have received it in error, please immediately advise the sender by reply email and then delete this email message. Any disclosure, copying, distribution or use of the information contained in this email message to or by anyone other than the intended recipient is strictly prohibited. From: Dinesh Babu Sent: 16 November 2014 23:08 To: [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]> Subject: RE: Socket error while syncing from LDAP In LSC, is it possible to introduce a pause after fetching certain number of records from LDAP and before proceeding any further? I think I am getting socket error because the LDAP socket queue has not got enough space to accommodate new requests from LSC when LSC requests data from LDAP very frequently for prolonged period of time. Regards, Dinesh Babu. Pitney Bowes Software 6 Hercules Way, Leavesden Park, Watford, Herts WD25 7GS Ph: +441923 279123<tel:%2B441923%20279123> www.pb.com/software<http://www.pb.com/software> [email protected]<http://[email protected]> Every connection is a new opportunity™ [cid:[email protected]] Please consider the environment before printing or forwarding this email. If you do print this email, please recycle the paper. This email message may contain confidential, proprietary and/or privileged information. It is intended only for the use of the intended recipient(s). If you have received it in error, please immediately advise the sender by reply email and then delete this email message. Any disclosure, copying, distribution or use of the information contained in this email message to or by anyone other than the intended recipient is strictly prohibited. From: Dinesh Babu Sent: 12 November 2014 18:10 To: [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]> Subject: Socket error while syncing from LDAP Hi, Has anyone seen this type of error during LSC syncing from LDAP? When LSC bombards LDAP with queries does the LDAP socket gets closed or becomes unavailable? 08:23:45,525 ERROR [org.lsc.jndi.JndiServices] (all-users-380033) Error while lo oking for (sAMAccountName=$DK5K00-SEVROOJC1QTD) in OU=testOrgUnitSync,DC=eo,DC=l ocal: javax.naming.CommunicationException: Software caused connection abort: soc ket write error [Root exception is java.net.SocketException: Software caused con nection abort: socket write error]; remaining name '' 08:23:45,527 ERROR [org.lsc.AbstractSynchronize] (all-users-383873) Error while synchronizing ID {samaccountname=$VS7K00-F4ISAC5NM0KE}: org.lsc.exception.LscSer viceException: javax.naming.NamingException: JBAS011843: Failed instantiate Init ialContextFactory com.sun.jndi.ldap.LdapCtxFactory from classloader ModuleClassL oader for Module "deployment.admin-services.war:main" from Service Module Loader [Root exception is javax.naming.CommunicationException: simple bind failed: [IP address of LDAP]:389 [Root exception is java.net.SocketException: Connection reset b y peer: socket write error]] Regards, Dinesh Babu. Pitney Bowes Software 6 Hercules Way, Leavesden Park, Watford, Herts WD25 7GS Ph: +441923 279123<tel:%2B441923%20279123> www.pb.com/software<http://www.pb.com/software> [email protected]<http://[email protected]> Every connection is a new opportunity™ [cid:[email protected]] Please consider the environment before printing or forwarding this email. If you do print this email, please recycle the paper. This email message may contain confidential, proprietary and/or privileged information. It is intended only for the use of the intended recipient(s). If you have received it in error, please immediately advise the sender by reply email and then delete this email message. Any disclosure, copying, distribution or use of the information contained in this email message to or by anyone other than the intended recipient is strictly prohibited. ________________________________ _______________________________________________________________ Ldap Synchronization Connector (LSC) - http://lsc-project.org lsc-users mailing list [email protected]<mailto:[email protected]> http://lists.lsc-project.org/listinfo/lsc-users ________________________________ ________________________________
_______________________________________________________________ Ldap Synchronization Connector (LSC) - http://lsc-project.org lsc-users mailing list [email protected] http://lists.lsc-project.org/listinfo/lsc-users
