Oops - just realized I hadn't simplified the logged info properly. I've edited it below to make it a bit more internally consistent.
Trev From: Trevor Fong Date: Wednesday, July 13, 2016 at 11:21 AM To: "[email protected]<mailto:[email protected]>" Subject: Syncing Large Groups from DB to LDAP Hi Everyone, We're trying to sync large groups from a database to our LDAP as per the method described at http://lsc-project.org/wiki/documentation/tutorial/synchronizegroups This works great for smaller groups but breaks when trying to sync very large groups (> 600,000 members) We get the following in our log: … Jul 12 15:29:48 - DEBUG - In object "cn=service1,ou=SERVICE,ou=PERMISSION,ou=GROUPS,ou=IDM,dc=dev": List of attributes considered for writing in destination: [member, cn, description, objectClass] Jul 12 15:29:48 - DEBUG - In object "cn=service1,ou=SERVICE,ou=PERMISSION,ou=GROUPS,ou=IDM,dc=dev": Attribute "member" is in FORCE status Jul 12 15:31:01 - DEBUG - In object "cn=service1,ou=SERVICE,ou=PERMISSION,ou=GROUPS,ou=IDM,dc=dev": Adding attribute "member" with values [<giant array of uid's>] Jul 12 15:33:47 - DEBUG - In object "cn=service1,ou=SERVICE,ou=PERMISSION,ou=GROUPS,ou=IDM,dc=dev": Attribute "cn" is in KEEP status Jul 12 15:33:47 - DEBUG - In object "cn=service1,ou=SERVICE,ou=PERMISSION,ou=GROUPS,ou=IDM,dc=dev": Adding attribute "cn" with values [service1] Jul 12 15:33:47 - DEBUG - In object "cn=service1,ou=SERVICE,ou=PERMISSION,ou=GROUPS,ou=IDM,dc=dev": Attribute "description" is in KEEP status Jul 12 15:33:47 - DEBUG - In object "cn=service1,ou=SERVICE,ou=PERMISSION,ou=GROUPS,ou=IDM,dc=dev": Adding attribute "description" with values [Service 1 Description] Jul 12 15:33:47 - DEBUG - In object "cn=service1,ou=SERVICE,ou=PERMISSION,ou=GROUPS,ou=IDM,dc=dev": Attribute "objectClass" is in KEEP status Jul 12 15:33:48 - DEBUG - In object "cn=service1,ou=SERVICE,ou=PERMISSION,ou=GROUPS,ou=IDM,dc=dev": Adding attribute "objectClass" with values [top, groupOfNames, nestedGroup] Jul 12 16:26:45 - INFO - All entries: 1, to modify entries: 1, successfully modified entries: 0, errors: 0 Jul 12 17:48:17 - ERROR - Error while synchronizing ID cn=service1,ou=SERVICE,ou=PERMISSION,ou=GROUPS,ou=IDM,dc=dev: java.lang.NullPointerException Jul 12 17:48:19 - DEBUG - java.lang.NullPointerException java.lang.NullPointerException: null at org.lsc.jndi.JndiServices.doApply(JndiServices.java:822) ~[lsc-core-2.1.3.jar:na] at org.lsc.jndi.JndiServices.apply(JndiServices.java:792) ~[lsc-core-2.1.3.jar:na] at org.lsc.jndi.SimpleJndiDstService.apply(SimpleJndiDstService.java:212) ~[lsc-core-2.1.3.jar:na] at org.lsc.SynchronizeTask.run(AbstractSynchronize.java:795) [lsc-core-2.1.3.jar:na] at org.lsc.SynchronizeTask.run(AbstractSynchronize.java:707) [lsc-core-2.1.3.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_80] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_80] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80] ... Looking at the LDAP access log, we see [12/Jul/2016:15:26:42 -0700] conn=110698 fd=290 slot=290 SSL connection from <IP> to <IP> [12/Jul/2016:15:26:42 -0700] conn=110698 SSL 256-bit AES [12/Jul/2016:15:26:42 -0700] conn=110698 op=0 BIND dn="uid=service-account,ou=SERVICES,dc=dev" method=128 version=3 [12/Jul/2016:15:26:42 -0700] conn=110698 op=0 RESULT err=0 tag=97 nentries=0 etime=0 dn="uid=service-account,ou=services,dc=dev" ... [12/Jul/2016:15:29:47 -0700] conn=110698 op=1 SRCH base="ou=SERVICE,ou=PERMISSION,ou=GROUPS,ou=IDM,dc=dev" scope=2 filter="(&(objectClass=groupOfNames)(cn=service1))" attrs="cn member description objectClass" [12/Jul/2016:15:29:47 -0700] conn=110698 op=1 RESULT err=0 tag=101 nentries=0 etime=0 ... [12/Jul/2016:16:26:46 -0700] conn=110698 op=3 UNBIND [12/Jul/2016:16:26:46 -0700] conn=110698 op=3 fd=290 closed - U1 It looks like the connection is only open for 1 hr; I'm suspecting that when the LDAP update is eventually attempted at 17:48:17, the connection is no longer open and causes the NullPointer exception. Is there a way to keep the connection open longer or some other approach that would be faster? Or anything at all? Thanks a lot, Trev _________________________________________________ Trevor Fong Senior Programmer Analyst Information Technology | Engage. Envision. Enable. The University of British Columbia [email protected]<mailto:[email protected]> | 1-604-827-5247<tel:604-827-5247> | it.ubc.ca<http://it.ubc.ca>
_______________________________________________________________ Ldap Synchronization Connector (LSC) - http://lsc-project.org lsc-users mailing list [email protected] http://lists.lsc-project.org/listinfo/lsc-users
