Hi Everyone,
We're trying to sync large groups from a database to our LDAP as per the method
described at
http://lsc-project.org/wiki/documentation/tutorial/synchronizegroups
This works great for smaller groups but breaks when trying to sync very large
groups (> 600,000 members)
We get the following in our log:
…
Jul 12 15:29:48 - DEBUG - In object
"cn=service1,ou=SERVICE,ou=PERMISSION,ou=GROUPS,ou=IDM,dc=dev": List of
attributes considered for writing in destination: [member, cn, description,
objectClass]
Jul 12 15:29:48 - DEBUG - In object
"cn=service1,ou=SERVICE,ou=PERMISSION,ou=GROUPS,ou=IDM,dc=dev": Attribute
"member" is in FORCE status
Jul 12 15:31:01 - DEBUG - In object
"cn=service1,ou=SERVICE,ou=PERMISSION,ou=GROUPS,ou=IDM,dc=dev": Adding
attribute "member" with
values [<giant array of uid's>]
Jul 12 15:33:47 - DEBUG - In object
"cn=service1,ou=SERVICE,ou=PERMISSION,ou=GROUPS,ou=IDM,dc=dev": Attribute "cn"
is in KEEP statu
s
Jul 12 15:33:47 - DEBUG - In object
"cn=service1,ou=SERVICE,ou=PERMISSION,ou=GROUPS,ou=IDM,dc=dev": Adding
attribute "cn" with valu
es [ca.ubc.service.iap]
Jul 12 15:33:47 - DEBUG - In object
"cn=service1,ou=SERVICE,ou=PERMISSION,ou=GROUPS,ou=IDM,dc=dev": Attribute
"description" is in K
EEP status
Jul 12 15:33:47 - DEBUG - In object
"cn=service1,ou=SERVICE,ou=PERMISSION,ou=GROUPS,ou=IDM,dc=dev": Adding
attribute "description"
with values [Service 1 Description]
Jul 12 15:33:47 - DEBUG - In object
"cn=service1,ou=SERVICE,ou=PERMISSION,ou=GROUPS,ou=IDM,dc=dev": Attribute
"objectClass" is in K
EEP status
Jul 12 15:33:48 - DEBUG - In object
"cn=service1,ou=SERVICE,ou=PERMISSION,ou=GROUPS,ou=IDM,dc=dev": Adding
attribute "objectClass"
with values [top, groupOfNames, nestedGroup]
Jul 12 16:26:45 - INFO - All entries: 1, to modify entries: 1, successfully
modified entries: 0, errors: 0
Jul 12 17:48:17 - ERROR - Error while synchronizing ID
cn=service1,ou=SERVICE,ou=PERMISSION,ou=GROUPS,ou=IDM,dc=dev:
java.lang.NullPointerException
Jul 12 17:48:19 - DEBUG - java.lang.NullPointerException
java.lang.NullPointerException: null
at org.lsc.jndi.JndiServices.doApply(JndiServices.java:822)
~[lsc-core-2.1.3.jar:na]
at org.lsc.jndi.JndiServices.apply(JndiServices.java:792)
~[lsc-core-2.1.3.jar:na]
at
org.lsc.jndi.SimpleJndiDstService.apply(SimpleJndiDstService.java:212)
~[lsc-core-2.1.3.jar:na]
at org.lsc.SynchronizeTask.run(AbstractSynchronize.java:795)
[lsc-core-2.1.3.jar:na]
at org.lsc.SynchronizeTask.run(AbstractSynchronize.java:707)
[lsc-core-2.1.3.jar:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_80]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_80]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
...
Looking at the LDAP access log, we see
[12/Jul/2016:15:26:42 -0700] conn=110698 fd=290 slot=290 SSL connection from
<IP> to <IP>
[12/Jul/2016:15:26:42 -0700] conn=110698 SSL 256-bit AES
[12/Jul/2016:15:26:42 -0700] conn=110698 op=0 BIND
dn="uid=service-account,ou=SERVICES,dc=dev" method=128 version=3
[12/Jul/2016:15:26:42 -0700] conn=110698 op=0 RESULT err=0 tag=97 nentries=0
etime=0 dn="uid=service-account,ou=services,dc=dev,dc=id,d
c=ubc,dc=ca"
...
[12/Jul/2016:15:29:47 -0700] conn=110698 op=1 SRCH
base="ou=SERVICE,ou=PERMISSION,ou=GROUPS,ou=IDM,dc=dev" scope=2
filter="(&(objectClass=groupOfNames)(cn=ca.ubc.service.iap))" attrs="cn member
description objectClass"
[12/Jul/2016:15:29:47 -0700] conn=110698 op=1 RESULT err=0 tag=101 nentries=0
etime=0
...
[12/Jul/2016:16:26:46 -0700] conn=110698 op=3 UNBIND
[12/Jul/2016:16:26:46 -0700] conn=110698 op=3 fd=290 closed - U1
It looks like the connection is only open for 1 hr; I'm suspecting that when
the LDAP update is eventually attempted at 17:48:17, the connection is no
longer open and causes the NullPointer exception. Is there a way to keep the
connection open longer or some other approach that would be faster? Or
anything at all?
Thanks a lot,
Trev
_________________________________________________
Trevor Fong
Senior Programmer Analyst
Information Technology | Engage. Envision. Enable.
The University of British Columbia
[email protected]<mailto:[email protected]> |
1-604-827-5247<tel:604-827-5247> | it.ubc.ca<http://it.ubc.ca>
_______________________________________________________________
Ldap Synchronization Connector (LSC) - http://lsc-project.org
lsc-users mailing list
[email protected]
http://lists.lsc-project.org/listinfo/lsc-users
