I had a feeling it was taking longer than it should. Our infrastructure is not a problem...it must be with the config.
Can you take a peek at my config and let me know if there is anything I should optimize? I am basically taking a user ID and Pin number from SQL and importing that as a user in LDAP. <?xml version="1.0" ?> <lsc xmlns="http://lsc-project.org/XSD/lsc-core-2.1.xsd"; revision="0"> <connections> <databaseConnection> <name>SQL</name> <url>jdbc:sqlserver://server:port;DatabseName=DB</url> <username>SQL2LDAP</username> <password>PASSWORD</password> <driver>com.microsoft.sqlserver.jdbc.SQLServerDriver</driver> </databaseConnection> <ldapConnection> <name>ldap-dst-conn</name> <url>ldap://localhost:389/dc=test,dc=com</url> <username>cn=Manager,dc=test,dc=com</username> <password>PASSWORD</password> <authentication>SIMPLE</authentication> <referral>IGNORE</referral> <derefAliases>NEVER</derefAliases> <version>VERSION_3</version> <pageSize>-1</pageSize> <factory>com.sun.jndi.ldap.LdapCtxFactory</factory> <tlsActivated>false</tlsActivated> </ldapConnection> </connections> <audits> <csvAudit> <name>csv</name> <append>true</append> <operations>create,delete</operations> <file>/tmp/dump.csv</file> <datasets>cn,dn</datasets> <separator>,</separator> </csvAudit> </audits> <tasks> <task> <name>TASK1</name> <bean>org.lsc.beans.SimpleBean</bean> <databaseSourceService> <name>sql-lpp-source-service</name> <connection reference="SQL" /> <requestNameForList>getInetOrgPersonList</requestNameForList> <requestNameForObject>getInetOrgPerson</requestNameForObject> <requestNameForClean>getInetOrgPersonClean</requestNameForClean> </databaseSourceService> <ldapDestinationService> <name>ldap-lpp-dst-service</name> <connection reference="ldap-dst-conn" /> <baseDn>ou=OU1,dc=test,dc=com</baseDn> <pivotAttributes> <string>uid</string> </pivotAttributes> <fetchedAttributes> <string>cn</string> <string>sn</string> <string>userPassword</string> <string>objectClass</string> <string>uid</string> </fetchedAttributes> <getAllFilter>(objectClass=inetorgperson)</getAllFilter> <getOneFilter>(&(objectClass=inetorgperson)(uid={uid}))</getOneFilter> </ldapDestinationService> <propertiesBasedSyncOptions> <mainIdentifier>"uid=" + srcBean.getDatasetFirstValueById("uid") + ",ou=OU1,dc=test,dc=com"</mainIdentifier> <defaultDelimiter>;</defaultDelimiter> <defaultPolicy>FORCE</defaultPolicy> <conditions> <create>true</create> <update>true</update> <delete>true</delete> <changeId>false</changeId> </conditions> <dataset> <name>objectClass</name> <policy>FORCE</policy> <defaultValues></defaultValues> <forceValues> <string>"inetOrgPerson"</string> <string>"organizationalPerson"</string> <string>"person"</string> <string>"top"</string> </forceValues> <delimiter>,</delimiter> </dataset> <!--<dataset> <name>userPassword</name> <policy>KEEP</policy> <defaultValues></defaultValues> <forceValues></forceValues> <createValues></createValues> </dataset>--> </propertiesBasedSyncOptions> </task> <task> <name>TASK2</name> <bean>org.lsc.beans.SimpleBean</bean> <databaseSourceService> <name>sql-res-source-service</name> <connection reference="SQL" /> <requestNameForList>ResgetInetOrgPersonList</requestNameForList> <requestNameForObject>ResgetInetOrgPerson</requestNameForObject> <requestNameForClean>ResgetInetOrgPersonClean</requestNameForClean> </databaseSourceService> <ldapDestinationService> <name>ldap-res-dst-service</name> <connection reference="ldap-dst-conn" /> <baseDn>ou=OU2,dc=test,dc=com</baseDn> <pivotAttributes> <string>uid</string> </pivotAttributes> <fetchedAttributes> <string>cn</string> <string>sn</string> <string>userPassword</string> <string>objectClass</string> <string>uid</string> </fetchedAttributes> <getAllFilter>(objectClass=inetorgperson)</getAllFilter> <getOneFilter>(&(objectClass=inetorgperson)(uid={uid}))</getOneFilter> </ldapDestinationService> <propertiesBasedSyncOptions> <mainIdentifier>"uid=" + srcBean.getDatasetFirstValueById("uid") + ",ou=OU2,dc=test,dc=com"</mainIdentifier> <defaultDelimiter>;</defaultDelimiter> <defaultPolicy>FORCE</defaultPolicy> <conditions> <create>true</create> <update>true</update> <delete>true</delete> <changeId>false</changeId> </conditions> <dataset> <name>objectClass</name> <policy>FORCE</policy> <defaultValues></defaultValues> <forceValues> <string>"inetOrgPerson"</string> <string>"organizationalPerson"</string> <string>"person"</string> <string>"top"</string> </forceValues> <delimiter>,</delimiter> </dataset> <!--<dataset> <name>userPassword</name> <policy>KEEP</policy> <defaultValues></defaultValues> <forceValues></forceValues> <createValues></createValues> </dataset>--> </propertiesBasedSyncOptions> </task> </tasks> <security> <encryption> <keyfile>etc/lsc.key</keyfile> <algorithm>AES</algorithm> <strength>128</strength> </encryption> </security> </lsc> <?xml version="1.0" encoding="UTF-8" standalone="no"?> <!DOCTYPE sqlMap PUBLIC "-//iBATIS.com//DTD SQL Map 2.0//EN" "http://www.ibatis.com/dtd/sql-map-2.dtd";> <sqlMap namespace="InetOrgPerson"> <select id="getInetOrgPerson" resultClass="java.util.HashMap" parameterClass="java.util.Map"> SELECT "ID" AS uid, "ID" AS sn, "ID" AS cn, "LPPpin" AS userPassword FROM table1 WHERE "ID" = #uid# </select> <select id="getInetOrgPersonList" resultClass="java.util.HashMap"> SELECT "ID" AS uid FROM table1 </select> <select id="getInetOrgPersonClean" resultClass="java.util.HashMap" parameterClass="java.util.Map"> SELECT "ID" AS uid FROM table1 WHERE "ID" = #uid# </select> <select id="ResgetInetOrgPerson" resultClass="java.util.HashMap" parameterClass="java.util.Map"> SELECT "ID2" AS uid, "ID2" AS sn, "ID2" AS cn, "ResidentPin" AS userPassword FROM table2 WHERE "ID2" = #uid# </select> <select id="ResgetInetOrgPersonList" resultClass="java.util.HashMap"> SELECT "ID2" AS uid FROM table2 </select> <select id="ResgetInetOrgPersonClean" resultClass="java.util.HashMap" parameterClass="java.util.Map"> SELECT "ID2" AS uid FROM table2 WHERE "ID2" = #uid# </select> </sqlMap> From: [email protected] [mailto:[email protected]] On Behalf Of Clément OUDOT Sent: Wednesday, September 30, 2015 7:10 AM To: [email protected] Subject: Re: [lsc-users] SQL to LDAP Sync takes several hours Le 30/09/2015 12:52, Parrish, Kyle a écrit : I have read through the documentation and understand how LSC works I just want to know if it is normal behavior for a sync of 100,000 entries to take 1.5+ hours with or without changes. If you read the doc, you understand that a SELECT will be done for each entry in the source, means LSC will do 100 000 SELECT. I have worked with databases that can take this load in a few minutes, it depends on the configuration of this database and on the infrastructure (memory, cpu, ...) So you should be able to run LSC on your database in far less than 1.5 hour, but it relies on how you configure the requests (Ibastis) and your database. -- Clément OUDOT Consultant en logiciels libres, Expert infrastructure et sécurité Savoir-faire Linux
_______________________________________________________________ Ldap Synchronization Connector (LSC) - http://lsc-project.org lsc-users mailing list [email protected] http://lists.lsc-project.org/listinfo/lsc-users
