Issue #864 has been updated by Raphaël Ouazana.
You proposal is interesting, in fact it means to replace time by memory. In case you have only 1000 entries you'll get better performance. But what about 1 billion entries? You won't be able to load all the entries in memory, while with the actual strategy you can load 1 billion keys and load progressively the entries. We can eventually implement different load strategies depending on some parameters or on the size of the dataset, but this would require many changes and I'm not sure it's worth the pain. ---------------------------------------- Feature #864: More Efficient Processing for Database Source http://tools.lsc-project.org/issues/864 Author: Trevor Fong Status: New Priority: Normal Assigned to: Category: Core Target version: Not planned Currently, LSC processes sync from a database source in the following fashion: 1. Run query for requestNameForList node to get a list of ID's to process from the source database 2. For each of those ID's run query for requestNameForObject to obtain the object to sync 3. Sync the object to the destination This could be made more efficient by doing the following: 1. Run query for requestNameForData node to get a result set from the source database containing all rows and columns to process 2. Iterate through the result set rows (each row being the object to sync) 3. Sync the object to the destination For the current method, for a database containing N number of accounts to sync, N+1 database connections and queries must be made, whereas for the proposed method, only 1 database connection and query need be made. If each List query takes M seconds and each Object query takes P seconds, total time for database activity would be P+(NxM) seconds. For the proposed method, total time for database activity would be P seconds. For example, if there were 1000 records, the List query takes 1 sec, and each Object database query returns in 0.1 sec: - current: total database time will be 1 + (1000 x 0.1) = 101 sec. - proposed: total database time will be 1 sec. -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://tools.lsc-project.org/my/account
_______________________________________________________________ Ldap Synchronization Connector (LSC) - http://lsc-project.org lsc-dev mailing list [email protected] http://lists.lsc-project.org/listinfo/lsc-dev
