Issue #862 has been updated by Frédéric POISSON. File AbstractSynchronize.java.patch added
Hello, I update the patch in order to have also some INFO / ERROR log when threadpool is shutting down. I found that ThreadPoolExecutor is able to give the number of [[https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ThreadPoolExecutor.html#getCompletedTaskCount%28%29]] with this method getCompletedTaskCount(). When i run an LSC task with more than 10K ids without optimizing the number of threadCount limit or the thread time limit, i have this on CONSOLE : <pre> May 25 08:42:09 - INFO - Thead count limit 10000 detected, threadPool org.lsc.SynchronizeThreadPoolExecutor@61862a7f[Running, pool size = 8, active threads = 8, queued tasks = 9992, completed tasks = 0] May 25 08:42:09 - INFO - Shutdown started on threadPool org.lsc.SynchronizeThreadPoolExecutor@61862a7f[Shutting down, pool size = 8, active threads = 8, queued tasks = 9992, completed tasks = 0] May 25 08:49:09 - INFO - threadPool org.lsc.SynchronizeThreadPoolExecutor@61862a7f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 10000] fully completed tasks May 25 08:49:09 - INFO - Thead count limit 10000 detected, threadPool org.lsc.SynchronizeThreadPoolExecutor@62150f9e[Running, pool size = 8, active threads = 8, queued tasks = 9992, completed tasks = 0] May 25 08:49:09 - INFO - Shutdown started on threadPool org.lsc.SynchronizeThreadPoolExecutor@62150f9e[Shutting down, pool size = 8, active threads = 8, queued tasks = 9992, completed tasks = 0] May 25 09:04:09 - ERROR - threadPool org.lsc.SynchronizeThreadPoolExecutor@62150f9e[Shutting down, pool size = 8, active threads = 8, queued tasks = 1934, completed tasks = 8058] partially completed tasks May 25 09:04:09 - INFO - Thead count limit 10000 detected, threadPool org.lsc.SynchronizeThreadPoolExecutor@6e20b53a[Running, pool size = 8, active threads = 8, queued tasks = 9992, completed tasks = 0] May 25 09:04:09 - INFO - Shutdown started on threadPool org.lsc.SynchronizeThreadPoolExecutor@6e20b53a[Shutting down, pool size = 8, active threads = 8, queued tasks = 9992, completed tasks = 0] May 25 09:19:09 - ERROR - threadPool org.lsc.SynchronizeThreadPoolExecutor@6e20b53a[Shutting down, pool size = 8, active threads = 8, queued tasks = 5410, completed tasks = 4582] partially completed tasks May 25 09:19:09 - INFO - Thead count limit 10000 detected, threadPool org.lsc.SynchronizeThreadPoolExecutor@661972b0[Running, pool size = 8, active threads = 8, queued tasks = 9992, completed tasks = 0] May 25 09:19:09 - INFO - Shutdown started on threadPool org.lsc.SynchronizeThreadPoolExecutor@661972b0[Shutting down, pool size = 8, active threads = 8, queued tasks = 9992, completed tasks = 0] </pre> As you can see with a thread count limit of 10000 and current pool limit of 900 seconds, the new thread pool created do not complete all the task. The first ERROR indicate 8058 completed task, the next one only 4582. It seems that the number of completed tasks is reduced over time. Do you have any patch available for feature #748 that i could test on my Lab ? My tests are only from OpenLDAP to OpenLDAP. Thanks, Regards, ---------------------------------------- Bug #862: java.lang.NullPointerException when all entries exceed 10,000 on multi threaded LSC run http://tools.lsc-project.org/issues/862 Author: Frédéric POISSON Status: Assigned Priority: Normal Assigned to: Raphaël Ouazana Category: Core Target version: 2.2 Problem in version: Hello, I've detected on some tasks with more than 10,000 ids for a single task, with a multi threaded LSC run, that i have some errors like this : <pre> ERROR - Error while synchronizing ID {...}: java.lang.NullPointerException </pre> I understand that AbstractSynchronize.java run only 10,000 entries at a time, and wait for the threadPool to finish all thread executions with a timeout of 900 seconds and then continue with the next 10,000 and so on. But at first initialization of the destination (or when there are many changes or heavy load on the servers...) it could take more than 11,11 ids per seconds to fully finish the thread executions. So the threadPool could stop some threads before their execution. That's why when i have the above ERROR i have also less ids from AbstractSynchronize.java counter than from real number of entries return by the get all filter of the source, and the above ERROR are also not count : <pre> INFO - All entries: 23042, to modify entries: 0, successfully modified entries: 0, errors: 0 </pre> That's why the 10,000 and the 900 seconds could be adapted with two JVM system properties (see patch). -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://tools.lsc-project.org/my/account
_______________________________________________________________ Ldap Synchronization Connector (LSC) - http://lsc-project.org lsc-dev mailing list [email protected] http://lists.lsc-project.org/listinfo/lsc-dev
