Being concerned with the slowness of the process I decided to track the execution time it takes for every script for a single user and I found:
16:24:17.582 INFO Search script finished. Time elapsed: 0.194 seconds Method: run 16:24:18.251 INFO Search script finished. Time elapsed: 0.166 seconds Method: run 16:24:18.843 INFO Update script finished. Time elapsed: 0.244 seconds Method: run 16:24:19.417 INFO Search script finished. Time elapsed: 0.161 seconds Method: run 16:24:19.996 INFO Search script finished. Time elapsed: 0.172 seconds Method: run The first most obvious question is: Why does Syncope call the search script 4 times? Additionally, I see that between the 2nd and 3rd times it calls the update script. The sum of the time elapsed for that sample is 0.937, however the time at the beginning of the line shows that the whole process can take about 3 seconds per user. Doing math, if I have 18k users then it will take: 18000*3 = 54000 seconds = 900 minutes = 15 hours (This is what is actually taking!!) What is it that Syncope does behind scenes? How can I optimize this process? Best Regards, Arnold Miller > Sent: Wednesday, March 27, 2019 at 6:33 PM > From: "Francesco Chicchiriccò" <[email protected]> > To: [email protected] > Subject: Re: Slow task > > On 27/03/19 17:21, Arnold Miller wrote: > > It is a standard Push task: Resource -> Push task -> run. > > A PushTask execution will simply generate as may PropagationTasks as > needed and then execute them; for a REST resource, it means calling the > create or update Groovy script. > > Hence: your guess is correct, if you have 18000 captured by the PushTask. > > Regards. > > > *Sent:* Wednesday, March 27, 2019 at 11:06 AM > > *From:* "Francesco Chicchiriccò" <[email protected]> > > *To:* [email protected] > > *Subject:* Re: Slow task > > Hi, > > what kind of task (the one running for 6+ hours) do you have? It's a > > standard PushTask, or a SchedTask you made on purpose? > > Regards. > > On 27/03/19 17:03, Arnold Miller wrote: > > > > No. > > My guess is that Syncope makes 18000 calls using either the create > > or the update Groovy script. > > Best Regards, > > > > Arnold Miller > > *Sent:* Wednesday, March 27, 2019 at 7:58 AM > > *From:* "Filipe Silva" <[email protected]> > > *To:* [email protected] > > *Subject:* Re: Slow task > > > > Hi > > > > Are you making only one call to get all the info? > > > > I use groovy and made pagination in the rest call. > > > > Regards > > > > Filipe Silva > > > > On 26/03/19 21:40, Arnold Miller wrote: > > > > Hi everyone, > > I have a task running for 6 hours trying to update the > > information of a total of 18000 users. It has processed only > > half of them. The task consists of calling a rest web service > > with about 10 parameters. If I run the web service call only > > it could take about 300 milliseconds. > > What am I doing wrong? > > Best Regards, > > Arnold Miller > > > -- > Francesco Chicchiriccò > > Tirasa - Open Source Excellence > http://www.tirasa.net/ > > Member at The Apache Software Foundation > Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail > http://home.apache.org/~ilgrosso/ > >
