RE: Some scalability questions
Thanks again, Francesco! It was really helpful. Regards, Andrei. From: Francesco Chicchiriccò [mailto:ilgro...@apache.org] Sent: Dienstag, 18. Juni 2013 15:26 To: user@syncope.apache.org Subject: Re: Some scalability questions On 18/06/2013 15:01, Andrei Shakirin wrote: Hi Francesco, Thanks a lot for your answers, they are really helpful during the meeting. Three other scenarios that I was not 100% sure about: Is it possible to make propagation/synchronization directly between external resources without storing data in internal Syncope DB? My ad hoc answer was no, is it correct? Correct: you need at least to store in Syncope the accountId. The rest of attributes might be defined virtual. 1. The customer would like to fulfil kind of governance report for external system without physical synchronization: for example, discover which users are not synchronized with ActiveDirectory. Is it possible to proceed this with Syncope? Yes: you need to empower the Dry run feature of SyncTask objects - possibly, you might need to extend current reporting features. 2. Is it possible to activate synchronization tasks not only on the base of time scheduler, but also event driven? Synchronization tasks are invoked via REST interface (as any other Syncope feature): if the external resource is able to perform such call on specific event, then you can also implement event-driver synchronization. For example, I know that this could be done with MS Active Directory. HTH Regards. From: Francesco Chicchiriccò [mailto:ilgro...@apache.org] Sent: Dienstag, 18. Juni 2013 10:17 To: user@syncope.apache.orgmailto:user@syncope.apache.org Subject: Re: Some scalability questions On 17/06/2013 20:01, Andrei Shakirin wrote: Hi, Unfortunately I'm still under heavy load in another project. Hope that can contribute in Syncope again. Hi Andrei, looking forward to this! One team is evaluating Syncope as IDM solution and has number of questions that requires more experience in Syncope architecture and code. Could you please give me any advises / entry points / doc links regarding these questions: 1. Is there experience to use Syncope under heavy load? Definitely yes: I am unfortunately not allowed to disclose more details, but Syncope has been deployed in some critical environments, with considerable number of entries (users + attributes + roles). 2. Are the third party components ConnId, Activiti, Quartz scalable? What could be the potential bottlenecks: workflows, synchronization with external resources? Syncope 1.1.X uses Quartz 2.1.Y that supports clustering [1]; ConnId and Activiti features are only triggered and controlled by Syncope own logic, so their scalability is dependent on Syncope's (see below). IMO, in Syncope 1.1.X the communication (propagation / synchronization) with external resource is the hottest side, from a scalability point of view; basically, such communication is just sequential; this is the reason behind SYNCOPE-141 and SYNCOPE-142, currently scheduled for 1.3.0 [2] 3. Do Syncope components hold the internal state and can cause a problem under the cluster? Not at all: from this point of view, Syncope is barely an OpenJPA application, that empowers JPA L2 cache [3] for distributed operations. 4. Is there any experience to use Syncope under cluster in app server? Yes: besides Tomcat (which is also reported in the wiki [4]), we have direct experience of Glassfish; I don't think that JBoss should cause particular troubles. 5. Can multiple Syncope instances share the same DB? Hum, not a best practice for sure: I think - but I am not 100% sure - that JPA lock mechanism might go nuts in this scenario. I've always been relying upon two distinct replicated DB instances instead, for a number of good reasons. 6. Which syncope components provide hooks for programmatic extensions? Which functionality can be extended? Take a look at wiki pages rooted at [5] as starting point, but there even are more possibilities: workflow [6], reports, authentication / authorization [7] [8] down to admin UI [9]. Actually, one can say that Syncope is extendable by design. ...and finally, there are also fellows providing enterprise support for it ;-) HTH Regards. [1] http://quartz-scheduler.org/documentation/quartz-2.1.x/configuration/ConfigJDBCJobStoreClustering [2] https://cwiki.apache.org/confluence/display/SYNCOPE/Roadmap#Roadmap-1.3.0%28Jazz%29 [3] http://openjpa.apache.org/builds/2.2.2/apache-openjpa/docs/ref_guide_event.html#ref_guide_event_conf [4] https://cwiki.apache.org/confluence/display/SYNCOPE/Setup+a+Syncope+cluster [5] https://cwiki.apache.org/confluence/display/SYNCOPE/Extending+Syncope [6] https://cwiki.apache.org/confluence/display/SYNCOPE/User+Workflow [7] http://blog.tirasa.net/blogs/index.php/massi/apache-syncope-and-active-directory-3 [8] http://syncope-user.1051894.n5.nabble.com/Connecting
Re: Some scalability questions
On 17/06/2013 20:01, Andrei Shakirin wrote: Hi, Unfortunately I'm still under heavy load in another project. Hope that can contribute in Syncope again. Hi Andrei, looking forward to this! One team is evaluating Syncope as IDM solution and has number of questions that requires more experience in Syncope architecture and code. Could you please give me any advises / entry points / doc links regarding these questions: 1.Is there experience to use Syncope under heavy load? Definitely yes: I am unfortunately not allowed to disclose more details, but Syncope has been deployed in some critical environments, with considerable number of entries (users + attributes + roles). Are the third party components ConnId, Activiti, Quartz scalable? What could be the potential bottlenecks: workflows, synchronization with external resources? Syncope 1.1.X uses Quartz 2.1.Y that supports clustering [1]; ConnId and Activiti features are only triggered and controlled by Syncope own logic, so their scalability is dependent on Syncope's (see below). IMO, in Syncope 1.1.X the communication (propagation / synchronization) with external resource is the hottest side, from a scalability point of view; basically, such communication is just sequential; this is the reason behind SYNCOPE-141 and SYNCOPE-142, currently scheduled for 1.3.0 [2] 2.Do Syncope components hold the internal state and can cause a problem under the cluster? Not at all: from this point of view, Syncope is barely an OpenJPA application, that empowers JPA L2 cache [3] for distributed operations. Is there any experience to use Syncope under cluster in app server? Yes: besides Tomcat (which is also reported in the wiki [4]), we have direct experience of Glassfish; I don't think that JBoss should cause particular troubles. Can multiple Syncope instances share the same DB? Hum, not a best practice for sure: I think - but I am not 100% sure - that JPA lock mechanism might go nuts in this scenario. I've always been relying upon two distinct replicated DB instances instead, for a number of good reasons. 3.Which syncope components provide hooks for programmatic extensions? Which functionality can be extended? Take a look at wiki pages rooted at [5] as starting point, but there even are more possibilities: workflow [6], reports, authentication / authorization [7] [8] down to admin UI [9]. Actually, one can say that Syncope is extendable by design. ...and finally, there are also fellows providing enterprise support for it ;-) HTH Regards. [1] http://quartz-scheduler.org/documentation/quartz-2.1.x/configuration/ConfigJDBCJobStoreClustering [2] https://cwiki.apache.org/confluence/display/SYNCOPE/Roadmap#Roadmap-1.3.0%28Jazz%29 [3] http://openjpa.apache.org/builds/2.2.2/apache-openjpa/docs/ref_guide_event.html#ref_guide_event_conf [4] https://cwiki.apache.org/confluence/display/SYNCOPE/Setup+a+Syncope+cluster [5] https://cwiki.apache.org/confluence/display/SYNCOPE/Extending+Syncope [6] https://cwiki.apache.org/confluence/display/SYNCOPE/User+Workflow [7] http://blog.tirasa.net/blogs/index.php/massi/apache-syncope-and-active-directory-3 [8] http://syncope-user.1051894.n5.nabble.com/Connecting-Syncope-to-CAS-tt5706823.html [9] http://blog.tirasa.net/blogs/index.php/coffeetime/customize-your-apache-syncope-console -- Francesco Chicchiriccò ASF Member, Apache Syncope PMC chair, Apache Cocoon PMC Member http://people.apache.org/~ilgrosso/
Re: Some scalability questions
Il 18/06/2013 10:16, Francesco Chicchiriccò ha scritto: On 17/06/2013 20:01, Andrei Shakirin wrote: Hi, Unfortunately I'm still under heavy load in another project. Hope that can contribute in Syncope again. Hi Andrei, looking forward to this! One team is evaluating Syncope as IDM solution and has number of questions that requires more experience in Syncope architecture and code. Could you please give me any advises / entry points / doc links regarding these questions: 1.Is there experience to use Syncope under heavy load? Definitely yes: I am unfortunately not allowed to disclose more details, but Syncope has been deployed in some critical environments, with considerable number of entries (users + attributes + roles). Are the third party components ConnId, Activiti, Quartz scalable? What could be the potential bottlenecks: workflows, synchronization with external resources? Syncope 1.1.X uses Quartz 2.1.Y that supports clustering [1]; ConnId and Activiti features are only triggered and controlled by Syncope own logic, so their scalability is dependent on Syncope's (see below). IMO, in Syncope 1.1.X the communication (propagation / synchronization) with external resource is the hottest side, from a scalability point of view; basically, such communication is just sequential; this is the reason behind SYNCOPE-141 and SYNCOPE-142, currently scheduled for 1.3.0 [2] 2.Do Syncope components hold the internal state and can cause a problem under the cluster? Not at all: from this point of view, Syncope is barely an OpenJPA application, that empowers JPA L2 cache [3] for distributed operations. Is there any experience to use Syncope under cluster in app server? Yes: besides Tomcat (which is also reported in the wiki [4]), we have direct experience of Glassfish; I don't think that JBoss should cause particular troubles. Can multiple Syncope instances share the same DB? Hum, not a best practice for sure: I think - but I am not 100% sure - that JPA lock mechanism might go nuts in this scenario. I've always been relying upon two distinct replicated DB instances instead, for a number of good reasons. 3.Which syncope components provide hooks for programmatic extensions? Which functionality can be extended? Take a look at wiki pages rooted at [5] as starting point, but there even are more possibilities: workflow [6], reports, authentication / authorization [7] [8] down to admin UI [9]. Actually, one can say that Syncope is extendable by design. ...and finally, there are also fellows providing enterprise support for it ;-) http://syncope.apache.org/professional-services.html HTH Regards. [1] http://quartz-scheduler.org/documentation/quartz-2.1.x/configuration/ConfigJDBCJobStoreClustering [2] https://cwiki.apache.org/confluence/display/SYNCOPE/Roadmap#Roadmap-1.3.0%28Jazz%29 [3] http://openjpa.apache.org/builds/2.2.2/apache-openjpa/docs/ref_guide_event.html#ref_guide_event_conf [4] https://cwiki.apache.org/confluence/display/SYNCOPE/Setup+a+Syncope+cluster [5] https://cwiki.apache.org/confluence/display/SYNCOPE/Extending+Syncope [6] https://cwiki.apache.org/confluence/display/SYNCOPE/User+Workflow [7] http://blog.tirasa.net/blogs/index.php/massi/apache-syncope-and-active-directory-3 [8] http://syncope-user.1051894.n5.nabble.com/Connecting-Syncope-to-CAS-tt5706823.html [9] http://blog.tirasa.net/blogs/index.php/coffeetime/customize-your-apache-syncope-console -- Francesco Chicchiriccò ASF Member, Apache Syncope PMC chair, Apache Cocoon PMC Member http://people.apache.org/~ilgrosso/
RE: Some scalability questions
Hi Francesco, Thanks a lot for your answers, they are really helpful during the meeting. Three other scenarios that I was not 100% sure about: 1. Is it possible to make propagation/synchronization directly between external resources without storing data in internal Syncope DB? My ad hoc answer was no, is it correct? 2. The customer would like to fulfil kind of governance report for external system without physical synchronization: for example, discover which users are not synchronized with ActiveDirectory. Is it possible to proceed this with Syncope? 3. Is it possible to activate synchronization tasks not only on the base of time scheduler, but also event driven? Regards, Andrei. From: Francesco Chicchiriccò [mailto:ilgro...@apache.org] Sent: Dienstag, 18. Juni 2013 10:17 To: user@syncope.apache.org Subject: Re: Some scalability questions On 17/06/2013 20:01, Andrei Shakirin wrote: Hi, Unfortunately I'm still under heavy load in another project. Hope that can contribute in Syncope again. Hi Andrei, looking forward to this! One team is evaluating Syncope as IDM solution and has number of questions that requires more experience in Syncope architecture and code. Could you please give me any advises / entry points / doc links regarding these questions: 1. Is there experience to use Syncope under heavy load? Definitely yes: I am unfortunately not allowed to disclose more details, but Syncope has been deployed in some critical environments, with considerable number of entries (users + attributes + roles). 2. Are the third party components ConnId, Activiti, Quartz scalable? What could be the potential bottlenecks: workflows, synchronization with external resources? Syncope 1.1.X uses Quartz 2.1.Y that supports clustering [1]; ConnId and Activiti features are only triggered and controlled by Syncope own logic, so their scalability is dependent on Syncope's (see below). IMO, in Syncope 1.1.X the communication (propagation / synchronization) with external resource is the hottest side, from a scalability point of view; basically, such communication is just sequential; this is the reason behind SYNCOPE-141 and SYNCOPE-142, currently scheduled for 1.3.0 [2] 3. Do Syncope components hold the internal state and can cause a problem under the cluster? Not at all: from this point of view, Syncope is barely an OpenJPA application, that empowers JPA L2 cache [3] for distributed operations. 4. Is there any experience to use Syncope under cluster in app server? Yes: besides Tomcat (which is also reported in the wiki [4]), we have direct experience of Glassfish; I don't think that JBoss should cause particular troubles. 5. Can multiple Syncope instances share the same DB? Hum, not a best practice for sure: I think - but I am not 100% sure - that JPA lock mechanism might go nuts in this scenario. I've always been relying upon two distinct replicated DB instances instead, for a number of good reasons. 6. Which syncope components provide hooks for programmatic extensions? Which functionality can be extended? Take a look at wiki pages rooted at [5] as starting point, but there even are more possibilities: workflow [6], reports, authentication / authorization [7] [8] down to admin UI [9]. Actually, one can say that Syncope is extendable by design. ...and finally, there are also fellows providing enterprise support for it ;-) HTH Regards. [1] http://quartz-scheduler.org/documentation/quartz-2.1.x/configuration/ConfigJDBCJobStoreClustering [2] https://cwiki.apache.org/confluence/display/SYNCOPE/Roadmap#Roadmap-1.3.0%28Jazz%29 [3] http://openjpa.apache.org/builds/2.2.2/apache-openjpa/docs/ref_guide_event.html#ref_guide_event_conf [4] https://cwiki.apache.org/confluence/display/SYNCOPE/Setup+a+Syncope+cluster [5] https://cwiki.apache.org/confluence/display/SYNCOPE/Extending+Syncope [6] https://cwiki.apache.org/confluence/display/SYNCOPE/User+Workflow [7] http://blog.tirasa.net/blogs/index.php/massi/apache-syncope-and-active-directory-3 [8] http://syncope-user.1051894.n5.nabble.com/Connecting-Syncope-to-CAS-tt5706823.html [9] http://blog.tirasa.net/blogs/index.php/coffeetime/customize-your-apache-syncope-console -- Francesco Chicchiriccò ASF Member, Apache Syncope PMC chair, Apache Cocoon PMC Member http://people.apache.org/~ilgrosso/
Re: Some scalability questions
On 18/06/2013 15:01, Andrei Shakirin wrote: Hi Francesco, Thanks a lot for your answers, they are really helpful during the meeting. Three other scenarios that I was not 100% sure about: 1.Is it possible to make propagation/synchronization directly between external resources without storing data in internal Syncope DB? My ad hoc answer was no, is it correct? Correct: you need at least to store in Syncope the accountId. The rest of attributes might be defined virtual. 2.The customer would like to fulfil kind of governance report for external system without physical synchronization: for example, discover which users are not synchronized with ActiveDirectory. Is it possible to proceed this with Syncope? Yes: you need to empower the Dry run feature of SyncTask objects - possibly, you might need to extend current reporting features. 3.Is it possible to activate synchronization tasks not only on the base of time scheduler, but also event driven? Synchronization tasks are invoked via REST interface (as any other Syncope feature): if the external resource is able to perform such call on specific event, then you can also implement event-driver synchronization. For example, I know that this could be done with MS Active Directory. HTH Regards. *From:*Francesco Chicchiriccò [mailto:ilgro...@apache.org] *Sent:* Dienstag, 18. Juni 2013 10:17 *To:* user@syncope.apache.org *Subject:* Re: Some scalability questions On 17/06/2013 20:01, Andrei Shakirin wrote: Hi, Unfortunately I'm still under heavy load in another project. Hope that can contribute in Syncope again. Hi Andrei, looking forward to this! One team is evaluating Syncope as IDM solution and has number of questions that requires more experience in Syncope architecture and code. Could you please give me any advises / entry points / doc links regarding these questions: 1.Is there experience to use Syncope under heavy load? Definitely yes: I am unfortunately not allowed to disclose more details, but Syncope has been deployed in some critical environments, with considerable number of entries (users + attributes + roles). 2.Are the third party components ConnId, Activiti, Quartz scalable? What could be the potential bottlenecks: workflows, synchronization with external resources? Syncope 1.1.X uses Quartz 2.1.Y that supports clustering [1]; ConnId and Activiti features are only triggered and controlled by Syncope own logic, so their scalability is dependent on Syncope's (see below). IMO, in Syncope 1.1.X the communication (propagation / synchronization) with external resource is the hottest side, from a scalability point of view; basically, such communication is just sequential; this is the reason behind SYNCOPE-141 and SYNCOPE-142, currently scheduled for 1.3.0 [2] 3.Do Syncope components hold the internal state and can cause a problem under the cluster? Not at all: from this point of view, Syncope is barely an OpenJPA application, that empowers JPA L2 cache [3] for distributed operations. 4.Is there any experience to use Syncope under cluster in app server? Yes: besides Tomcat (which is also reported in the wiki [4]), we have direct experience of Glassfish; I don't think that JBoss should cause particular troubles. 5.Can multiple Syncope instances share the same DB? Hum, not a best practice for sure: I think - but I am not 100% sure - that JPA lock mechanism might go nuts in this scenario. I've always been relying upon two distinct replicated DB instances instead, for a number of good reasons. 6.Which syncope components provide hooks for programmatic extensions? Which functionality can be extended? Take a look at wiki pages rooted at [5] as starting point, but there even are more possibilities: workflow [6], reports, authentication / authorization [7] [8] down to admin UI [9]. Actually, one can say that Syncope is extendable by design. ...and finally, there are also fellows providing enterprise support for it ;-) HTH Regards. [1] http://quartz-scheduler.org/documentation/quartz-2.1.x/configuration/ConfigJDBCJobStoreClustering [2] https://cwiki.apache.org/confluence/display/SYNCOPE/Roadmap#Roadmap-1.3.0%28Jazz%29 [3] http://openjpa.apache.org/builds/2.2.2/apache-openjpa/docs/ref_guide_event.html#ref_guide_event_conf [4] https://cwiki.apache.org/confluence/display/SYNCOPE/Setup+a+Syncope+cluster [5] https://cwiki.apache.org/confluence/display/SYNCOPE/Extending+Syncope [6] https://cwiki.apache.org/confluence/display/SYNCOPE/User+Workflow [7] http://blog.tirasa.net/blogs/index.php/massi/apache-syncope-and-active-directory-3 [8] http://syncope-user.1051894.n5.nabble.com/Connecting-Syncope-to-CAS-tt5706823.html [9] http://blog.tirasa.net/blogs/index.php/coffeetime/customize-your-apache-syncope-console -- Francesco Chicchiriccò ASF Member, Apache Syncope PMC chair, Apache Cocoon PMC Member http://people.apache.org/~ilgrosso/
