Re: [Architecture] [Dev][IAM] Moving File Based Artifacts to Artifact Store
I've dealt with customers who treat user-stores as artifacts. These are mainly customers on multi-tenancy deployment patterns. While its rare that they add user-stores (tenants), they follow a strict process when pushing these to production. They create the user-stores in the dev environment using the UI, but beyond that the UI is never used. Those are pushed to upper environments by dev-ops folks. Regarding SPs, I think these are artifacts. I won't treat SPs as configs. SPs are simply applications. Specially when IAM is being used with API Manager, Applications (SPs) are created quite often in lower environments using UIs and they need to be pushed to production through dev-ops processes. I think the same should be true for IDPs as well, although I agree IDPs are much lesser added/updated. On Mon, Jul 8, 2019 at 11:08 AM Johann Nallathamby wrote: > Hi Ruwan, > > Interestingly we both agree on database persistence, however I am trying > to find more conviction on treating Identity Admin UI configurations as > "artifacts" and not merely "configurations". I would like to take this > opportunity to weigh in my opinion on the matter, just to make sure there > is enough reasoning behind this change. I believe this is an important > distinction because, > 1. If we agree with "artifacts" then we can't stop with user stores, but > have to eventually transition all Identity Admin UI configurations to this > model. Though there is no technical limitations in this approach, it takes > a lot of effort, and that effort better be worth it. > 2. It changes the complete outlook of the product, and could differentiate > us from competitors, in a good way or bad way. > > In my understanding I think there are three *independent* questions to > answer. > 1. File based persistence vs. Database persistence for these configurations > 2. Providing file based representation for configurations > 3. Definition of artifacts vs. configurations > > I think on the first point we agree that database persistence is a better > fit for IS, if not for anything, just because all Identity Admin UI > configurations except user stores are currently in database for IS and we > don't see any major problem with it. I think this is what is being > discussed in the mail thread [1], and I tend to agree on most of the points > @KasunG > Gajasinghe is mentioning. > > On the second point, we already have this in IS, although we currently > only have it for SPs; we need to cover the rest as well. I guess we can > agree on this need as well, as we've had many enterprise customers who have > had strong requirements for import/export, automation, product upgrades, > environment migration, etc. and asked for some file format to export to. > Having it as a human readable text file makes only makes more sense. > However, I don't know if we've figured out how to handle secrets in these > files. > > On the third point, to me "artifacts" mean more than file persistence and > file representation. > > If we go by the definition of, anything that is created by the application > user at runtime is an artifact, then I will have to agree that SP, IdP, > user stores, XACML and everything else is an artifact. However, in a > practical sense, I see some differences between, APIs, mediation sequences, > BPEL processes, Siddhi scripts, XACML policies, and service providers or > identity providers or user stores. > > 1. APIs, mediation sequences, BPEL processes, Siddhi scripts, and XACML > policies have development time aspect, hence involves development time > governance. Where as service providers, identity providers and user stores > don't really require that. > > 2. Artifacts generally have lifecycles, versioning, metadata and > properties. Are we sure we want to make all Identity Admin UI > configurations also fit into that definition? > > 3. Do we need change management and continuous delivery for Identity Admin > UI configurations like service providers, identity providers and user > stores? Have we had any customer request for this level development > governance for IS? > > 4. CI/CD can be generally expected as a key requirement for organizations > that aspire to have a good integration or API management story. But I don't > think it is right to expect it from an IAM prospect. In most cases IAM is > the first product they onboard even before EI or API Manager or other > enterprise applications. > > 5. In 90% of the cases Identity Admin UI configurations are going to be > done by one or couple of people in an organization, which is not the case > with other artifacts mentioned above. And we don't expect changes to > service providers, identity providers and user stores, as frequent as other > artifacts. Will this not be considered an overhead for something that one > or two people change very rarely? > > 6. Most Identity Admins are not big fans of developer tools. They prefer > the old fashioned management UIs. If so does the whole development > governance story
Re: [Architecture] [Dev][IAM] Moving File Based Artifacts to Artifact Store
Hi Ruwan, Interestingly we both agree on database persistence, however I am trying to find more conviction on treating Identity Admin UI configurations as "artifacts" and not merely "configurations". I would like to take this opportunity to weigh in my opinion on the matter, just to make sure there is enough reasoning behind this change. I believe this is an important distinction because, 1. If we agree with "artifacts" then we can't stop with user stores, but have to eventually transition all Identity Admin UI configurations to this model. Though there is no technical limitations in this approach, it takes a lot of effort, and that effort better be worth it. 2. It changes the complete outlook of the product, and could differentiate us from competitors, in a good way or bad way. In my understanding I think there are three *independent* questions to answer. 1. File based persistence vs. Database persistence for these configurations 2. Providing file based representation for configurations 3. Definition of artifacts vs. configurations I think on the first point we agree that database persistence is a better fit for IS, if not for anything, just because all Identity Admin UI configurations except user stores are currently in database for IS and we don't see any major problem with it. I think this is what is being discussed in the mail thread [1], and I tend to agree on most of the points @KasunG Gajasinghe is mentioning. On the second point, we already have this in IS, although we currently only have it for SPs; we need to cover the rest as well. I guess we can agree on this need as well, as we've had many enterprise customers who have had strong requirements for import/export, automation, product upgrades, environment migration, etc. and asked for some file format to export to. Having it as a human readable text file makes only makes more sense. However, I don't know if we've figured out how to handle secrets in these files. On the third point, to me "artifacts" mean more than file persistence and file representation. If we go by the definition of, anything that is created by the application user at runtime is an artifact, then I will have to agree that SP, IdP, user stores, XACML and everything else is an artifact. However, in a practical sense, I see some differences between, APIs, mediation sequences, BPEL processes, Siddhi scripts, XACML policies, and service providers or identity providers or user stores. 1. APIs, mediation sequences, BPEL processes, Siddhi scripts, and XACML policies have development time aspect, hence involves development time governance. Where as service providers, identity providers and user stores don't really require that. 2. Artifacts generally have lifecycles, versioning, metadata and properties. Are we sure we want to make all Identity Admin UI configurations also fit into that definition? 3. Do we need change management and continuous delivery for Identity Admin UI configurations like service providers, identity providers and user stores? Have we had any customer request for this level development governance for IS? 4. CI/CD can be generally expected as a key requirement for organizations that aspire to have a good integration or API management story. But I don't think it is right to expect it from an IAM prospect. In most cases IAM is the first product they onboard even before EI or API Manager or other enterprise applications. 5. In 90% of the cases Identity Admin UI configurations are going to be done by one or couple of people in an organization, which is not the case with other artifacts mentioned above. And we don't expect changes to service providers, identity providers and user stores, as frequent as other artifacts. Will this not be considered an overhead for something that one or two people change very rarely? 6. Most Identity Admins are not big fans of developer tools. They prefer the old fashioned management UIs. If so does the whole development governance story have a big impact with developer tools? On Wed, Jun 5, 2019 at 9:34 AM Ruwan Abeykoon wrote: > Hi All, > The "User Store" configuration can be considered as a deployment artifact > if we look at the following aspects. > > a) "Secondary User Stores" are added, removed and updated per tenant > basis. (Same as SP, and IdP configs) > I don't think adding, removing or updating per tenant can be a qualification criteria for artifacts. There are some Resident IdP configurations which are updated per tenant and clearly doesn't qualify as artifacts. E.g. SAML2 SSO Entity ID of the tenant. My concern applies to SPs and IdPs as well. > b) "User Store", "IdP" and "SP", XACML policies, etc behaves as a > collection of business rules, defining the authentication and authorization > flows per the tenant. > XACML, adaptive authentication scripts and in future authentication flow designer output can definitely be treated as artifacts, which like you said define authentication/authorization flows
Re: [Architecture] [Dev][IAM] Moving File Based Artifacts to Artifact Store
Hi Johann, *>> Where is this requirement coming from? To have multiple storage locations for configurations?* I addition to what is said previously, Another key requirement coming from the concept of "Zero Migration upgrade", which we are trying to achieve as much as possible going forward. Customers already have previous File-Based artifacts should be able to continue as it is, after a product upgrade, as long as their deployment architecture does not change. Cheers, Ruwan A On Thu, Jul 4, 2019 at 9:03 AM Johann Nallathamby wrote: > Ignore the question Isura, I think Ruwan's reply contains the answer. > > Regards, > Johann. > > On Thu, Jul 4, 2019 at 8:48 AM Johann Nallathamby wrote: > >> Hi Isura, >> >> On Fri, Jun 7, 2019 at 9:16 AM Isura Karunaratne wrote: >> >>> >>> >>> On Wed, Jun 5, 2019 at 9:34 AM Ruwan Abeykoon wrote: >>> Hi All, The "User Store" configuration can be considered as a deployment artifact if we look at the following aspects. a) "Secondary User Stores" are added, removed and updated per tenant basis. (Same as SP, and IdP configs) b) "User Store", "IdP" and "SP", XACML policies, etc behaves as a collection of business rules, defining the authentication and authorization flows per the tenant. c) A change in a particular "User Store" usually affect the "Authentication" decisions done via SP config. Hence they have tight coupling. d) All "User Store", "SP", "IdP", etc, need to be taken as one unit, when we consider environment to environment promotion of these configs. (Dev->QA->staging->Prod) Hence IMHO, treating "User Store" as the file-based artifact was a right decision, when our products have been designed for deployment on bear-metal or VM. However moving forward to container, and cloud nativeness, posses the challenge on sharing these artifacts. Also, considering the CI/CD pipelines, the governance aspect of changing the configurations, etc, these type of configs need to be considered as artifacts. We might need to version control these artifacts in future and may need to push and pull them from VCS systems. What we need to do is to have a delegation pattern implemented for all current file based (and registry based artifacts), where we can switch the repository from file based one to different system. DB based repository would be the first such(simple) implementation. We may need to implement Git based repository when we properly support cloud use cases for example. We can abstract the storage system, and retain all the parsing and generation logic unchanged for artifacts. it would be a minimal change and most versatile way to extend IMO. We would need to implement property or "environment variable" binding logic, to get proper support for environment to environment promotion of artifacts. yet, it can be done with a separate effort than this IMO. Hence +1 to treat - Persist data as a blob (marshalled to text form) - In a separate table structure. Cheers, Ruwan A On Tue, Jun 4, 2019 at 3:50 PM Johann Nallathamby wrote: > +1 to get rid of the artifacts for user stores. I think this was a > wrong decision we made early on. > > On Tue, Jun 4, 2019 at 1:19 PM Hasanthi Purnima Dissanayake < > hasan...@wso2.com> wrote: > >> Hi All, >> >> *Problem * >> Currently, some artifacts like userstores , tenants' data, etc are >> stored in the file system (not in the database). So when using a >> clustered >> setup those artifacts should be shared among all the nodes by using one >> of >> the following file sharing mechanisms. >> >>- Dep Sync >>- rSync >>- Shared File System >> >> >> *Solution * >> In order to avoid a shared file system and to reduce the deployment >> and maintenance overhead, those artifacts ca be persisted in the database >> itself. >> >> *Approach* >> After discussing with @Ruwan Abeykoon and @Isura >> Karunaratne we have two options to persist above >> discussed artifact details. >> >>- In the configuration store which is already implemented as >>discussed in [1][2]. >>- In a separate table structure. >> >> If we are to go with option 01, then we need to consider the >> artifacts as configurations and persist in the existing schema. >> > The advantage of using this is we can re-use the existing >> implementation including the database schema and existing rest APIs and >> functionalities (pagination, searching, etc) . >> > > >>> Hi all, >>> >>> The drawback is the conceptual difference between an artifact and >> configuration. >> > > I think this is not a problem. In fact I believe we
Re: [Architecture] [Dev][IAM] Moving File Based Artifacts to Artifact Store
Ignore the question Isura, I think Ruwan's reply contains the answer. Regards, Johann. On Thu, Jul 4, 2019 at 8:48 AM Johann Nallathamby wrote: > Hi Isura, > > On Fri, Jun 7, 2019 at 9:16 AM Isura Karunaratne wrote: > >> >> >> On Wed, Jun 5, 2019 at 9:34 AM Ruwan Abeykoon wrote: >> >>> Hi All, >>> The "User Store" configuration can be considered as a deployment >>> artifact if we look at the following aspects. >>> >>> a) "Secondary User Stores" are added, removed and updated per tenant >>> basis. (Same as SP, and IdP configs) >>> b) "User Store", "IdP" and "SP", XACML policies, etc behaves as a >>> collection of business rules, defining the authentication and authorization >>> flows per the tenant. >>> c) A change in a particular "User Store" usually affect the >>> "Authentication" decisions done via SP config. Hence they have tight >>> coupling. >>> d) All "User Store", "SP", "IdP", etc, need to be taken as one unit, >>> when we consider environment to environment promotion of these configs. >>> (Dev->QA->staging->Prod) >>> >>> Hence IMHO, treating "User Store" as the file-based artifact was a right >>> decision, when our products have been designed for deployment on bear-metal >>> or VM. However moving forward to container, and cloud nativeness, posses >>> the challenge on sharing these artifacts. >>> >>> Also, considering the CI/CD pipelines, the governance aspect of changing >>> the configurations, etc, these type of configs need to be considered as >>> artifacts. We might need to version control these artifacts in future and >>> may need to push and pull them from VCS systems. >>> >>> What we need to do is to have a delegation pattern implemented for all >>> current file based (and registry based artifacts), where we can switch the >>> repository from file based one to different system. DB based repository >>> would be the first such(simple) implementation. We may need to implement >>> Git based repository when we properly support cloud use cases for example. >>> We can abstract the storage system, and retain all the parsing and >>> generation logic unchanged for artifacts. it would be a minimal change and >>> most versatile way to extend IMO. >>> >>> We would need to implement property or "environment variable" binding >>> logic, to get proper support for environment to environment promotion of >>> artifacts. yet, it can be done with a separate effort than this IMO. >>> >>> Hence +1 to treat >>> >>>- Persist data as a blob (marshalled to text form) >>>- In a separate table structure. >>> >>> Cheers, >>> Ruwan A >>> >>> >>> On Tue, Jun 4, 2019 at 3:50 PM Johann Nallathamby >>> wrote: >>> +1 to get rid of the artifacts for user stores. I think this was a wrong decision we made early on. On Tue, Jun 4, 2019 at 1:19 PM Hasanthi Purnima Dissanayake < hasan...@wso2.com> wrote: > Hi All, > > *Problem * > Currently, some artifacts like userstores , tenants' data, etc are > stored in the file system (not in the database). So when using a clustered > setup those artifacts should be shared among all the nodes by using one of > the following file sharing mechanisms. > >- Dep Sync >- rSync >- Shared File System > > > *Solution * > In order to avoid a shared file system and to reduce the deployment > and maintenance overhead, those artifacts ca be persisted in the database > itself. > > *Approach* > After discussing with @Ruwan Abeykoon and @Isura > Karunaratne we have two options to persist above > discussed artifact details. > >- In the configuration store which is already implemented as >discussed in [1][2]. >- In a separate table structure. > > If we are to go with option 01, then we need to consider the artifacts > as configurations and persist in the existing schema. > The advantage of using this is we can re-use the existing > implementation including the database schema and existing rest APIs and > functionalities (pagination, searching, etc) . > >>> >> Hi all, >> >> >>> The drawback is the conceptual difference between an artifact and > configuration. > I think this is not a problem. In fact I believe we made a wrong decision of considering user stores as artifacts. I can't remember exactly as to why we decided like that. User stores are not development artifacts; they are one time configurations. They don't have metadata, versioning, lifecycles or any other properties associated with other artifacts in WSO2. > Further if we are to use the configuration store there is no way to > include specific input validations for the user store configuration > feature. > I am not sure I completely understand this. But please check the SAML2 metadata for identity provider feature where I believe we've done some
Re: [Architecture] [Dev][IAM] Moving File Based Artifacts to Artifact Store
Hi Isura, On Fri, Jun 7, 2019 at 9:16 AM Isura Karunaratne wrote: > > > On Wed, Jun 5, 2019 at 9:34 AM Ruwan Abeykoon wrote: > >> Hi All, >> The "User Store" configuration can be considered as a deployment artifact >> if we look at the following aspects. >> >> a) "Secondary User Stores" are added, removed and updated per tenant >> basis. (Same as SP, and IdP configs) >> b) "User Store", "IdP" and "SP", XACML policies, etc behaves as a >> collection of business rules, defining the authentication and authorization >> flows per the tenant. >> c) A change in a particular "User Store" usually affect the >> "Authentication" decisions done via SP config. Hence they have tight >> coupling. >> d) All "User Store", "SP", "IdP", etc, need to be taken as one unit, when >> we consider environment to environment promotion of these configs. >> (Dev->QA->staging->Prod) >> >> Hence IMHO, treating "User Store" as the file-based artifact was a right >> decision, when our products have been designed for deployment on bear-metal >> or VM. However moving forward to container, and cloud nativeness, posses >> the challenge on sharing these artifacts. >> >> Also, considering the CI/CD pipelines, the governance aspect of changing >> the configurations, etc, these type of configs need to be considered as >> artifacts. We might need to version control these artifacts in future and >> may need to push and pull them from VCS systems. >> >> What we need to do is to have a delegation pattern implemented for all >> current file based (and registry based artifacts), where we can switch the >> repository from file based one to different system. DB based repository >> would be the first such(simple) implementation. We may need to implement >> Git based repository when we properly support cloud use cases for example. >> We can abstract the storage system, and retain all the parsing and >> generation logic unchanged for artifacts. it would be a minimal change and >> most versatile way to extend IMO. >> >> We would need to implement property or "environment variable" binding >> logic, to get proper support for environment to environment promotion of >> artifacts. yet, it can be done with a separate effort than this IMO. >> >> Hence +1 to treat >> >>- Persist data as a blob (marshalled to text form) >>- In a separate table structure. >> >> Cheers, >> Ruwan A >> >> >> On Tue, Jun 4, 2019 at 3:50 PM Johann Nallathamby >> wrote: >> >>> +1 to get rid of the artifacts for user stores. I think this was a wrong >>> decision we made early on. >>> >>> On Tue, Jun 4, 2019 at 1:19 PM Hasanthi Purnima Dissanayake < >>> hasan...@wso2.com> wrote: >>> Hi All, *Problem * Currently, some artifacts like userstores , tenants' data, etc are stored in the file system (not in the database). So when using a clustered setup those artifacts should be shared among all the nodes by using one of the following file sharing mechanisms. - Dep Sync - rSync - Shared File System *Solution * In order to avoid a shared file system and to reduce the deployment and maintenance overhead, those artifacts ca be persisted in the database itself. *Approach* After discussing with @Ruwan Abeykoon and @Isura Karunaratne we have two options to persist above discussed artifact details. - In the configuration store which is already implemented as discussed in [1][2]. - In a separate table structure. If we are to go with option 01, then we need to consider the artifacts as configurations and persist in the existing schema. >>> The advantage of using this is we can re-use the existing implementation including the database schema and existing rest APIs and functionalities (pagination, searching, etc) . >>> >>> >> > Hi all, > > >> The drawback is the conceptual difference between an artifact and configuration. >>> >>> I think this is not a problem. In fact I believe we made a wrong >>> decision of considering user stores as artifacts. I can't remember exactly >>> as to why we decided like that. User stores are not development artifacts; >>> they are one time configurations. They don't have metadata, versioning, >>> lifecycles or any other properties associated with other artifacts in WSO2. >>> >>> Further if we are to use the configuration store there is no way to include specific input validations for the user store configuration feature. >>> >>> I am not sure I completely understand this. But please check the SAML2 >>> metadata for identity provider feature where I believe we've done some >>> validations on the properties. I think they were done through interceptors >>> in identity provider service. >>> >> Yes. If we have an interception layer, we can do the validation in that > level. Currently, we don't have an intercepting layer for configuration > management, we
Re: [Architecture] [Dev][IAM] Moving File Based Artifacts to Artifact Store
On Wed, Jun 5, 2019 at 9:34 AM Ruwan Abeykoon wrote: > Hi All, > The "User Store" configuration can be considered as a deployment artifact > if we look at the following aspects. > > a) "Secondary User Stores" are added, removed and updated per tenant > basis. (Same as SP, and IdP configs) > b) "User Store", "IdP" and "SP", XACML policies, etc behaves as a > collection of business rules, defining the authentication and authorization > flows per the tenant. > c) A change in a particular "User Store" usually affect the > "Authentication" decisions done via SP config. Hence they have tight > coupling. > d) All "User Store", "SP", "IdP", etc, need to be taken as one unit, when > we consider environment to environment promotion of these configs. > (Dev->QA->staging->Prod) > > Hence IMHO, treating "User Store" as the file-based artifact was a right > decision, when our products have been designed for deployment on bear-metal > or VM. However moving forward to container, and cloud nativeness, posses > the challenge on sharing these artifacts. > > Also, considering the CI/CD pipelines, the governance aspect of changing > the configurations, etc, these type of configs need to be considered as > artifacts. We might need to version control these artifacts in future and > may need to push and pull them from VCS systems. > > What we need to do is to have a delegation pattern implemented for all > current file based (and registry based artifacts), where we can switch the > repository from file based one to different system. DB based repository > would be the first such(simple) implementation. We may need to implement > Git based repository when we properly support cloud use cases for example. > We can abstract the storage system, and retain all the parsing and > generation logic unchanged for artifacts. it would be a minimal change and > most versatile way to extend IMO. > > We would need to implement property or "environment variable" binding > logic, to get proper support for environment to environment promotion of > artifacts. yet, it can be done with a separate effort than this IMO. > > Hence +1 to treat > >- Persist data as a blob (marshalled to text form) >- In a separate table structure. > > Cheers, > Ruwan A > > > On Tue, Jun 4, 2019 at 3:50 PM Johann Nallathamby wrote: > >> +1 to get rid of the artifacts for user stores. I think this was a wrong >> decision we made early on. >> >> On Tue, Jun 4, 2019 at 1:19 PM Hasanthi Purnima Dissanayake < >> hasan...@wso2.com> wrote: >> >>> Hi All, >>> >>> *Problem * >>> Currently, some artifacts like userstores , tenants' data, etc are >>> stored in the file system (not in the database). So when using a clustered >>> setup those artifacts should be shared among all the nodes by using one of >>> the following file sharing mechanisms. >>> >>>- Dep Sync >>>- rSync >>>- Shared File System >>> >>> >>> *Solution * >>> In order to avoid a shared file system and to reduce the deployment and >>> maintenance overhead, those artifacts ca be persisted in the database >>> itself. >>> >>> *Approach* >>> After discussing with @Ruwan Abeykoon and @Isura >>> Karunaratne we have two options to persist above >>> discussed artifact details. >>> >>>- In the configuration store which is already implemented as >>>discussed in [1][2]. >>>- In a separate table structure. >>> >>> If we are to go with option 01, then we need to consider the artifacts >>> as configurations and persist in the existing schema. >>> >> The advantage of using this is we can re-use the existing implementation >>> including the database schema and existing rest APIs and functionalities >>> (pagination, searching, etc) . >>> >> >> > Hi all, > The drawback is the conceptual difference between an artifact and >>> configuration. >>> >> >> I think this is not a problem. In fact I believe we made a wrong decision >> of considering user stores as artifacts. I can't remember exactly as to why >> we decided like that. User stores are not development artifacts; they are >> one time configurations. They don't have metadata, versioning, lifecycles >> or any other properties associated with other artifacts in WSO2. >> >> >>> Further if we are to use the configuration store there is no way to >>> include specific input validations for the user store configuration feature. >>> >> >> I am not sure I completely understand this. But please check the SAML2 >> metadata for identity provider feature where I believe we've done some >> validations on the properties. I think they were done through interceptors >> in identity provider service. >> > Yes. If we have an interception layer, we can do the validation in that level. Currently, we don't have an intercepting layer for configuration management, we can treat this as an improvement to the config management feature. >>> If we are to go with the option 02, then the flow will be as follows. >>> >>> *Existing Flow* >>> >>> [image: Untitled Diagram (9).png]
Re: [Architecture] [Dev][IAM] Moving File Based Artifacts to Artifact Store
Hi All, The "User Store" configuration can be considered as a deployment artifact if we look at the following aspects. a) "Secondary User Stores" are added, removed and updated per tenant basis. (Same as SP, and IdP configs) b) "User Store", "IdP" and "SP", XACML policies, etc behaves as a collection of business rules, defining the authentication and authorization flows per the tenant. c) A change in a particular "User Store" usually affect the "Authentication" decisions done via SP config. Hence they have tight coupling. d) All "User Store", "SP", "IdP", etc, need to be taken as one unit, when we consider environment to environment promotion of these configs. (Dev->QA->staging->Prod) Hence IMHO, treating "User Store" as the file-based artifact was a right decision, when our products have been designed for deployment on bear-metal or VM. However moving forward to container, and cloud nativeness, posses the challenge on sharing these artifacts. Also, considering the CI/CD pipelines, the governance aspect of changing the configurations, etc, these type of configs need to be considered as artifacts. We might need to version control these artifacts in future and may need to push and pull them from VCS systems. What we need to do is to have a delegation pattern implemented for all current file based (and registry based artifacts), where we can switch the repository from file based one to different system. DB based repository would be the first such(simple) implementation. We may need to implement Git based repository when we properly support cloud use cases for example. We can abstract the storage system, and retain all the parsing and generation logic unchanged for artifacts. it would be a minimal change and most versatile way to extend IMO. We would need to implement property or "environment variable" binding logic, to get proper support for environment to environment promotion of artifacts. yet, it can be done with a separate effort than this IMO. Hence +1 to treat - Persist data as a blob (marshalled to text form) - In a separate table structure. Cheers, Ruwan A On Tue, Jun 4, 2019 at 3:50 PM Johann Nallathamby wrote: > +1 to get rid of the artifacts for user stores. I think this was a wrong > decision we made early on. > > On Tue, Jun 4, 2019 at 1:19 PM Hasanthi Purnima Dissanayake < > hasan...@wso2.com> wrote: > >> Hi All, >> >> *Problem * >> Currently, some artifacts like userstores , tenants' data, etc are stored >> in the file system (not in the database). So when using a clustered setup >> those artifacts should be shared among all the nodes by using one of the >> following file sharing mechanisms. >> >>- Dep Sync >>- rSync >>- Shared File System >> >> >> *Solution * >> In order to avoid a shared file system and to reduce the deployment and >> maintenance overhead, those artifacts ca be persisted in the database >> itself. >> >> *Approach* >> After discussing with @Ruwan Abeykoon and @Isura >> Karunaratne we have two options to persist above >> discussed artifact details. >> >>- In the configuration store which is already implemented as >>discussed in [1][2]. >>- In a separate table structure. >> >> If we are to go with option 01, then we need to consider the artifacts as >> configurations and persist in the existing schema. >> > The advantage of using this is we can re-use the existing implementation >> including the database schema and existing rest APIs and functionalities >> (pagination, searching, etc) . >> > > >> The drawback is the conceptual difference between an artifact and >> configuration. >> > > I think this is not a problem. In fact I believe we made a wrong decision > of considering user stores as artifacts. I can't remember exactly as to why > we decided like that. User stores are not development artifacts; they are > one time configurations. They don't have metadata, versioning, lifecycles > or any other properties associated with other artifacts in WSO2. > > >> Further if we are to use the configuration store there is no way to >> include specific input validations for the user store configuration feature. >> > > I am not sure I completely understand this. But please check the SAML2 > metadata for identity provider feature where I believe we've done some > validations on the properties. I think they were done through interceptors > in identity provider service. > > >> >> If we are to go with the option 02, then the flow will be as follows. >> >> *Existing Flow* >> >> [image: Untitled Diagram (9).png] >> >> >> >> >> >> >> *Suggested Flow* >> >> [image: Untitled Diagram (10).png] >> >> >> >>- With the suggested approach, as the storage mechanisms, file system >>and database can be used and any other storage mechanism is pluggable. >> >> I don't agree with classifying user stores as artifacts. Therefore I > guess for me this option doesn't qualify at all :). > > However, just to understand this option better, have we come across any
Re: [Architecture] [Dev][IAM] Moving File Based Artifacts to Artifact Store
+1 to get rid of the artifacts for user stores. I think this was a wrong decision we made early on. On Tue, Jun 4, 2019 at 1:19 PM Hasanthi Purnima Dissanayake < hasan...@wso2.com> wrote: > Hi All, > > *Problem * > Currently, some artifacts like userstores , tenants' data, etc are stored > in the file system (not in the database). So when using a clustered setup > those artifacts should be shared among all the nodes by using one of the > following file sharing mechanisms. > >- Dep Sync >- rSync >- Shared File System > > > *Solution * > In order to avoid a shared file system and to reduce the deployment and > maintenance overhead, those artifacts ca be persisted in the database > itself. > > *Approach* > After discussing with @Ruwan Abeykoon and @Isura > Karunaratne we have two options to persist above > discussed artifact details. > >- In the configuration store which is already implemented as discussed >in [1][2]. >- In a separate table structure. > > If we are to go with option 01, then we need to consider the artifacts as > configurations and persist in the existing schema. > The advantage of using this is we can re-use the existing implementation > including the database schema and existing rest APIs and functionalities > (pagination, searching, etc) . > > The drawback is the conceptual difference between an artifact and > configuration. > I think this is not a problem. In fact I believe we made a wrong decision of considering user stores as artifacts. I can't remember exactly as to why we decided like that. User stores are not development artifacts; they are one time configurations. They don't have metadata, versioning, lifecycles or any other properties associated with other artifacts in WSO2. > Further if we are to use the configuration store there is no way to > include specific input validations for the user store configuration feature. > I am not sure I completely understand this. But please check the SAML2 metadata for identity provider feature where I believe we've done some validations on the properties. I think they were done through interceptors in identity provider service. > > If we are to go with the option 02, then the flow will be as follows. > > *Existing Flow* > > [image: Untitled Diagram (9).png] > > > > > > > *Suggested Flow* > > [image: Untitled Diagram (10).png] > > > >- With the suggested approach, as the storage mechanisms, file system >and database can be used and any other storage mechanism is pluggable. > > I don't agree with classifying user stores as artifacts. Therefore I guess for me this option doesn't qualify at all :). However, just to understand this option better, have we come across any customer who has asked us to support multiple types of storage mechanisms for artifacts and/or configurations? Where is this requirement coming from? I've only seen such requirements for user stores. For configurations I feel this is just over engineering. May be it is a valid requirement for artifacts? Even if we agree that there are valid reasons to support this then it has to be supported for all configurations and/or artifacts. > >- There should be a way to identify the repository where the data is >loaded from. The repository can be the file system, database or any other >storage mechanism. > > It sounds like this can get too complicated. > >- In both the read write operations the enduser should have the >control to decide the storage mechanism. > > Hmm.. this sounds more like a requirement to optimize database read write performance. Doesn't sound right for artifacts. > >- If the user needs to migrate a userstore from one storage mechanism >(file system) to another then they can do it via UI. > > Again too many options for the user can make the product fragile. > When persisting the data in the database there are two options we can use : > >- Persist data as a blob > > If we persist as blob then we lose the granular control over each property for validation, transformation, etc. > >- Persists data as key value pair > > +1 for this. > If we are to go with the option one then we can persist the file as a blob > and reuse most of the existing parsing logics. > Given the understanding I think I prefer option 1 with properties. Thanks & Regards, Johann. > > Highly appreciate your suggestions and feedbacks on the above approach. > > [1] [Architecture][IAM][JDBC based Configuration Store] Database Schema > [2] [Architecture] [IS] JDBC based Configuration Store for WSO2 IS > > Thanks, > Hasanthi > > -- > > Hasanthi Dissanayake | Senior Software Engineer | WSO2 Inc. > (m) +94718407133 | (w) +94112145345 | Email: hasan...@wso2.com > > -- *Johann Dilantha Nallathamby* | Associate Director/Solutions Architect | WSO2 Inc. (m) +94 (77) 7776950 | (w) +94 (11) 2145345 | (e) joh...@wso2.com [image: Signature.jpg] ___ Architecture mailing list
[Architecture] [Dev][IAM] Moving File Based Artifacts to Artifact Store
Hi All, *Problem * Currently, some artifacts like userstores , tenants' data, etc are stored in the file system (not in the database). So when using a clustered setup those artifacts should be shared among all the nodes by using one of the following file sharing mechanisms. - Dep Sync - rSync - Shared File System *Solution * In order to avoid a shared file system and to reduce the deployment and maintenance overhead, those artifacts ca be persisted in the database itself. *Approach* After discussing with @Ruwan Abeykoon and @Isura Karunaratne we have two options to persist above discussed artifact details. - In the configuration store which is already implemented as discussed in [1][2]. - In a separate table structure. If we are to go with option 01, then we need to consider the artifacts as configurations and persist in the existing schema. The advantage of using this is we can re-use the existing implementation including the database schema and existing rest APIs and functionalities (pagination, searching, etc) . The drawback is the conceptual difference between an artifact and configuration. Further if we are to use the configuration store there is no way to include specific input validations for the userstore configuration feature. If we are to go with the option 02, then the flow will be as follows. *Existing Flow* [image: Untitled Diagram (9).png] *Suggested Flow* [image: Untitled Diagram (10).png] - With the suggested approach, as the storage mechanisms, file system and database can be used and any other storage mechanism is pluggable. - There should be a way to identify the repository where the data is loaded from. The repository can be the file system, database or any other storage mechanism. - In both the read write operations the enduser should have the control to decide the storage mechanism. - If the user needs to migrate a userstore from one storage mechanism (file system) to another then they can do it via UI. When persisting the data in the database there are two options we can use : - Persist data as a blob - Persists data as key value pair If we are to go with the option one then we can persist the file as a blob and reuse most of the existing parsing logics. Highly appreciate your suggestions and feedbacks on the above approach. [1] [Architecture][IAM][JDBC based Configuration Store] Database Schema [2] [Architecture] [IS] JDBC based Configuration Store for WSO2 IS Thanks, Hasanthi -- Hasanthi Dissanayake | Senior Software Engineer | WSO2 Inc. (m) +94718407133 | (w) +94112145345 | Email: hasan...@wso2.com ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
