Re: [Architecture] [App Factory] Per Developer Repos
Hi all, If the app owner choose the repository type as github while creating an application, we need to have the prerequisite [1] IMO. [1] All the users of this organization need to have a GitHub account. Moreover when the repository is created under the organization context, we will have to create two teams for each repository at the same time as follows: 1. team_appowners_appKey : Has to be created with 'push pull and administrative' permission. Needs to add the repository created earlier. 2. team_developers_appKey : Has to be created with 'pull only' permission. Needs to add the repository created earlier. So at the time of team selection for the application we will be able to add them to the earlier created teams accordingly. WDYT? On Mon, Jan 6, 2014 at 11:56 AM, Janaka Ranabahu jan...@wso2.com wrote: Hi Manisha, On Mon, Jan 6, 2014 at 11:51 AM, Manisha Gayathri mani...@wso2.comwrote: After the offline discussion with Dimuthu, we decided to go ahead with the per developer repo for now. Per developer build, deploy and test will be considered later. So what you are suggesting is that the developer is not allowed to build the project? IMO, all of develop, build, deploy and test by a developer comes under one user story. We should at least have a user story for build and deployment for a developer, if we are only going to work with repo part now. WDYT? Thanks, Janaka Thanks Manisha On Mon, Jan 6, 2014 at 11:45 AM, Manisha Gayathri mani...@wso2.comwrote: Thanks for bringing this up Janaka. Was going to address this separately. What I thought was, the ideal scenario would be, the developer should be able to build, deploy and test in their own isolated environment. But this incorporates a lot of work and changes. In that case, the user story will be like once the developer does the Pull Request he waits to test until the Pull Merge Accept Notification comes from the app owner. This is inconvenient from the developer's perspective. Dimuthu, how can we address this? Thanks Manisha On Mon, Jan 6, 2014 at 11:32 AM, Janaka Ranabahu jan...@wso2.comwrote: Hi Manisha, On Mon, Jan 6, 2014 at 11:02 AM, Manisha Gayathri mani...@wso2.comwrote: Hi all, According to the diagram in Sanjiva's mail, I have come up with a sequence diagram of the scenarios that would cover in this user story. Please refer [1] For M11, we are hoping to implement the scenarios up to Add Developer user story. [1]. http://www.websequencediagrams.com/?lz=dGl0bGUgUGVyIERldmVsb3BlciBSZXBvIEltcGwgd2l0aCBHaXRIdWIKCm9wdCBDcmVhdGUgTmV3IFRlbmFudAogAAIHQWRtaW4gLT4AKAc6ACMIYSBHaXQgb3JnYW5pemF0aW9uACESQUYAJgluAD8bQUY6IFByb3ZpZGUgZwBGDyBjb250ZXh0ICYgY3JlZGVudGlhbHMAfCJuIGFwcCBvd25lciB0ZWFtAIFtBnB1c2ggcGVybWlzcwCBHRVHaXRodWI6YwCBXAhkZXYAOQZpZiBub3QgZXhpc3RpbmcgYW5kIGFkZCB1c2VyIHRvACAJCmVuZACCQhFBcHAKICBBcHBPAIELBQCCCBIAGgdGAIJVE25ldyByZXBvIGluIHRoZSBvcmcuAIINCAogAIM-BwCCaAUAWAc6IFJlc3BvbnNlAINhBgCDcQVkZXRhaQCCMwUAXwZKZW5raW5zOiBBZGQAgxcFYnVpbGQgdGFzayBmb3IAgjgFcmVwbwCBRQpBZGQAhEEKAIE_DwCEWgk6IGludml0AIM0BnRvIGpvaW4gcHJvamVjdAogAIUACwCEFgdTdXJlIACBaRFGb3JrAIFnBW1haW4AgQEGAIFdDABjC0hlcmUncyB5b3VyIGZvcmtlZAAnCACBQSYALQdkZXYAgV8PAIYrCldyaXRlcyBDb2RlAIEyEACGCghHaXQgUHVsbACBUhAAggwLACsUAIIrC0xvY2FsIElERQCDAgYASx5zaACDaw0Ag0IJQgA0BwCDVQcAgxIHIENsb3VkOiBEZXBsb3kgZACIBQkAVRUAgz8LAIguCnRlc3QvZGVidWcAhVQKUHVsbCBSZXF1ZXMAg0UVABsFcgAXCQCFVQYAhRkKABMMIHJlY2VpdmVkAIYUEACFRgt2aQCGAgUARgkAhkALAIkQCE1lcmdlAIR_Gk5vdGlmeSBhY2NlcHQAhyEFs=vs2010 The given diagram only contains up to committing the changes. How are we going to handle build and deployment. Can developers have their own build and own isolated deployment with their own repo? Or are they only allowed to test once they merge with the central repo. Thanks, Janaka Thanks Manisha On Sun, Jan 5, 2014 at 3:07 AM, Dimuthu Leelarathne dimut...@wso2.com wrote: Hi Manisha and all, Please see my comments inline. On Friday, January 3, 2014, Manisha Gayathri wrote: Hi all, For the M11 of App Factory, we are implementing per developer repo feature to give a github like experience for the developers. The user story for this federated development will be as follows: - Once an app-owner creates an application, the application will be getting a repo in gitblit - When a developer logs in, he gets an option to fork the main repo of the app Fork option should not be something done by developer when he logs in. Here is the user story we want from Sanjiva's mail titled per-developer git repos for App Factory. Here is the link from the mail [1]. The diagram here is very precise. Basically the fork request should be sent by AF when a developer is added to the project. And it should be done by AF on a listener when add user to application event is fired, so the developer do not do it. And as per the mail states GitBlit does not support PR(pull request) git workflow. Then thinking along this line, IMO we should NOT
Re: [Architecture] [App Factory] Per Developer Repos
On Wed, Jan 8, 2014 at 1:54 PM, Punnadi Gunarathna punn...@wso2.com wrote: Hi all, If the app owner choose the repository type as github while creating an application, we need to have the prerequisite [1] IMO. [1] All the users of this organization need to have a GitHub account. Yes. This is an issue. And we need to provide the proper name in the way it appears in the github. What happens in Github is a suggestion list appears when we are typing the member name. Moreover when the repository is created under the organization context, we will have to create two teams for each repository at the same time as follows: 1. team_appowners_appKey : Has to be created with 'push pull and administrative' permission. Needs to add the repository created earlier. 2. team_developers_appKey : Has to be created with 'pull only' permission. Needs to add the repository created earlier. To add a team, it is not mandatory to add a repo AFAIK. Adding a repo is optional. The team creation should happen in tenant creation time. At that point no repo is available. Repos are available only after the app is created. At that point, 1. Tenant admin registers the tenant in github 2. Create the team appowners and developers under that organization 3. Create the app. Only at this time, the repo is created. At the time of the repo creation, we can assign the repo to the team with the team ID. (using the API call PUT /teams/:id/repos/:org/:repo) So at the time of team selection for the application we will be able to add them to the earlier created teams accordingly. WDYT? On Mon, Jan 6, 2014 at 11:56 AM, Janaka Ranabahu jan...@wso2.com wrote: Hi Manisha, On Mon, Jan 6, 2014 at 11:51 AM, Manisha Gayathri mani...@wso2.comwrote: After the offline discussion with Dimuthu, we decided to go ahead with the per developer repo for now. Per developer build, deploy and test will be considered later. So what you are suggesting is that the developer is not allowed to build the project? IMO, all of develop, build, deploy and test by a developer comes under one user story. We should at least have a user story for build and deployment for a developer, if we are only going to work with repo part now. WDYT? Thanks, Janaka Thanks Manisha On Mon, Jan 6, 2014 at 11:45 AM, Manisha Gayathri mani...@wso2.comwrote: Thanks for bringing this up Janaka. Was going to address this separately. What I thought was, the ideal scenario would be, the developer should be able to build, deploy and test in their own isolated environment. But this incorporates a lot of work and changes. In that case, the user story will be like once the developer does the Pull Request he waits to test until the Pull Merge Accept Notification comes from the app owner. This is inconvenient from the developer's perspective. Dimuthu, how can we address this? Thanks Manisha On Mon, Jan 6, 2014 at 11:32 AM, Janaka Ranabahu jan...@wso2.comwrote: Hi Manisha, On Mon, Jan 6, 2014 at 11:02 AM, Manisha Gayathri mani...@wso2.comwrote: Hi all, According to the diagram in Sanjiva's mail, I have come up with a sequence diagram of the scenarios that would cover in this user story. Please refer [1] For M11, we are hoping to implement the scenarios up to Add Developer user story. [1]. http://www.websequencediagrams.com/?lz=dGl0bGUgUGVyIERldmVsb3BlciBSZXBvIEltcGwgd2l0aCBHaXRIdWIKCm9wdCBDcmVhdGUgTmV3IFRlbmFudAogAAIHQWRtaW4gLT4AKAc6ACMIYSBHaXQgb3JnYW5pemF0aW9uACESQUYAJgluAD8bQUY6IFByb3ZpZGUgZwBGDyBjb250ZXh0ICYgY3JlZGVudGlhbHMAfCJuIGFwcCBvd25lciB0ZWFtAIFtBnB1c2ggcGVybWlzcwCBHRVHaXRodWI6YwCBXAhkZXYAOQZpZiBub3QgZXhpc3RpbmcgYW5kIGFkZCB1c2VyIHRvACAJCmVuZACCQhFBcHAKICBBcHBPAIELBQCCCBIAGgdGAIJVE25ldyByZXBvIGluIHRoZSBvcmcuAIINCAogAIM-BwCCaAUAWAc6IFJlc3BvbnNlAINhBgCDcQVkZXRhaQCCMwUAXwZKZW5raW5zOiBBZGQAgxcFYnVpbGQgdGFzayBmb3IAgjgFcmVwbwCBRQpBZGQAhEEKAIE_DwCEWgk6IGludml0AIM0BnRvIGpvaW4gcHJvamVjdAogAIUACwCEFgdTdXJlIACBaRFGb3JrAIFnBW1haW4AgQEGAIFdDABjC0hlcmUncyB5b3VyIGZvcmtlZAAnCACBQSYALQdkZXYAgV8PAIYrCldyaXRlcyBDb2RlAIEyEACGCghHaXQgUHVsbACBUhAAggwLACsUAIIrC0xvY2FsIElERQCDAgYASx5zaACDaw0Ag0IJQgA0BwCDVQcAgxIHIENsb3VkOiBEZXBsb3kgZACIBQkAVRUAgz8LAIguCnRlc3QvZGVidWcAhVQKUHVsbCBSZXF1ZXMAg0UVABsFcgAXCQCFVQYAhRkKABMMIHJlY2VpdmVkAIYUEACFRgt2aQCGAgUARgkAhkALAIkQCE1lcmdlAIR_Gk5vdGlmeSBhY2NlcHQAhyEFs=vs2010 The given diagram only contains up to committing the changes. How are we going to handle build and deployment. Can developers have their own build and own isolated deployment with their own repo? Or are they only allowed to test once they merge with the central repo. Thanks, Janaka Thanks Manisha On Sun, Jan 5, 2014 at 3:07 AM, Dimuthu Leelarathne dimut...@wso2.com wrote: Hi Manisha and all, Please see my comments inline. On Friday, January 3, 2014, Manisha Gayathri wrote: Hi all, For the M11 of App Factory, we are implementing per developer repo feature to give a github like experience for the developers. The
[Architecture] Improving performance of Balana
Hi All, As It is discussed; There are two easy ways that could improve the performance of Balana. 1. Parallel evaluation of XACML policies. In Balana, 1st, It would be check whether each policies is valid according to the target element And then it starts the evaluating of matched policies.. It is great, if we can do this parallel Or also there may some other points to improve this. 2. Use thread local variables to store attribute values as suggested by Stefano [1]. Related jira [2] Any volunteer to work on this? [1] http://stackoverflow.com/questions/20401808/wso2-identity-server-improve-the-performance-of-an-attributefindermodule-for-at/ [2] https://wso2.org/jira/browse/COMMONS-104 Thanks, Asela. -- Thanks Regards, Asela ATL Mobile : +94 777 625 933 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] Using caching in HumanTask's PeopleQueryEvaluator for performance improvement.
Great improvements hasitha. This should help us get consistent task creation time. Lets add the cache invalidation config time to human task configuration file under the people query evaluator section. 30 seconds should be fine as we are making it configurable. Regards Nandika On Thu, Jan 9, 2014 at 7:46 AM, Hasitha Aravinda hasi...@wso2.com wrote: Hi all, In humantask engine we are facing following problems, - Every task operations requires users/roles validation. If the task has expression based user assignment or excluded owners, this validation becomes expensive since it requires additional user store calls. ( Many of them are redundant calls) - Many operations are sequential operations. (eg. loadTask - claim - start , loadTask- complete). This causes to invoke redundant user store calls. - If the user store has many users/role these calls will take some time to complete and eventually task engine becomes slow. So the main idea is, to avoid those redundant user store calls by caching task's users/roles related data at the humantask engine side. I have completed the caching implementation for PeopleQueryEvaluator and results of following operation will be cached at humantask engine. - Is Existing User - (User name, true/false) - Is Existing Role - (Role name, true/false) - Role name List for User. (User name, role list) - User name List for Role. (Role name, user list) I conducted a task creation test in order to measure the performance improvement with caching. For this test, I used a LDAP which has 20 groups and each group has 1000 of users. The Humantask has expression based user assignment, including 4 potential owner groups, 2 excluded owner groups and 3 business administrator groups. I ran a soapUI load test to create 1000 tasks using 10 concurrency with 0 test delay. Following are the result. *min (ms)* *max (ms)* *avg (ms)* *last (ms)* *total count* *tps* *bytes* *bps* *err* *rat* *Without caching* 898 3653 1698.29 1077 1000 5.87 354000 2080 0 0 *With Caching* 43 2467 184.16 127 1000 53.72 354000 19017 0 0 According to the results, there is 9X performance improvement with caching. Now I am working on making cache expiry time configurable via humantask configuration. I think default 30s is ok for most of the scenarios. *User Name List for Role* operation can return a large result set. So my question is, is it ok to cache such a large result sets ? Suggestions and thoughts are welcome. Thanks, Hasitha. -- Hasitha Aravinda, Software Engineer, WSO2 Inc. Email: hasi...@wso2.com Mobile: +94 71 8 210 200 -- Nandika Jayawardana Senior Technical Lead WSO2 Inc ; http://wso2.com lean.enterprise.middleware ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] carbon native linux platform packages
Did my previous email make any sense? Is the recommendation to install separate servers because of JVM isolation, or is there another reason? On Wed, Jan 8, 2014 at 11:25 AM, chris snow chsnow...@gmail.com wrote: What is the driver for recommending to install separate servers for each product? In production environments, having separate JVMs is more robust, but this is too heavy weight for development environments. Therefore, I'm thinking we should allow an administrator to deploy multiple features to a single carbon instance if required. If JVM isolation is required by the administrator, then a LXC container could be created for each server instance that requires its own JVM. I think this approach is similar to that taken by the Ubuntu tomcat package. An administrator would install one tomcat instance (apt-get install tomcat). Multiple wars can be deployed to the single tomcat instance, for example in a development environment. Prod environments could use lxc to install multiple tomcats to give each war its own jvm if required. Does this make sense? On 8 Jan 2014 06:36, Geeth Munasinghe ge...@wso2.com wrote: +1 Interesting idea. I think installing features on carbon kernel will not be flexible thing here. Instead we should package a single server to install. Because if some one wants to install both ESB and App Server as separate servers on the same machine, then person has to install carbon kernal twice and install set of features on them. Thanks Geeth *G. K. S. Munasinghe* *Software Engineer,* *WSO2, Inc. http://wso2.com http://wso2.com/ * *lean.enterprise.middleware.* email: ge...@wso2.com phone:(+94) 777911226 On Wed, Jan 8, 2014 at 11:47 AM, Nirmal Fernando nir...@wso2.com wrote: Hi Chris, The idea is very interesting indeed. Thanks for bringing it up. If you have a look at p2-profile's pom of a product, you would get an idea about the set of features that are required by that product. On Wed, Jan 8, 2014 at 1:49 AM, chris snow chsnow...@gmail.com wrote: Hi Lasantha, I'm hoping that it would be possible to automate installing carbon core and also automate installing features. My hope is that features would be able to coexist on the same carbon core platform without interfering with each other. It should be up to the administrator to decide which features to install together, for example to install AS and BPS should be as simple as: $ apt-get install wso2-as wso2-bps The distribution's standard locations should be used, e.g. /etc/wso2/as/... for configuration files, and /var/log for log files. Many thanks, Chris On 7 Jan 2014 09:08, Lasantha Fernando lasan...@wso2.com wrote: Hi Chris, Looks interesting. In your approach, are you planning to automate installing features on top of a carbon-kernel to get the relevant features for a product? Also, how would it work if installing multiple WSO2 products, or different product features on top of a single carbon kernel? Thanks, Lasantha On 7 January 2014 02:12, chris snow chsnow...@gmail.com wrote: Today I was thinking of more examples to help explain my previous email. Here is one example from the eclipse world: Package: eclipse-jdt (3.8.0~rc4-1) - http://packages.debian.org/wheezy/eclipse-jdt dep: default-jre Standard Java or Java compatible Runtime or java5-runtime virtual package provided by default-jre, gcj-4.6-jre, gcj-4.7-jre, gcj-jre, openjdk-6-jre, openjdk-7-jre or java6-runtime virtual package provided by default-jre, openjdk-6-jre, openjdk-7-jre dep: eclipse-platform (= 3.8.0~rc4-1) Eclipse platform without development plug-ins .. Therefore: $ apt-get install eclipse-jdt # eclipse-jdt 3.8.0~rc4-1 also installs eclipse-platform = 3.8.0~rc4-1 With chunk releases for Carbon 4.2+ being backward compatible, could the same principle be applied: $ apt-get install wso2-as # wso2-as 5.0 also installs wso2-core-platform = 4.20 On Sun, Jan 5, 2014 at 3:46 PM, chris snow chsnow...@gmail.comwrote: Has anyone ever looked at creating native linux platform installers for WSO2 products? For example: - DEB for debian based distros - RPM for redhat based distros I've been thinking of how the tomcat package works on ubuntu where the latest major version (e.g. 7) completely replaces the previous version. However, from what I understand, wso2 features require a certain major + minor version + minimum chunk version of Carbon so there would need to be a packaged version of Carbon for each supported Carbon release, for example: - wso2-carbon-core-42.deb - wso2-carbon-core-50.deb - wso2-carbon-core-51.deb - etc However, products with a different major + minor version wso2-carbon-core-42, wso2-carbon-core-50, and wso2-carbon-core-51 could co-exist as they would be considered different packages (though ports would need to be selected as to avoid clashing). With the approach, you could perform the following to
Re: [Architecture] [App Factory] Per Developer Repos
The GithubRepositoryProvider that is already implemented in AF provide API to get the Github username of the AF user and store as claim.Once the user is invited to a project contributer/developer permission will be added to that user provider Github for the repo using Github API. BTW are we going to implement this feature for Github first and then for Gitblit? On Thu, Jan 9, 2014 at 11:54 AM, Manisha Gayathri mani...@wso2.com wrote: Hi Shiro, Github does not provide APIs to create users in github. Instead it can add existing github users to existing github organizations. Therefore creating github user ids for AF users from AF side is not possible. Thanks Manisha On Thu, Jan 9, 2014 at 11:44 AM, Shiroshica Kulatilake sh...@wso2.com wrote: Hi, In this case aren't we using Github as the tool to manage repos in AF - so shouldn't AF deal with creating user ids in GitHub ? This is instead of asking for github id's from team members ? Thank you, Shiro On Thu, Jan 9, 2014 at 11:22 AM, Punnadi Gunarathna punn...@wso2.com wrote: Hi All, While inviting developers to the newly created application, those developers should get added to the pull only permission team in GitHub for it's corresponding repository. In order to do that we will have to pass the correct GitHub user ids of those developers. So there is a problem which is we need to have the developer's GitHub user id in AF front. Any thoughts? On Wed, Jan 8, 2014 at 2:49 PM, Punnadi Gunarathna punn...@wso2.com wrote: On Wed, Jan 8, 2014 at 2:11 PM, Manisha Gayathri mani...@wso2.com wrote: On Wed, Jan 8, 2014 at 2:08 PM, Manisha Gayathri mani...@wso2.com wrote: On Wed, Jan 8, 2014 at 1:54 PM, Punnadi Gunarathna punn...@wso2.com wrote: Hi all, If the app owner choose the repository type as github while creating an application, we need to have the prerequisite [1] IMO. [1] All the users of this organization need to have a GitHub account. Yes. This is an issue. And we need to provide the proper name in the way it appears in the github. What happens in Github is a suggestion list appears when we are typing the member name. Moreover when the repository is created under the organization context, we will have to create two teams for each repository at the same time as follows: 1. team_appowners_appKey : Has to be created with 'push pull and administrative' permission. Needs to add the repository created earlier. 2. team_developers_appKey : Has to be created with 'pull only' permission. Needs to add the repository created earlier. To add a team, it is not mandatory to add a repo AFAIK. Adding a repo is optional. Yes it is not mandatory to add a repo to a team. Also I think there has to be 2 teams for each repo not just one team for the whole organization. What we are trying to achieve here is that to give different permission levels for two teams for the created repo. So I believe it is needed to add the corresponding repo to address that matter. Please correct me if i am wrong. The team creation should happen in tenant creation time. At that point no repo is available. Repos are available only after the app is created. At that point, 1. Tenant admin registers the tenant in github 2. Create the team appowners and developers under that organization 3. Create the app. Only at this time, the repo is created. CORRECTION 4. After the repo is created, when allocating users to the app, we acan assign the repo to the team with the team ID. (using the API call PUT /teams/:id/repos/:org/:repo) At the time of the repo creation, we can assign the repo to the team with the team ID. (using the API call PUT /teams/:id/repos/:org/:repo) So at the time of team selection for the application we will be able to add them to the earlier created teams accordingly. WDYT? On Mon, Jan 6, 2014 at 11:56 AM, Janaka Ranabahu jan...@wso2.com wrote: Hi Manisha, On Mon, Jan 6, 2014 at 11:51 AM, Manisha Gayathri mani...@wso2.com wrote: After the offline discussion with Dimuthu, we decided to go ahead with the per developer repo for now. Per developer build, deploy and test will be considered later. So what you are suggesting is that the developer is not allowed to build the project? IMO, all of develop, build, deploy and test by a developer comes under one user story. We should at least have a user story for build and deployment for a developer, if we are only going to work with repo part now. WDYT? Thanks, Janaka Thanks Manisha On Mon, Jan 6, 2014 at 11:45 AM, Manisha Gayathri mani...@wso2.com wrote: Thanks for bringing this up Janaka. Was going to address this separately. What I thought was, the ideal scenario would be, the developer should be able to build, deploy and test in their own isolated environment. But this incorporates a lot of work and changes. In that case, the user story will be like once the developer does the Pull
