Re: [Dev] [PC]Unable to change states in lifecycle

[Thilini Cooray]

Hi,

On Mon, Aug 1, 2016 at 11:19 AM, Yasima Dewmini  wrote:

> Hi Thilini,
>
> Only thing I'm doing is changing the permission of the user roles to deny
> the write permission.
> When trying to change state following error comes.
>
>

As I mentioned above, the lifecycle executor which gets fired in your
scenario tries to update a resource. Since the "user1" doesn't have the
write access to the resource "/_system/governance/processes/process1/1.0.0.
", you are unable to update.

Please check on the executor why this resource needs to be updated on the
lifecycle change.
As per now, user1 needs to have write access.



> [2016-08-01 11:17:01,378] ERROR {rxt.asset} -  Failed to invoke action:
> Submit for the asset of id: "02cbe9d0-5482-4d68-b507-c76b4ede6310".The
> following exception was thrown: JavaException:
> org.wso2.carbon.registry.core.secure.AuthorizationFailedException: Resource
> update failed. User user1 is not authorized to update the resource
> /_system/governance/processes/process1/1.0.0.
> [2016-08-01 11:17:01,379] ERROR
> {JAGGERY.extensions.assets.default.apis.asset:jag} -
> org.wso2.carbon.registry.core.secure.AuthorizationFailedException: Resource
> update failed. User user1 is not authorized to update the resource
> /_system/governance/processes/process1/1.0.0.
>
> Regards,
> Yasima.
>


Thanks.



>
>
> On Mon, Aug 1, 2016 at 11:06 AM, Thilini Cooray  wrote:
>
>> Hi,
>>
>> Unless the executor itself doesn't try to update any resources, the
>> process owner does not require write access.
>> Do you observe any server side errors mentioning user not authorized to
>> access the resource etc.?
>>
>> Thanks.
>>
>> On Mon, Aug 1, 2016 at 10:57 AM, Yasima Dewmini  wrote:
>>
>>> Hi,
>>>
>>> I'm creating a new life cycle for PC. I want to remove write permission
>>> of the process owner when the process is in In-Review state. So, I wrote an
>>> executor to be executed at Development state to remove that permission.
>>> When write permission is removed, the process owner is not allowed to
>>> change the states of life cycle. But the process owner has life cycle
>>> permission.
>>>
>>> Does any user need write permission to change life cycle states? Isn't
>>> life cycle permission enough to change states?
>>>
>>> Any clarification on this issue is appreciated.
>>>
>>> Regards,
>>> Yasima.
>>>
>>> --
>>> http://wso2.com/signatureYasima Dewmini
>>> Software Engineer, WSO2, Inc.
>>> Email: yas...@wso2.com
>>> Mobile: +94713117081
>>>
>>> ___
>>> Dev mailing list
>>> Dev@wso2.org
>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>
>>>
>>
>>
>> --
>> Best Regards,
>>
>> *Thilini Cooray*
>> Software Engineer
>> Mobile : +94 (0) 774 570 112 <%2B94%20%280%29%20773%20451194>
>> E-mail : thili...@wso2.com
>>
>> WSO2 Inc. www.wso2.com
>> lean.enterprise.middleware
>>
>
>
>
> --
> http://wso2.com/signatureYasima Dewmini
> Software Engineer, WSO2, Inc.
> Email: yas...@wso2.com
> Mobile: +94713117081
>



-- 
Best Regards,

*Thilini Cooray*
Software Engineer
Mobile : +94 (0) 774 570 112 <%2B94%20%280%29%20773%20451194>
E-mail : thili...@wso2.com

WSO2 Inc. www.wso2.com
lean.enterprise.middleware
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] [PC]Unable to change states in lifecycle

[Yasima Dewmini]

Hi Thilini,

Only thing I'm doing is changing the permission of the user roles to deny
the write permission.
When trying to change state following error comes.

[2016-08-01 11:17:01,378] ERROR {rxt.asset} -  Failed to invoke action:
Submit for the asset of id: "02cbe9d0-5482-4d68-b507-c76b4ede6310".The
following exception was thrown: JavaException:
org.wso2.carbon.registry.core.secure.AuthorizationFailedException: Resource
update failed. User user1 is not authorized to update the resource
/_system/governance/processes/process1/1.0.0.
[2016-08-01 11:17:01,379] ERROR
{JAGGERY.extensions.assets.default.apis.asset:jag} -
org.wso2.carbon.registry.core.secure.AuthorizationFailedException: Resource
update failed. User user1 is not authorized to update the resource
/_system/governance/processes/process1/1.0.0.

Regards,
Yasima.


On Mon, Aug 1, 2016 at 11:06 AM, Thilini Cooray  wrote:

> Hi,
>
> Unless the executor itself doesn't try to update any resources, the
> process owner does not require write access.
> Do you observe any server side errors mentioning user not authorized to
> access the resource etc.?
>
> Thanks.
>
> On Mon, Aug 1, 2016 at 10:57 AM, Yasima Dewmini  wrote:
>
>> Hi,
>>
>> I'm creating a new life cycle for PC. I want to remove write permission
>> of the process owner when the process is in In-Review state. So, I wrote an
>> executor to be executed at Development state to remove that permission.
>> When write permission is removed, the process owner is not allowed to
>> change the states of life cycle. But the process owner has life cycle
>> permission.
>>
>> Does any user need write permission to change life cycle states? Isn't
>> life cycle permission enough to change states?
>>
>> Any clarification on this issue is appreciated.
>>
>> Regards,
>> Yasima.
>>
>> --
>> http://wso2.com/signatureYasima Dewmini
>> Software Engineer, WSO2, Inc.
>> Email: yas...@wso2.com
>> Mobile: +94713117081
>>
>> ___
>> Dev mailing list
>> Dev@wso2.org
>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>
>>
>
>
> --
> Best Regards,
>
> *Thilini Cooray*
> Software Engineer
> Mobile : +94 (0) 774 570 112 <%2B94%20%280%29%20773%20451194>
> E-mail : thili...@wso2.com
>
> WSO2 Inc. www.wso2.com
> lean.enterprise.middleware
>



-- 
http://wso2.com/signatureYasima Dewmini
Software Engineer, WSO2, Inc.
Email: yas...@wso2.com
Mobile: +94713117081
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

[Dev] [Architecture] WSO2 Application Server 6.0.0-M3 Released !

[Chanaka Cooray]

WSO2 Application Server 6.0.0-M3 Released !
Welcome to WSO2 Application Server, the successor of WSO2 Carbon based
Application Server. WSO2 Application Server 6.0.0 is a complete revamp and
is based on vanilla Apache Tomcat. WSO2 provides a number of features by
means of extensions to Tomcat to add/enhance the functionality. It provides
first class support for generic web applications, JAX-RS/JAX-WS web
applications and jaggery applications. The performance of the server and
individual application can be monitored by integrating WSO2 Application
Server with WSO2 Data Analytics Server.

WSO2 Application Server is an open source project and it is available under
the Apache Software License (v2.0)
.

Source and product distribution for this release is available here
.
Key Features

   - HTTP Statistics Monitoring
   - Webapp Classloading Runtimes
   - Jaggery Runtime Support
   - Accessing Applications using Single Sign-On

Fixed Issues

   - https://wso2.org/jira/browse/WSAS-2194?filter=13173

Known Issues

   - https://wso2.org/jira/browse/WSAS-2248?filter=13174

Reporting Issues

Issues, documentation errors and feature requests regarding
WSO2 Application Server can be reported through the public issue tracking
system. https://wso2.org/jira/browse/WSAS.
Contact usWSO2 Application Server developers can be contacted via the
Development and Architecture mailing lists.
Alternatively, questions can also be raised in the stackoverflow forum :
http://stackoverflow.com/questions/tagged/wso2

Thank you for your interest in WSO2 Application Server.

*-The WSO2 Application Server Development Team -*

-- 
Chanaka Cooray
Software Engineer, WSO2 Inc. http://wso2.com
Email: chana...@wso2.com
Mobile: +94713149860

___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] [PC]Unable to change states in lifecycle

[Thilini Cooray]

Hi,

Unless the executor itself doesn't try to update any resources, the process
owner does not require write access.
Do you observe any server side errors mentioning user not authorized to
access the resource etc.?

Thanks.

On Mon, Aug 1, 2016 at 10:57 AM, Yasima Dewmini  wrote:

> Hi,
>
> I'm creating a new life cycle for PC. I want to remove write permission of
> the process owner when the process is in In-Review state. So, I wrote an
> executor to be executed at Development state to remove that permission.
> When write permission is removed, the process owner is not allowed to
> change the states of life cycle. But the process owner has life cycle
> permission.
>
> Does any user need write permission to change life cycle states? Isn't
> life cycle permission enough to change states?
>
> Any clarification on this issue is appreciated.
>
> Regards,
> Yasima.
>
> --
> http://wso2.com/signatureYasima Dewmini
> Software Engineer, WSO2, Inc.
> Email: yas...@wso2.com
> Mobile: +94713117081
>
> ___
> Dev mailing list
> Dev@wso2.org
> http://wso2.org/cgi-bin/mailman/listinfo/dev
>
>


-- 
Best Regards,

*Thilini Cooray*
Software Engineer
Mobile : +94 (0) 774 570 112 <%2B94%20%280%29%20773%20451194>
E-mail : thili...@wso2.com

WSO2 Inc. www.wso2.com
lean.enterprise.middleware
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

[Dev] [PC]Unable to change states in lifecycle

[Yasima Dewmini]

Hi,

I'm creating a new life cycle for PC. I want to remove write permission of
the process owner when the process is in In-Review state. So, I wrote an
executor to be executed at Development state to remove that permission.
When write permission is removed, the process owner is not allowed to
change the states of life cycle. But the process owner has life cycle
permission.

Does any user need write permission to change life cycle states? Isn't life
cycle permission enough to change states?

Any clarification on this issue is appreciated.

Regards,
Yasima.

-- 
http://wso2.com/signatureYasima Dewmini
Software Engineer, WSO2, Inc.
Email: yas...@wso2.com
Mobile: +94713117081
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] Fwd: GSOC2016: [ML][CEP] [SAMOA]Predictive analytic with online data for WSO2 Machine Learner-Samoa Integration

[Mahesh Dananjaya]

Hi Maheshakya,
I have a small problem. As we did all the time when i build a samoa
streaming extension for the cep by providing samoa dependencies, i put that
jar into cep_home/repository/component/lib/ folder.The start cep and try to
develop the execution plan by providing my extension. i am remotely
debugging the samoa streaming extension that connected to our samoa
predictive learning topology. While on the run i get this exception.
java.lang.ClassNotFoundException:
org.apache.samoa.topology.ComponentFactory cannot be found by
streaming_1.0_SNAPSHOT_1.0.0.

But without cep my samoa core is running fine. It seems to be samoa is not
recognizing in the cep side. Is this kind of a dependency problem. In my
extension side i think nothing wrong. When i replace the streamProcessor
extension with locally build cep event stream emulation java class all the
things run fine. The probelm occurs only when this is running with cep. Do
i need to add anything on the cep side to underestand samoa in cep side.can
you please help me with this.thank you.
regards,
Mahesh.


On Wed, Jul 27, 2016 at 3:59 PM, Mahesh Dananjaya  wrote:

> Hi Srinath,
>
> "I think this is OK. I assume this is much easier. Let's do this and check
> the performance",
> I also think so. I am currently on this and have a progress in this.for
> your question,
> "If we use MOA directly, would we loose the distributed support in SAOMA.
> Let's do a call when you can, so we can dsicuss this in detail."
>  I have to check for that.i think  if we are using MOA, we can use
> distributed clusters. As i wen through their documentation MOA itself
> cannot support distribution. But samoa can. What samoa does is providing
> streaming and clustering support by wrapping MOA algorithms. So i think we
> dont need to go for that option,directly MOA. Because now we can handle
> samoa building blocks.So we had 2 options for integrating it with cep
> without exploiting samoa architecture which is highly modular,scalable and
> flexible.
> 1. Develop Samoa topologies with basic samoa building blocks which make
> use of MOA algorithms.
> 2. Creating New streaming options with samoa stream building blocks which
> can feed cep siddhi events into samoa streams and get results samoa streams
> to cep back.
>
> As 2nd option is easy and  take reasonable time i am currently developing
> some modules to integrate cep streams into samoa which can be easily
> further extended to 1 option as well. So i had to modify stream and
> entrance modules for that and i think is has good progress.
>  So now i can feed my custom input stream to samoa topologies. That means
> i can easily integrate cep event stream into samoa instance stream.
> currently i am verifying the streaming clustering algorithms and its
> results with my custom input input streams which can be connected to samoa
> instance streams. As i have already developed siddhi extension for
> streaming, i can use them to feed my custom input streams now.  As the
> initial step i am go with the streaming clustering algorithms. Those are in
> my GSOC github repo [1].
> clustering - Streaming Clustering Support with samoa and CEP
> streaming - Streaming extension for samoa for cep evet streams
>
> i am currenlty working on the verification of results with some of our
> custom streams and then we will just have to integrate it with my
> extensions, which are already developed for cep as my first part.thank you.
> regards,
> Mahesh.
>
> [1]
> https://github.com/dananjayamahesh/GSOC2016/tree/master/gsoc/samoa/streaming/src/main/java/org/gsoc/samoa/streaming
>
> On Fri, Jul 22, 2016 at 3:17 PM, Srinath Perera  wrote:
>
>> Hi Mahesh,
>>
>> On Thu, Jul 21, 2016 at 2:10 PM, Mahesh Dananjaya <
>> dananjayamah...@gmail.com> wrote:
>>
>>> Hi All,
>>> I am onto connecting cep streams with samoa streams to data analysis
>>> using samoa framework. To connect samoa with cep siddhi event streams what
>>> i we can do is that try to convert cep streams into samoa streams or else
>>> writing wrpper for samoa for cep  streasm to be used. In both cases i have
>>> to covert siddhi cep streasm into samoa streams. Samoa is using MOA to
>>> analyse data. Moo contains ML framework to analyse stream data. Samoa is
>>> wrapping MOA withsome of its classes.
>>>
>>> Samoa streams is based on MOA, Instance and InstanceStreams. Samoa see
>>> streams as a stream of instances [1]. So if we are going to convert cep
>>> events into samoa instances , it will take time. But if we have some
>>> similarity between cep siddhi streams and samoa streasm we can reduce the
>>> time.
>>> 1. What is the underlying infrastructure for cep siddhi streasm.?
>>> 2. Are there anything as Instances or InstanceStreams kind of
>>> implmentation underlying cep streams?
>>> 3. How can i get more underestanding on CEP siddhi streams.
>>>
>>> On the other hand i can use my cep siddhi extension and put those events
>>> into event queue and 

[Dev] [DEV][C5] Cannot build product-mb with UUF dependency

[Sajini De Silva]

Hi,

org.wso2.carbon.andes.rest.service.feature.group 3.5.0.SNAPSHOT
requires package org.wso2.msf4j [2.0.0,3.0.0). We also have another
dependency for org.wso2.carbon.uufW. org.wso2.carbon.uuf uses
org.wso2.msf4j 2.0.0 version only in 1.0.0-SNAPSHOT version.

We have a dependency for org.wso2.uuf.handlebars.feature also. But we UUF
1.0.0-SNAPSHOT release org.wso2.uuf.handlebars.feature is not released.
Therefore we cannot build our product.

This is a blocker for us right now. Can you please look in to this?

Thanks,

-- 
Sajini De SIlva
Senior Software Engineer; WSO2 Inc.; http://wso2.com ,
Email: saj...@wso2.com
Blog: http://sajinid.blogspot.com/
Git hub profile: https://github.com/sajinidesilva

Phone: +94 712797729
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] SnakeYAML is already an OSGi bundle. Why do we have SnakeYAML in orbit?

[Shan Mahanama]

Hi Isuru,

Sorry about the delay to reply. We are looking into this issue and will fix
this in the next version (5.2.0).

Thanks,
Shan.

On Wed, Jul 27, 2016 at 12:30 PM, Isuru Perera  wrote:

> Hi Shan,
>
> Any updates on this?
>
> On Thu, Jun 30, 2016 at 9:58 AM, Shan Mahanama  wrote:
>
>> Hi Kishanthan,
>>
>> I will check it and update you as soon as possible.
>>
>> Thanks,
>> Shan.
>>
>> On Wed, Jun 29, 2016 at 6:20 PM, Kishanthan Thangarajah <
>> kishant...@wso2.com> wrote:
>>
>>> Thanks Isuru.
>>>
>>> @Shan, let's check on this and try to fix it.
>>>
>>> On Wed, Jun 29, 2016 at 6:12 PM, Isuru Perera  wrote:
>>>
 Hi Kishanthan,

 I created a JIRA: https://wso2.org/jira/browse/CARBON-15972

 I just saw that stackoverflow question when searching for some other
 issue.

 Thanks!

 Best Regards,
 It would also be great if you can create a jira for this with the
 information so that we could add this for next kernel release.

 On Wed, Jun 29, 2016 at 4:51 PM, Kishanthan Thangarajah <
 kishant...@wso2.com> wrote:

> Thanks Isuru. We can try and add this option, which would remove the
> need for Dynamic-Import.
>
> On Wed, Jun 29, 2016 at 3:42 PM, Isuru Perera  wrote:
>
>> I just saw that SnakeYaml provides a "CustomClassLoaderConstructor".
>> Could you please check on that.
>>
>> See:
>> http://stackoverflow.com/questions/4940379/using-snakeyaml-under-osgi/4944346#4944346
>>
>>
>> On Thu, May 19, 2016 at 1:31 PM, Isuru Perera 
>> wrote:
>>
>>> Thanks for info.
>>>
>>> Shall we ask the snakeyaml developers to add the Dynamic-Import
>>> header. May be we can create an issue [1]. Otherwise, that library is 
>>> not
>>> useful in an OSGi container, right?
>>>
>>> [1]
>>> https://bitbucket.org/asomov/snakeyaml/issues?status=new=open
>>>
>>> On Thu, May 5, 2016 at 7:25 AM, Kishanthan Thangarajah <
>>> kishant...@wso2.com> wrote:
>>>
 This is because we had to add Dynamic-Import header to this bundle,
 which is missing in the original bundle, as snakeyaml was loading 
 classes
 dynamically when trying to build and construct the bean object for the
 configuration.

 On Wed, May 4, 2016 at 12:16 PM, Isuru Perera 
 wrote:

> Hi,
>
> $subject [1]. See manifest entries in latest JARs [2]
>
> [1]
> https://github.com/wso2/orbit/tree/master/snakeyaml/1.16.0.wso2v1
> [2]
> http://search.maven.org/#search|gav|1|g%3A%22org.yaml%22%20AND%20a%3A%22snakeyaml%22
> 
>
> --
> Isuru Perera
> Associate Technical Lead | WSO2, Inc. | http://wso2.com/
> Lean . Enterprise . Middleware
>
> about.me/chrishantha
> Contact: +IsuruPereraWSO2
> 
>



 --
 *Kishanthan Thangarajah*
 Associate Technical Lead,
 Platform Technologies Team,
 WSO2, Inc.
 lean.enterprise.middleware

 Mobile - +94773426635
 Blog - *http://kishanthan.wordpress.com
 *
 Twitter - *http://twitter.com/kishanthan
 *

>>>
>>>
>>>
>>> --
>>> Isuru Perera
>>> Associate Technical Lead | WSO2, Inc. | http://wso2.com/
>>> Lean . Enterprise . Middleware
>>>
>>> about.me/chrishantha
>>> Contact: +IsuruPereraWSO2
>>> 
>>>
>>
>>
>>
>> --
>> Isuru Perera
>> Associate Technical Lead | WSO2, Inc. | http://wso2.com/
>> Lean . Enterprise . Middleware
>>
>> about.me/chrishantha
>> Contact: +IsuruPereraWSO2
>> 
>>
>
>
>
> --
> *Kishanthan Thangarajah*
> Technical Lead,
> Platform Technologies Team,
> WSO2, Inc.
> lean.enterprise.middleware
>
> Mobile - +94773426635
> Blog - *http://kishanthan.wordpress.com
> *
> Twitter - *http://twitter.com/kishanthan
> *
>



 --
 *Kishanthan Thangarajah*
 Technical Lead,
 Platform Technologies Team,
 WSO2, Inc.
 lean.enterprise.middleware

 Mobile - +94773426635
 Blog - *http://kishanthan.wordpress.com
 *
 Twitter - *http://twitter.com/kishanthan
 *

>>>
>>>
>>>
>>> --
>>> *Kishanthan Thangarajah*

Re: [Dev] GSoC2016-Contact Development Team Feature

[Kavith Lokuhewage]

Hi Nathiesha,

How is the progress on converting project to a maven project? Please let us
know if you face any issues.

BTW, [1] is a good article on osgi services with eclipse plugins. We can
use them for further decoupling publishers and report generators.

Thanks,

[1] http://www.vogella.com/tutorials/OSGiServices/article.html

On 29 Jul 2016 5:19 p.m., "Nathiesha Maddage" 
wrote:

> Hi Jasintha and Kavith,
>
> Here is the project plan for the upcoming 3/4 weeks.
>
>   *July 30th-1st Aug*
>
>
>-  Add and configure pom.xml
>-  Use Maven tycho plugin
>-  Build project using Maven
>
>
>
>   *2nd Aug-6th Aug*
>
>
>- Write a REST web service to publish errors, deploy in tomcat and
>testing.
>-  Add options for the current plugin project to use above web service
>-  Add current project classes and methods to be comply with above
>changes
>
>
>   *7th Aug-11th Aug*
>
>
>- Define an extension point of the plugin
>-  Add methods to get the registered plugins and their Project keys,
>to maintain a map
>- Add methods to group errors under their required plugins
>-  Change the publisher methods to create multiple issues in Jira
>
>
>   *12th Aug -15th Aug*
>
>
>- Improve ErrorPublisher interfaces
>-  Write OSGI services for ErrorPublisher interfaces
>
>
>
>   *16th Aug – 17th Aug*
>
>
>- Create Feature Project using the plugin
>
>
> * 18th Aug – 20th Aug*
>
>
>-  Refine the code
>-  Fix bugs
>- Testing
>
>
>  *21st Aug – 22nd Aug*
>
>
>-  Documentation
>
>
> Thanks and regards
>
> Nathiesha
>
>
> On Fri, Jul 29, 2016 at 9:48 AM, Nathiesha Maddage <
> nathieshamadd...@gmail.com> wrote:
>
>> Hi all,
>>
>> Here are the minutes of the meeting I had yesterday (28/07/16) with
>> Jasintha and Kavith.
>>
>> During the meeting the current project progress was discussed and
>> suggestions were made to improve and finalize the development of the
>> project. Listed below are the key points discussed during the meeting.
>>
>> 1.   Instructions were given on how to add pom.xml, to change its
>> fields to match with the project , to use Maven tycho plugin and to build
>> the project using maven.
>>
>> 2.   Suggestions were made on creating and improving interfaces like
>> ErrorPublisher and ReportGenerator, and how to call the publish method of
>> all the registered publishers using an iterator in the main controller
>> class.
>>
>> 3.   We planned to add another option for the user, where he does
>> not have any gmail and jira credentials, but a remote server would take
>> care of publishing the error using a common general account. For that I was
>> asked to write a REST web service that would handle this procedure, and to
>> deploy it in tomcat for testing. Possibility of using micro services for
>> this was also discussed.
>>
>> 4.   We planned to add multiple project keys for the jira create
>> issue option where required, and to assign the project key based on the
>> plugin that is responsible for the error. For that purpose, it was
>> suggested to create an extension point of the project, so other plugins can
>> extend it and define their own project key. Using this all the plugins that
>> have extended the error reporter plugin and their respective project keys
>> can be fetched, and a map can be maintained, to be used during the error
>> reporting process.
>>
>> 5.   To expose the Publisher classes, OSGI services will be
>> introduced to the project.
>>
>> 6.   It was suggested to create a feature by adding this plugin.
>>
>>
>> Furthermore instructions were given on the documentation required for the
>> project, and time frame for documentation was suggested. And I was asked to
>> send a project plan for the upcoming 3 weeks.
>>
>>
>> Thanks and regards
>>
>> Nathiesha
>>
>>
>> On Tue, Jul 26, 2016 at 7:39 AM, Nathiesha Maddage <
>> nathieshamadd...@gmail.com> wrote:
>>
>>> Hi Kavith,
>>>
>>> I accepted the invitation.
>>>
>>> Thanks and regards
>>> Nathiesha
>>>
>>> On Mon, Jul 25, 2016 at 8:42 PM, Kavith Lokuhewage 
>>> wrote:
>>>
 Hi Nathiesha,

 I have sent the invitation for the meeting at 2pm-3pm on 28/07/2016 [1].

 Thanks,

 [1] https://goo.gl/PLEWQd


 *Kavith Lokuhewage*
 Senior Software Engineer
 WSO2 Inc. - http://wso2.com
 lean . enterprise . middleware
 Mobile - +94779145123
 Linkedin 
 Twitter 

 On Mon, Jul 25, 2016 at 8:13 PM, Nathiesha Maddage <
 nathieshamadd...@gmail.com> wrote:

> Hi Kavith,
>
> How about Thursday evening, for the meeting? I am free after 1 pm.
>
> Thank you for the examples. I was going to refactor the code by
> introducing the interfaces as you suggested during the mid evaluation. 

Re: [Dev] [GSoC Dockerfiles] Meeting minutes

[Abhishek Tiwari]

Hi All,

I have updated the instructions to build and run the test framework [1]. I
am also able to build kubernetes platform on Mac OSX.

[1]. https://github.com/abhishek0198/test-framework

Thanks
Abhishek

On Mon, Aug 1, 2016 at 6:15 AM, Abhishek Tiwari <
abhishek.tiwari0...@gmail.com> wrote:

> Hi Imesh,
>
> The project is currently structured around standard Golang project created
> by eclipse, you can add eclipse plugin, create a new project with existing
> sources and it should compile fine. I will add complete instructions on
> Github and update this thread.
>
> Chamila,
> The tests I am working on are custom tests which aims at verifying few
> endpoints using CURL. I tried using existing tests for ESB, and that didn't
> help either since setup steps of those tests requires a lot of efforts. If
> developers from specific projects can help out with configuration, then we
> can run existing full fledged tests.
>
> Thanks
> Abhishek
>
> On Thu, Jul 28, 2016 at 9:08 PM, Chamila De Alwis 
> wrote:
>
>>
>> On Thu, Jul 28, 2016 at 4:19 AM, Abhishek Tiwari <
>> abhishek.tiwari0...@gmail.com> wrote:
>>
>>> 2. I am currently working on adding the ability to run smoke tests from
>>> the framework. The idea is to have the capability of having product
>>> specific smoke tests, then teams with more product knowledge can add smoke
>>> tests. I will add smoke tests for ESB at this point.
>>>
>>
>> Would these be custom tests that are specific to WSO2 Dockerfiles context
>> or generic tests that aim to test the product code? It would be better if
>> we can reuse the generic tests, as new tests would not be attractive to the
>> developers.
>>
>>
>> Regards,
>> Chamila de Alwis
>> Committer and PMC Member - Apache Stratos
>> Senior Software Engineer | WSO2
>> Blog: https://medium.com/@chamilad
>>
>>
>>
>
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] [GSoC Dockerfiles] Meeting minutes

[Abhishek Tiwari]

Hi Imesh,

The project is currently structured around standard Golang project created
by eclipse, you can add eclipse plugin, create a new project with existing
sources and it should compile fine. I will add complete instructions on
Github and update this thread.

Chamila,
The tests I am working on are custom tests which aims at verifying few
endpoints using CURL. I tried using existing tests for ESB, and that didn't
help either since setup steps of those tests requires a lot of efforts. If
developers from specific projects can help out with configuration, then we
can run existing full fledged tests.

Thanks
Abhishek

On Thu, Jul 28, 2016 at 9:08 PM, Chamila De Alwis  wrote:

>
> On Thu, Jul 28, 2016 at 4:19 AM, Abhishek Tiwari <
> abhishek.tiwari0...@gmail.com> wrote:
>
>> 2. I am currently working on adding the ability to run smoke tests from
>> the framework. The idea is to have the capability of having product
>> specific smoke tests, then teams with more product knowledge can add smoke
>> tests. I will add smoke tests for ESB at this point.
>>
>
> Would these be custom tests that are specific to WSO2 Dockerfiles context
> or generic tests that aim to test the product code? It would be better if
> we can reuse the generic tests, as new tests would not be attractive to the
> developers.
>
>
> Regards,
> Chamila de Alwis
> Committer and PMC Member - Apache Stratos
> Senior Software Engineer | WSO2
> Blog: https://medium.com/@chamilad
>
>
>
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] [IS] EC2 Performance Analysis : Sudden TPS drop in User Add in 500 concurrency with 10million users

[Bhathiya Jayasekara]

Hi Ishara,

On Sun, Jul 31, 2016 at 1:04 PM, Ishara Karunarathna 
wrote:

>
> With 2s worm up and then 10s ramp up period
>

Earlier we had a ramp up period of 60s. Not sure that has any effect here.
Just mentioning.

Thanks,
Bhathiya



 Did you look at the behaviour of latency?

 Thanks

 Malith


 On Fri, Jul 29, 2016 at 2:57 PM, Indunil Upeksha Rathnayake <
 indu...@wso2.com> wrote:

> Hi,
>
> We are currently engaged into a performance analysis where we are
> analyzing performance for User Add, Update, Authentication operations. The
> testing has been carried out in a following environment with 500
> concurrency and users up to 10 million.
>
> *Environment :*
>
> m3.2xlarge ( 8 core, 30GB, SSD 2x80 GB) 3 instances.
> MySQL 5.7
> Ubuntu 14.04
> Openldap-2.4.31
> IS 5.1.0
>
> In order to optimize the MYSQL server, following server parameters
> have been tuned accordingly. We have referred MYSQL documentation [1] as
> well as have performed analysis using several MYSQL tuners in [2].
>
> (1) *max_connections : 1000* (The maximum permitted number of
> simultaneous client connections.)
>
> (2) *join_buffer_size : 259968* (The minimum size of the buffer that
> is used for plain index scans, range index scans, and joins that do not 
> use
> indexes and thus perform full table scans.)
>
> (3) *innodb_buffer_pool_size : 5207959552 <5207959552>* (size of the
> memory area where InnoDB caches table and index data)
>
> (4) *innodb_log_buffer_size : 16777216* (size of the buffer for
> transactions that have not been committed yet)
>
> (5) *innodb_buffer_pool_instances : 1* (The number of buffer pool
> instances. According to the mysql documentation[1], on systems with a 
> large
> amount of memory, we can improve concurrency by dividing the buffer pool
> into multiple buffer pool instances. But couldn't change since it's a read
> only variable)
>
> (6) *key_buffer_size : 38400* (size of the buffer used for index
> blocks)
>
> (7) *table_open_cache : 4000* (The number of open tables for all
> threads)
>
> (8) *sort_buffer_size : 400* (Each session that must perform a
> sort allocates a buffer of this size)
>
> (9) *read_buffer_size : 100* (Each thread that does a sequential
> scan for a table allocates a buffer of this size for each table it scans.
> If we do many sequential scans, we might want to increase this value)
>
> (10) *query_cache_type : 0 *
>
> (11) *query_cache_limit : 1048576* (Do not cache results that are
> larger than this number of bytes)
>
> (12) *query_cache_size : 1048576* (The amount of memory allocated for
> caching query results)
>
> (13) *thread_stack : 262144* (The stack size for each thread)
>
> (14) *net_buffer_length : 16384* (Each client thread is associated
> with a connection buffer and result buffer. Both begin with a size given 
> by
> net_buffer_length but are dynamically enlarged up to max_allowed_packet
> bytes as needed)
>
> (15) *max_allowed_packet : 4194304* (The maximum size of one packet
> or any generated/intermediate string)
>
> (16) *thread_cache_size : 30* (no of threads the server should cache
> for reuse)
>
>
>
> IS has been configured as follows to optimize the performance.
>
> (1) JVM Heap Settings (-Xms -Xmx) changed as follows:
>
> *Xms : 2g *
>
> *Xmx : 2g *
>
> (2) Removed following entry from
> /repository/conf/tomcat/catalina-server.xml to disable http 
> access
> logs.
>
>  directory="${carbon.home}/repository/logs" prefix="http_access_"
> suffix=".log" pattern="combined" />
>
> (3) Tuned following parameters in axis2client.xml file.
>
> 1000
>
> 3
>
> (4) Added following additional parameters to optimize database
> connection pool.
>
> 6
>
> 600
>
> 20
>
> (5) Tuning Tomcat parameters in
> /repository/conf/tomcat/catalina-server.xml.
>
> *acceptorThreadCount = 8 *
>
> *maxThreads="750" *
>
> *minSpareThreads="150" *
>
> *maxKeepAliveRequests="600" *
>
> *acceptCount="600"*
>
>
>
> JMeter has been configured as follows to optimize the performance.
>
> (1) JVM Heap Settings (-Xms -Xmx) changed as follows:
>
> *Xms : 1g *
>
> *Xmx : 1g *
>
>
> We were able to optimize the environment up to some level. But*
> currently the TPS is dropping from the initial TPS 1139.5/s to 198.1/s in
> around 610 user count.(User Add)*
>
> Appreciate your help on figuring out whether we need to do any
> modifications to the optimizations in MYSQL, IS 

Re: [Dev] GSOC 2016 - Project 21 : MongoDB Userstore Development

[Asantha Thilina]

Hi Tharindu,

yes sure i am comfortable with any time and look forward for that

Thanks,
Asantha

On Sun, Jul 31, 2016 at 12:15 PM, Tharindu Edirisinghe 
wrote:

> Hi Asantha,
>
> Good work so far. Since we are reaching the final stage of GSoC, let's
> have a hangout at the end of this week and plan the final steps to complete
> the project.
>
> I will go through the deliverables and provide feedback soon.
>
> Thanks,
> Tharindu
>
> On Mon, Jul 25, 2016 at 12:08 PM, Asantha Thilina <
> asanthathil...@gmail.com> wrote:
>
>> Hi Tharindu,
>>
>> I have cleaned up all the bugs and removed all unnecessary codes and
>> committed all the changes into my repository[1]
>>
>> [1] https://github.com/asanthamax/mongodbuserstore
>>
>> Thanks,
>> Asantha
>>
>> On Sat, Jul 23, 2016 at 8:26 AM, Asantha Thilina <
>> asanthathil...@gmail.com> wrote:
>>
>>> Hi Tharindu,
>>>
>>> here i added all the commitments to spread sheet[1] and also i updated
>>> my blog with about java document api[2] and also i added findbug report[3]
>>> and also i updated my blog with the code quality analysis process i have
>>> done[4]
>>>
>>> [1]
>>> https://docs.google.com/spreadsheets/d/1jwSR_qFV-LfOS7ZTZ8CpiFFTYPoudDQF7zQhvblivaE/edit?usp=sharing
>>> [2]
>>> http://asanthamax.blogspot.com/2016/07/mongodb-user-store-package-hierarchy.html
>>> [3]https://googledrive.com/host/0B__ZE1ru1jkXZFdES3BOaHVyZ3M/
>>> [4]
>>> http://asanthamax.blogspot.com/2016/07/findbugs-in-mongodbuserstoremanager-in.html
>>>
>>> Thanks,
>>> Asantha
>>>
>>> On Fri, Jul 22, 2016 at 8:45 AM, Asantha Thilina <
>>> asanthathil...@gmail.com> wrote:
>>>
 Hi Tharindu,

 I have created the API documentation and hosted in my google drive[1] i
 will be preparing code inspection report as well will share it as soon as
 well i also added github page also to the userstore script in here[2] as
 well
 [1] http://googledrive.com/host/0B__ZE1ru1jkXbDVoSDlveV9yNk0
 [2] https://asanthamax.github.io/mongodbuserstore/

 Thanks,
 Asantha

 On Tue, Jul 12, 2016 at 11:59 AM, Asantha Thilina <
 asanthathil...@gmail.com> wrote:

> Hi Tharindu,
> sure we will arrange a hangout on next week and sure i will refer the
> link and do the necessary inspect in my code for any security
> vulnerabilities and update you sooner
>
> Thanks,
> Asantha
>
> On Tue, Jul 12, 2016 at 9:56 AM, Tharindu Edirisinghe <
> tharin...@wso2.com> wrote:
>
>> Hi Asantha,
>>
>> Thanks for the update. I'm out of the country this week and won't be
>> available. Let's have a hangout next week and discuss the final steps of
>> the project.
>>
>> Along with find bugs, if you haven't enabled findsecuritybugs plugin,
>> you can refer [1] and check if any security weaknesses are there in the
>> code.
>>
>> [1]
>> http://tharindue.blogspot.in/2016/06/static-code-analysis-for-java-using.html
>>
>> Thanks,
>> TharinduE
>>
>> On Tue, Jul 12, 2016 at 5:32 PM, Asantha Thilina <
>> asanthathil...@gmail.com> wrote:
>>
>>> Hi Tharindu,
>>> I have clean up the code according to find bug plugin and also for
>>> rollback purposes since it only concern when modifying or adding new 
>>> user
>>> or role to user store i have manually added rollback option where if 
>>> error
>>> occured all the changes done to db in code segment will be discard from 
>>> db
>>> i updated the my repository with recent changes[1] now i preparing the
>>> documentation and test scripts.
>>>
>>> [1] https://github.com/asanthamax/mongodbuserstore
>>>
>>> Thanks,
>>> Asantha
>>>
>>> On Mon, Jul 4, 2016 at 2:54 AM, Asantha Thilina <
>>> asanthathil...@gmail.com> wrote:
>>>
 Hi Tharindu,

 thanks for the reply ,sure i will try out that and update the
 progress as soon as possible

 Thanks,
 Asantha

 On Mon, Jul 4, 2016 at 1:38 AM, Tharindu Edirisinghe <
 tharin...@wso2.com> wrote:

> Hi Asantha,
>
> While you are adding the roll back transaction capability, also
> look into testing a high load and try to generate database level
> exceptions. Once thing you can try out is writing a client for 
> performing
> userstore manager operations such as add user, add role, login, 
> logout etc.
> and play it with a high concurrency using Jmeter. Then we should be 
> able to
> get a better understanding of transaction management with respective 
> to
> MongoDB.
>
> Regards,
> TharinduE
>
> On Mon, Jul 4, 2016 at 10:57 AM, Asantha Thilina <
> asanthathil...@gmail.com> wrote:
>
>> Hi all,
>>
>> i will be trying for implement roll back transactions 

Re: [Dev] [IS] EC2 Performance Analysis : Sudden TPS drop in User Add in 500 concurrency with 10million users

[Ishara Karunarathna]

Hi Prabath,

Maduranga is running same test in the openstack once its completed will
compare all.

BR,
Ishara

On Sun, Jul 31, 2016 at 10:14 PM, Prabath Siriwardana 
wrote:

> Can you please compare the results you are getting now with the results we
> got a week before in the same setup...? I guess we could get  ~1200 tps
> with 500 concurrency for 1M users, without any drop in the tps...?
>
> Thanks & regards,
> -Prabath
>
> On Sun, Jul 31, 2016 at 12:34 AM, Ishara Karunarathna 
> wrote:
>
>>
>>
>> On Sun, Jul 31, 2016 at 12:59 PM, Malith Jayasinghe 
>> wrote:
>>
>>>
>>>
>>> On Sun, Jul 31, 2016 at 12:49 PM, Ishara Karunarathna 
>>> wrote:
>>>
 Hi Malith,

 On Sun, Jul 31, 2016 at 12:37 PM, Malith Jayasinghe 
 wrote:

> HI Indunil,
> Just a few question regarding this performance test you have done:
>
> What is the reason for selecting the concurrency = 500 here?
>
 This is the user expected concurrency level. Thats the reason we user
 this.

>
> Have you tested the behaviour for lower concurrency levels?
>
> *"currently the TPS is dropping from the initial TPS 1139.5/s to
> 198.1/s in around 610 user count.(User Add)" - *How did you
> notice/measure this drop in TPS? Did you analyze the jmeter results
> offline? After it drops, does it improve after some time or does it stay
> the same?
>
 We test this with the Jmeter summery report.
 with latest results if we start again few min (2min) we can get this
 max tps and come down to around 250tps

>>>
>>> Ok so it comes down to 250tps and stays there?  Are you running these
>>> tests without a warm-up period?
>>>
>> Nop.
>>
>> With 2s worm up and then 10s ramp up period
>>
>>>
>
> Did you look at the behaviour of latency?
>
> Thanks
>
> Malith
>
>
> On Fri, Jul 29, 2016 at 2:57 PM, Indunil Upeksha Rathnayake <
> indu...@wso2.com> wrote:
>
>> Hi,
>>
>> We are currently engaged into a performance analysis where we are
>> analyzing performance for User Add, Update, Authentication operations. 
>> The
>> testing has been carried out in a following environment with 500
>> concurrency and users up to 10 million.
>>
>> *Environment :*
>>
>> m3.2xlarge ( 8 core, 30GB, SSD 2x80 GB) 3 instances.
>> MySQL 5.7
>> Ubuntu 14.04
>> Openldap-2.4.31
>> IS 5.1.0
>>
>> In order to optimize the MYSQL server, following server parameters
>> have been tuned accordingly. We have referred MYSQL documentation [1] as
>> well as have performed analysis using several MYSQL tuners in [2].
>>
>> (1) *max_connections : 1000* (The maximum permitted number of
>> simultaneous client connections.)
>>
>> (2) *join_buffer_size : 259968* (The minimum size of the buffer that
>> is used for plain index scans, range index scans, and joins that do not 
>> use
>> indexes and thus perform full table scans.)
>>
>> (3) *innodb_buffer_pool_size : 5207959552 <5207959552>* (size of the
>> memory area where InnoDB caches table and index data)
>>
>> (4) *innodb_log_buffer_size : 16777216* (size of the buffer for
>> transactions that have not been committed yet)
>>
>> (5) *innodb_buffer_pool_instances : 1* (The number of buffer pool
>> instances. According to the mysql documentation[1], on systems with a 
>> large
>> amount of memory, we can improve concurrency by dividing the buffer pool
>> into multiple buffer pool instances. But couldn't change since it's a 
>> read
>> only variable)
>>
>> (6) *key_buffer_size : 38400* (size of the buffer used for index
>> blocks)
>>
>> (7) *table_open_cache : 4000* (The number of open tables for all
>> threads)
>>
>> (8) *sort_buffer_size : 400* (Each session that must perform a
>> sort allocates a buffer of this size)
>>
>> (9) *read_buffer_size : 100* (Each thread that does a sequential
>> scan for a table allocates a buffer of this size for each table it scans.
>> If we do many sequential scans, we might want to increase this value)
>>
>> (10) *query_cache_type : 0 *
>>
>> (11) *query_cache_limit : 1048576* (Do not cache results that are
>> larger than this number of bytes)
>>
>> (12) *query_cache_size : 1048576* (The amount of memory allocated
>> for caching query results)
>>
>> (13) *thread_stack : 262144* (The stack size for each thread)
>>
>> (14) *net_buffer_length : 16384* (Each client thread is associated
>> with a connection buffer and result buffer. Both begin with a size given 
>> by
>> net_buffer_length but are dynamically enlarged up to max_allowed_packet
>> bytes as needed)
>>
>> (15) *max_allowed_packet : 4194304* (The 

Re: [Dev] [IS] EC2 Performance Analysis : Sudden TPS drop in User Add in 500 concurrency with 10million users

[Prabath Siriwardana]

Can you please compare the results you are getting now with the results we
got a week before in the same setup...? I guess we could get  ~1200 tps
with 500 concurrency for 1M users, without any drop in the tps...?

Thanks & regards,
-Prabath

On Sun, Jul 31, 2016 at 12:34 AM, Ishara Karunarathna 
wrote:

>
>
> On Sun, Jul 31, 2016 at 12:59 PM, Malith Jayasinghe 
> wrote:
>
>>
>>
>> On Sun, Jul 31, 2016 at 12:49 PM, Ishara Karunarathna 
>> wrote:
>>
>>> Hi Malith,
>>>
>>> On Sun, Jul 31, 2016 at 12:37 PM, Malith Jayasinghe 
>>> wrote:
>>>
 HI Indunil,
 Just a few question regarding this performance test you have done:

 What is the reason for selecting the concurrency = 500 here?

>>> This is the user expected concurrency level. Thats the reason we user
>>> this.
>>>

 Have you tested the behaviour for lower concurrency levels?

 *"currently the TPS is dropping from the initial TPS 1139.5/s to
 198.1/s in around 610 user count.(User Add)" - *How did you
 notice/measure this drop in TPS? Did you analyze the jmeter results
 offline? After it drops, does it improve after some time or does it stay
 the same?

>>> We test this with the Jmeter summery report.
>>> with latest results if we start again few min (2min) we can get this max
>>> tps and come down to around 250tps
>>>
>>
>> Ok so it comes down to 250tps and stays there?  Are you running these
>> tests without a warm-up period?
>>
> Nop.
>
> With 2s worm up and then 10s ramp up period
>
>>

 Did you look at the behaviour of latency?

 Thanks

 Malith


 On Fri, Jul 29, 2016 at 2:57 PM, Indunil Upeksha Rathnayake <
 indu...@wso2.com> wrote:

> Hi,
>
> We are currently engaged into a performance analysis where we are
> analyzing performance for User Add, Update, Authentication operations. The
> testing has been carried out in a following environment with 500
> concurrency and users up to 10 million.
>
> *Environment :*
>
> m3.2xlarge ( 8 core, 30GB, SSD 2x80 GB) 3 instances.
> MySQL 5.7
> Ubuntu 14.04
> Openldap-2.4.31
> IS 5.1.0
>
> In order to optimize the MYSQL server, following server parameters
> have been tuned accordingly. We have referred MYSQL documentation [1] as
> well as have performed analysis using several MYSQL tuners in [2].
>
> (1) *max_connections : 1000* (The maximum permitted number of
> simultaneous client connections.)
>
> (2) *join_buffer_size : 259968* (The minimum size of the buffer that
> is used for plain index scans, range index scans, and joins that do not 
> use
> indexes and thus perform full table scans.)
>
> (3) *innodb_buffer_pool_size : 5207959552 <5207959552>* (size of the
> memory area where InnoDB caches table and index data)
>
> (4) *innodb_log_buffer_size : 16777216* (size of the buffer for
> transactions that have not been committed yet)
>
> (5) *innodb_buffer_pool_instances : 1* (The number of buffer pool
> instances. According to the mysql documentation[1], on systems with a 
> large
> amount of memory, we can improve concurrency by dividing the buffer pool
> into multiple buffer pool instances. But couldn't change since it's a read
> only variable)
>
> (6) *key_buffer_size : 38400* (size of the buffer used for index
> blocks)
>
> (7) *table_open_cache : 4000* (The number of open tables for all
> threads)
>
> (8) *sort_buffer_size : 400* (Each session that must perform a
> sort allocates a buffer of this size)
>
> (9) *read_buffer_size : 100* (Each thread that does a sequential
> scan for a table allocates a buffer of this size for each table it scans.
> If we do many sequential scans, we might want to increase this value)
>
> (10) *query_cache_type : 0 *
>
> (11) *query_cache_limit : 1048576* (Do not cache results that are
> larger than this number of bytes)
>
> (12) *query_cache_size : 1048576* (The amount of memory allocated for
> caching query results)
>
> (13) *thread_stack : 262144* (The stack size for each thread)
>
> (14) *net_buffer_length : 16384* (Each client thread is associated
> with a connection buffer and result buffer. Both begin with a size given 
> by
> net_buffer_length but are dynamically enlarged up to max_allowed_packet
> bytes as needed)
>
> (15) *max_allowed_packet : 4194304* (The maximum size of one packet
> or any generated/intermediate string)
>
> (16) *thread_cache_size : 30* (no of threads the server should cache
> for reuse)
>
>
>
> IS has been configured as follows to optimize the performance.
>
> (1) JVM Heap Settings (-Xms -Xmx) changed as follows:
>
> *Xms : 2g *
>

Re: [Dev] GSOC 2016 - Project 21 : MongoDB Userstore Development

[Tharindu Edirisinghe]

Hi Asantha,

Good work so far. Since we are reaching the final stage of GSoC, let's have
a hangout at the end of this week and plan the final steps to complete the
project.

I will go through the deliverables and provide feedback soon.

Thanks,
Tharindu

On Mon, Jul 25, 2016 at 12:08 PM, Asantha Thilina 
wrote:

> Hi Tharindu,
>
> I have cleaned up all the bugs and removed all unnecessary codes and
> committed all the changes into my repository[1]
>
> [1] https://github.com/asanthamax/mongodbuserstore
>
> Thanks,
> Asantha
>
> On Sat, Jul 23, 2016 at 8:26 AM, Asantha Thilina  > wrote:
>
>> Hi Tharindu,
>>
>> here i added all the commitments to spread sheet[1] and also i updated my
>> blog with about java document api[2] and also i added findbug report[3] and
>> also i updated my blog with the code quality analysis process i have done[4]
>>
>> [1]
>> https://docs.google.com/spreadsheets/d/1jwSR_qFV-LfOS7ZTZ8CpiFFTYPoudDQF7zQhvblivaE/edit?usp=sharing
>> [2]
>> http://asanthamax.blogspot.com/2016/07/mongodb-user-store-package-hierarchy.html
>> [3]https://googledrive.com/host/0B__ZE1ru1jkXZFdES3BOaHVyZ3M/
>> [4]
>> http://asanthamax.blogspot.com/2016/07/findbugs-in-mongodbuserstoremanager-in.html
>>
>> Thanks,
>> Asantha
>>
>> On Fri, Jul 22, 2016 at 8:45 AM, Asantha Thilina <
>> asanthathil...@gmail.com> wrote:
>>
>>> Hi Tharindu,
>>>
>>> I have created the API documentation and hosted in my google drive[1] i
>>> will be preparing code inspection report as well will share it as soon as
>>> well i also added github page also to the userstore script in here[2] as
>>> well
>>> [1] http://googledrive.com/host/0B__ZE1ru1jkXbDVoSDlveV9yNk0
>>> [2] https://asanthamax.github.io/mongodbuserstore/
>>>
>>> Thanks,
>>> Asantha
>>>
>>> On Tue, Jul 12, 2016 at 11:59 AM, Asantha Thilina <
>>> asanthathil...@gmail.com> wrote:
>>>
 Hi Tharindu,
 sure we will arrange a hangout on next week and sure i will refer the
 link and do the necessary inspect in my code for any security
 vulnerabilities and update you sooner

 Thanks,
 Asantha

 On Tue, Jul 12, 2016 at 9:56 AM, Tharindu Edirisinghe <
 tharin...@wso2.com> wrote:

> Hi Asantha,
>
> Thanks for the update. I'm out of the country this week and won't be
> available. Let's have a hangout next week and discuss the final steps of
> the project.
>
> Along with find bugs, if you haven't enabled findsecuritybugs plugin,
> you can refer [1] and check if any security weaknesses are there in the
> code.
>
> [1]
> http://tharindue.blogspot.in/2016/06/static-code-analysis-for-java-using.html
>
> Thanks,
> TharinduE
>
> On Tue, Jul 12, 2016 at 5:32 PM, Asantha Thilina <
> asanthathil...@gmail.com> wrote:
>
>> Hi Tharindu,
>> I have clean up the code according to find bug plugin and also for
>> rollback purposes since it only concern when modifying or adding new user
>> or role to user store i have manually added rollback option where if 
>> error
>> occured all the changes done to db in code segment will be discard from 
>> db
>> i updated the my repository with recent changes[1] now i preparing the
>> documentation and test scripts.
>>
>> [1] https://github.com/asanthamax/mongodbuserstore
>>
>> Thanks,
>> Asantha
>>
>> On Mon, Jul 4, 2016 at 2:54 AM, Asantha Thilina <
>> asanthathil...@gmail.com> wrote:
>>
>>> Hi Tharindu,
>>>
>>> thanks for the reply ,sure i will try out that and update the
>>> progress as soon as possible
>>>
>>> Thanks,
>>> Asantha
>>>
>>> On Mon, Jul 4, 2016 at 1:38 AM, Tharindu Edirisinghe <
>>> tharin...@wso2.com> wrote:
>>>
 Hi Asantha,

 While you are adding the roll back transaction capability, also
 look into testing a high load and try to generate database level
 exceptions. Once thing you can try out is writing a client for 
 performing
 userstore manager operations such as add user, add role, login, logout 
 etc.
 and play it with a high concurrency using Jmeter. Then we should be 
 able to
 get a better understanding of transaction management with respective to
 MongoDB.

 Regards,
 TharinduE

 On Mon, Jul 4, 2016 at 10:57 AM, Asantha Thilina <
 asanthathil...@gmail.com> wrote:

> Hi all,
>
> i will be trying for implement roll back transactions capability
> in MongoDB to add to MongoDB User Store[2] i found that MongoDB use a
> concept call two phase commit[1] i trying to implement that inside my 
> code
> but it seems to be little complex and i think it may arise performance
> issue when adding high load of dataset at once is there any other 
> 

Re: [Dev] [IoTS] when will the IoTS be GA?

[Sumedha Rubasinghe]

Hi Jason,
UIs in IoTS Master is not fully functional due to on going work I mentioned
b4.

Will get back to you with an alternative plan by tomorrow.

On Jul 31, 2016 2:48 PM, "云展智创"  wrote:

Hi Sumedha,

Yes, I thought the IoTS would be GA on mid of July. I have a project using
IoTS to manage some smart devices. IoTS is the core backend component in
the project.  I also need the cdmf-devicetype-archetype tools to create the
plugins. Could you give me some advices on how to build the latest stable
release from the code?

by the way, I found that the real time device statistics data could not be
showed on the graph of device view in 1.0.0 Alpha release. Is that a know
error? Thanks.

－－
Zhanwen Zhou (Jason), General Manager
+86 13922218435
zhanwen.z...@smartcloudex.com
Guangzhou Smart Cloudex Technology Co., Ltd.
Business: API Management, IOT

On Jul 31, 2016, at 11:20 AM, Sumedha Rubasinghe  wrote:

Hi Jason,
After the IoTS 1.0.0 Alpha, we are in the middle of a strategic change
which involves our mobile device management capabilities.
As you may already be aware, we have brought down all core device
management capabilities into one level and implementing specific device
management plugins on top of this. These device management plugins also
cover our Android, iOS, Windows mobile plugins (as well as plugins for all
other devices).

Originally 1.0.0 GA release date of IoTS was scheduled for July. But due to
a customer commitment we have shifted our focus to stabilising Android
device management until end of August.

Specific new release date of IoTS 1.0.0 GA will be decided after August.

If you have a project that gets impacted due to this shift please fill up
our contact us form @ http://wso2.com/contact/ with your concern and we
will see what we can do.

Thanks,
Sumedha


On Sat, Jul 30, 2016 at 3:03 PM, 云展智创  wrote:

> Hi,
>
> Could anyone tell me when will the IoTS be GA? Thanks a lot.
>
> －－
> Zhanwen Zhou (Jason),
> +86 13922218435
> zhanwen.z...@smartcloudex.com
> Guangzhou Smart Cloudex Technology Co., Ltd.
> Business: API Management, IOT
>
> ___
> Dev mailing list
> Dev@wso2.org
> http://wso2.org/cgi-bin/mailman/listinfo/dev
>



-- 
/sumedha
m: +94 773017743
b :  bit.ly/sumedha
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] [IoTS] when will the IoTS be GA?

[云展智创]

Hi Sumedha,

Yes, I thought the IoTS would be GA on mid of July. I have a project using IoTS 
to manage some smart devices. IoTS is the core backend component in the 
project.  I also need the cdmf-devicetype-archetype tools to create the 
plugins. Could you give me some advices on how to build the latest stable 
release from the code? 

by the way, I found that the real time device statistics data could not be 
showed on the graph of device view in 1.0.0 Alpha release. Is that a know 
error? Thanks.

－－
Zhanwen Zhou (Jason), General Manager
+86 13922218435
zhanwen.z...@smartcloudex.com
Guangzhou Smart Cloudex Technology Co., Ltd.
Business: API Management, IOT

On Jul 31, 2016, at 11:20 AM, Sumedha Rubasinghe  wrote:

> Hi Jason,
> After the IoTS 1.0.0 Alpha, we are in the middle of a strategic change which 
> involves our mobile device management capabilities. 
> As you may already be aware, we have brought down all core device management 
> capabilities into one level and implementing specific device management 
> plugins on top of this. These device management plugins also cover our 
> Android, iOS, Windows mobile plugins (as well as plugins for all other 
> devices).
> 
> Originally 1.0.0 GA release date of IoTS was scheduled for July. But due to a 
> customer commitment we have shifted our focus to stabilising Android device 
> management until end of August.
> 
> Specific new release date of IoTS 1.0.0 GA will be decided after August.
> 
> If you have a project that gets impacted due to this shift please fill up our 
> contact us form @ http://wso2.com/contact/ with your concern and we will see 
> what we can do.
>  
> Thanks,
> Sumedha
> 
> 
> On Sat, Jul 30, 2016 at 3:03 PM, 云展智创  wrote:
> Hi,
> 
> Could anyone tell me when will the IoTS be GA? Thanks a lot.
> 
> －－
> Zhanwen Zhou (Jason),
> +86 13922218435
> zhanwen.z...@smartcloudex.com
> Guangzhou Smart Cloudex Technology Co., Ltd.
> Business: API Management, IOT
> 
> ___
> Dev mailing list
> Dev@wso2.org
> http://wso2.org/cgi-bin/mailman/listinfo/dev
> 
> 
> 
> -- 
> /sumedha
> m: +94 773017743
> b :  bit.ly/sumedha

___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] [IS] EC2 Performance Analysis : Sudden TPS drop in User Add in 500 concurrency with 10million users

[Ishara Karunarathna]

On Sun, Jul 31, 2016 at 12:59 PM, Malith Jayasinghe 
wrote:

>
>
> On Sun, Jul 31, 2016 at 12:49 PM, Ishara Karunarathna 
> wrote:
>
>> Hi Malith,
>>
>> On Sun, Jul 31, 2016 at 12:37 PM, Malith Jayasinghe 
>> wrote:
>>
>>> HI Indunil,
>>> Just a few question regarding this performance test you have done:
>>>
>>> What is the reason for selecting the concurrency = 500 here?
>>>
>> This is the user expected concurrency level. Thats the reason we user
>> this.
>>
>>>
>>> Have you tested the behaviour for lower concurrency levels?
>>>
>>> *"currently the TPS is dropping from the initial TPS 1139.5/s to 198.1/s
>>> in around 610 user count.(User Add)" - *How did you notice/measure
>>> this drop in TPS? Did you analyze the jmeter results offline? After it
>>> drops, does it improve after some time or does it stay the same?
>>>
>> We test this with the Jmeter summery report.
>> with latest results if we start again few min (2min) we can get this max
>> tps and come down to around 250tps
>>
>
> Ok so it comes down to 250tps and stays there?  Are you running these
> tests without a warm-up period?
>
Nop.

With 2s worm up and then 10s ramp up period

>
>>>
>>> Did you look at the behaviour of latency?
>>>
>>> Thanks
>>>
>>> Malith
>>>
>>>
>>> On Fri, Jul 29, 2016 at 2:57 PM, Indunil Upeksha Rathnayake <
>>> indu...@wso2.com> wrote:
>>>
 Hi,

 We are currently engaged into a performance analysis where we are
 analyzing performance for User Add, Update, Authentication operations. The
 testing has been carried out in a following environment with 500
 concurrency and users up to 10 million.

 *Environment :*

 m3.2xlarge ( 8 core, 30GB, SSD 2x80 GB) 3 instances.
 MySQL 5.7
 Ubuntu 14.04
 Openldap-2.4.31
 IS 5.1.0

 In order to optimize the MYSQL server, following server parameters have
 been tuned accordingly. We have referred MYSQL documentation [1] as well as
 have performed analysis using several MYSQL tuners in [2].

 (1) *max_connections : 1000* (The maximum permitted number of
 simultaneous client connections.)

 (2) *join_buffer_size : 259968* (The minimum size of the buffer that
 is used for plain index scans, range index scans, and joins that do not use
 indexes and thus perform full table scans.)

 (3) *innodb_buffer_pool_size : 5207959552 <5207959552>* (size of the
 memory area where InnoDB caches table and index data)

 (4) *innodb_log_buffer_size : 16777216* (size of the buffer for
 transactions that have not been committed yet)

 (5) *innodb_buffer_pool_instances : 1* (The number of buffer pool
 instances. According to the mysql documentation[1], on systems with a large
 amount of memory, we can improve concurrency by dividing the buffer pool
 into multiple buffer pool instances. But couldn't change since it's a read
 only variable)

 (6) *key_buffer_size : 38400* (size of the buffer used for index
 blocks)

 (7) *table_open_cache : 4000* (The number of open tables for all
 threads)

 (8) *sort_buffer_size : 400* (Each session that must perform a
 sort allocates a buffer of this size)

 (9) *read_buffer_size : 100* (Each thread that does a sequential
 scan for a table allocates a buffer of this size for each table it scans.
 If we do many sequential scans, we might want to increase this value)

 (10) *query_cache_type : 0 *

 (11) *query_cache_limit : 1048576* (Do not cache results that are
 larger than this number of bytes)

 (12) *query_cache_size : 1048576* (The amount of memory allocated for
 caching query results)

 (13) *thread_stack : 262144* (The stack size for each thread)

 (14) *net_buffer_length : 16384* (Each client thread is associated
 with a connection buffer and result buffer. Both begin with a size given by
 net_buffer_length but are dynamically enlarged up to max_allowed_packet
 bytes as needed)

 (15) *max_allowed_packet : 4194304* (The maximum size of one packet or
 any generated/intermediate string)

 (16) *thread_cache_size : 30* (no of threads the server should cache
 for reuse)



 IS has been configured as follows to optimize the performance.

 (1) JVM Heap Settings (-Xms -Xmx) changed as follows:

 *Xms : 2g *

 *Xmx : 2g *

 (2) Removed following entry from
 /repository/conf/tomcat/catalina-server.xml to disable http access
 logs.

 >>> directory="${carbon.home}/repository/logs" prefix="http_access_"
 suffix=".log" pattern="combined" />

 (3) Tuned following parameters in axis2client.xml file.

 1000

 3

 (4) Added following additional parameters to optimize database
 connection pool.

 6

 600

 

Re: [Dev] [IS] EC2 Performance Analysis : Sudden TPS drop in User Add in 500 concurrency with 10million users

[Malith Jayasinghe]

On Sun, Jul 31, 2016 at 12:49 PM, Ishara Karunarathna 
wrote:

> Hi Malith,
>
> On Sun, Jul 31, 2016 at 12:37 PM, Malith Jayasinghe 
> wrote:
>
>> HI Indunil,
>> Just a few question regarding this performance test you have done:
>>
>> What is the reason for selecting the concurrency = 500 here?
>>
> This is the user expected concurrency level. Thats the reason we user
> this.
>
>>
>> Have you tested the behaviour for lower concurrency levels?
>>
>> *"currently the TPS is dropping from the initial TPS 1139.5/s to 198.1/s
>> in around 610 user count.(User Add)" - *How did you notice/measure
>> this drop in TPS? Did you analyze the jmeter results offline? After it
>> drops, does it improve after some time or does it stay the same?
>>
> We test this with the Jmeter summery report.
> with latest results if we start again few min (2min) we can get this max
> tps and come down to around 250tps
>

Ok so it comes down to 250tps and stays there?  Are you running these tests
without a warm-up period?

>
>>
>> Did you look at the behaviour of latency?
>>
>> Thanks
>>
>> Malith
>>
>>
>> On Fri, Jul 29, 2016 at 2:57 PM, Indunil Upeksha Rathnayake <
>> indu...@wso2.com> wrote:
>>
>>> Hi,
>>>
>>> We are currently engaged into a performance analysis where we are
>>> analyzing performance for User Add, Update, Authentication operations. The
>>> testing has been carried out in a following environment with 500
>>> concurrency and users up to 10 million.
>>>
>>> *Environment :*
>>>
>>> m3.2xlarge ( 8 core, 30GB, SSD 2x80 GB) 3 instances.
>>> MySQL 5.7
>>> Ubuntu 14.04
>>> Openldap-2.4.31
>>> IS 5.1.0
>>>
>>> In order to optimize the MYSQL server, following server parameters have
>>> been tuned accordingly. We have referred MYSQL documentation [1] as well as
>>> have performed analysis using several MYSQL tuners in [2].
>>>
>>> (1) *max_connections : 1000* (The maximum permitted number of
>>> simultaneous client connections.)
>>>
>>> (2) *join_buffer_size : 259968* (The minimum size of the buffer that is
>>> used for plain index scans, range index scans, and joins that do not use
>>> indexes and thus perform full table scans.)
>>>
>>> (3) *innodb_buffer_pool_size : 5207959552 <5207959552>* (size of the
>>> memory area where InnoDB caches table and index data)
>>>
>>> (4) *innodb_log_buffer_size : 16777216* (size of the buffer for
>>> transactions that have not been committed yet)
>>>
>>> (5) *innodb_buffer_pool_instances : 1* (The number of buffer pool
>>> instances. According to the mysql documentation[1], on systems with a large
>>> amount of memory, we can improve concurrency by dividing the buffer pool
>>> into multiple buffer pool instances. But couldn't change since it's a read
>>> only variable)
>>>
>>> (6) *key_buffer_size : 38400* (size of the buffer used for index
>>> blocks)
>>>
>>> (7) *table_open_cache : 4000* (The number of open tables for all
>>> threads)
>>>
>>> (8) *sort_buffer_size : 400* (Each session that must perform a sort
>>> allocates a buffer of this size)
>>>
>>> (9) *read_buffer_size : 100* (Each thread that does a sequential
>>> scan for a table allocates a buffer of this size for each table it scans.
>>> If we do many sequential scans, we might want to increase this value)
>>>
>>> (10) *query_cache_type : 0 *
>>>
>>> (11) *query_cache_limit : 1048576* (Do not cache results that are
>>> larger than this number of bytes)
>>>
>>> (12) *query_cache_size : 1048576* (The amount of memory allocated for
>>> caching query results)
>>>
>>> (13) *thread_stack : 262144* (The stack size for each thread)
>>>
>>> (14) *net_buffer_length : 16384* (Each client thread is associated with
>>> a connection buffer and result buffer. Both begin with a size given by
>>> net_buffer_length but are dynamically enlarged up to max_allowed_packet
>>> bytes as needed)
>>>
>>> (15) *max_allowed_packet : 4194304* (The maximum size of one packet or
>>> any generated/intermediate string)
>>>
>>> (16) *thread_cache_size : 30* (no of threads the server should cache
>>> for reuse)
>>>
>>>
>>>
>>> IS has been configured as follows to optimize the performance.
>>>
>>> (1) JVM Heap Settings (-Xms -Xmx) changed as follows:
>>>
>>> *Xms : 2g *
>>>
>>> *Xmx : 2g *
>>>
>>> (2) Removed following entry from
>>> /repository/conf/tomcat/catalina-server.xml to disable http access
>>> logs.
>>>
>>> >> directory="${carbon.home}/repository/logs" prefix="http_access_"
>>> suffix=".log" pattern="combined" />
>>>
>>> (3) Tuned following parameters in axis2client.xml file.
>>>
>>> 1000
>>>
>>> 3
>>>
>>> (4) Added following additional parameters to optimize database
>>> connection pool.
>>>
>>> 6
>>>
>>> 600
>>>
>>> 20
>>>
>>> (5) Tuning Tomcat parameters in
>>> /repository/conf/tomcat/catalina-server.xml.
>>>
>>> *acceptorThreadCount = 8 *
>>>
>>> *maxThreads="750" *
>>>
>>> *minSpareThreads="150" *
>>>
>>> *maxKeepAliveRequests="600" *
>>>
>>> *acceptCount="600"*
>>>
>>>
>>>
>>> JMeter has been 

Re: [Dev] [IS] EC2 Performance Analysis : Sudden TPS drop in User Add in 500 concurrency with 10million users

[Ishara Karunarathna]

Hi Malith,

On Sun, Jul 31, 2016 at 12:37 PM, Malith Jayasinghe 
wrote:

> HI Indunil,
> Just a few question regarding this performance test you have done:
>
> What is the reason for selecting the concurrency = 500 here?
>
This is the user expected concurrency level. Thats the reason we user this.

>
> Have you tested the behaviour for lower concurrency levels?
>
> *"currently the TPS is dropping from the initial TPS 1139.5/s to 198.1/s
> in around 610 user count.(User Add)" - *How did you notice/measure
> this drop in TPS? Did you analyze the jmeter results offline? After it
> drops, does it improve after some time or does it stay the same?
>
We test this with the Jmeter summery report.
with latest results if we start again few min (2min) we can get this max
tps and come down to around 250tps

>
>
> Did you look at the behaviour of latency?
>
> Thanks
>
> Malith
>
>
> On Fri, Jul 29, 2016 at 2:57 PM, Indunil Upeksha Rathnayake <
> indu...@wso2.com> wrote:
>
>> Hi,
>>
>> We are currently engaged into a performance analysis where we are
>> analyzing performance for User Add, Update, Authentication operations. The
>> testing has been carried out in a following environment with 500
>> concurrency and users up to 10 million.
>>
>> *Environment :*
>>
>> m3.2xlarge ( 8 core, 30GB, SSD 2x80 GB) 3 instances.
>> MySQL 5.7
>> Ubuntu 14.04
>> Openldap-2.4.31
>> IS 5.1.0
>>
>> In order to optimize the MYSQL server, following server parameters have
>> been tuned accordingly. We have referred MYSQL documentation [1] as well as
>> have performed analysis using several MYSQL tuners in [2].
>>
>> (1) *max_connections : 1000* (The maximum permitted number of
>> simultaneous client connections.)
>>
>> (2) *join_buffer_size : 259968* (The minimum size of the buffer that is
>> used for plain index scans, range index scans, and joins that do not use
>> indexes and thus perform full table scans.)
>>
>> (3) *innodb_buffer_pool_size : 5207959552 <5207959552>* (size of the
>> memory area where InnoDB caches table and index data)
>>
>> (4) *innodb_log_buffer_size : 16777216* (size of the buffer for
>> transactions that have not been committed yet)
>>
>> (5) *innodb_buffer_pool_instances : 1* (The number of buffer pool
>> instances. According to the mysql documentation[1], on systems with a large
>> amount of memory, we can improve concurrency by dividing the buffer pool
>> into multiple buffer pool instances. But couldn't change since it's a read
>> only variable)
>>
>> (6) *key_buffer_size : 38400* (size of the buffer used for index
>> blocks)
>>
>> (7) *table_open_cache : 4000* (The number of open tables for all
>> threads)
>>
>> (8) *sort_buffer_size : 400* (Each session that must perform a sort
>> allocates a buffer of this size)
>>
>> (9) *read_buffer_size : 100* (Each thread that does a sequential
>> scan for a table allocates a buffer of this size for each table it scans.
>> If we do many sequential scans, we might want to increase this value)
>>
>> (10) *query_cache_type : 0 *
>>
>> (11) *query_cache_limit : 1048576* (Do not cache results that are larger
>> than this number of bytes)
>>
>> (12) *query_cache_size : 1048576* (The amount of memory allocated for
>> caching query results)
>>
>> (13) *thread_stack : 262144* (The stack size for each thread)
>>
>> (14) *net_buffer_length : 16384* (Each client thread is associated with
>> a connection buffer and result buffer. Both begin with a size given by
>> net_buffer_length but are dynamically enlarged up to max_allowed_packet
>> bytes as needed)
>>
>> (15) *max_allowed_packet : 4194304* (The maximum size of one packet or
>> any generated/intermediate string)
>>
>> (16) *thread_cache_size : 30* (no of threads the server should cache for
>> reuse)
>>
>>
>>
>> IS has been configured as follows to optimize the performance.
>>
>> (1) JVM Heap Settings (-Xms -Xmx) changed as follows:
>>
>> *Xms : 2g *
>>
>> *Xmx : 2g *
>>
>> (2) Removed following entry from
>> /repository/conf/tomcat/catalina-server.xml to disable http access
>> logs.
>>
>> > directory="${carbon.home}/repository/logs" prefix="http_access_"
>> suffix=".log" pattern="combined" />
>>
>> (3) Tuned following parameters in axis2client.xml file.
>>
>> 1000
>>
>> 3
>>
>> (4) Added following additional parameters to optimize database connection
>> pool.
>>
>> 6
>>
>> 600
>>
>> 20
>>
>> (5) Tuning Tomcat parameters in
>> /repository/conf/tomcat/catalina-server.xml.
>>
>> *acceptorThreadCount = 8 *
>>
>> *maxThreads="750" *
>>
>> *minSpareThreads="150" *
>>
>> *maxKeepAliveRequests="600" *
>>
>> *acceptCount="600"*
>>
>>
>>
>> JMeter has been configured as follows to optimize the performance.
>>
>> (1) JVM Heap Settings (-Xms -Xmx) changed as follows:
>>
>> *Xms : 1g *
>>
>> *Xmx : 1g *
>>
>>
>> We were able to optimize the environment up to some level. But*
>> currently the TPS is dropping from the initial TPS 1139.5/s to 198.1/s in
>> around 610 user count.(User Add)*
>>
>> 

Re: [Dev] [IS] EC2 Performance Analysis : Sudden TPS drop in User Add in 500 concurrency with 10million users

[Malith Jayasinghe]

HI Indunil,
Just a few question regarding this performance test you have done:

What is the reason for selecting the concurrency = 500 here?

Have you tested the behaviour for lower concurrency levels?

*"currently the TPS is dropping from the initial TPS 1139.5/s to 198.1/s in
around 610 user count.(User Add)" - *How did you notice/measure this
drop in TPS? Did you analyze the jmeter results offline? After it drops,
does it improve after some time or does it stay the same?

Did you look at the behaviour of latency?

Thanks

Malith


On Fri, Jul 29, 2016 at 2:57 PM, Indunil Upeksha Rathnayake <
indu...@wso2.com> wrote:

> Hi,
>
> We are currently engaged into a performance analysis where we are
> analyzing performance for User Add, Update, Authentication operations. The
> testing has been carried out in a following environment with 500
> concurrency and users up to 10 million.
>
> *Environment :*
>
> m3.2xlarge ( 8 core, 30GB, SSD 2x80 GB) 3 instances.
> MySQL 5.7
> Ubuntu 14.04
> Openldap-2.4.31
> IS 5.1.0
>
> In order to optimize the MYSQL server, following server parameters have
> been tuned accordingly. We have referred MYSQL documentation [1] as well as
> have performed analysis using several MYSQL tuners in [2].
>
> (1) *max_connections : 1000* (The maximum permitted number of
> simultaneous client connections.)
>
> (2) *join_buffer_size : 259968* (The minimum size of the buffer that is
> used for plain index scans, range index scans, and joins that do not use
> indexes and thus perform full table scans.)
>
> (3) *innodb_buffer_pool_size : 5207959552 <5207959552>* (size of the
> memory area where InnoDB caches table and index data)
>
> (4) *innodb_log_buffer_size : 16777216* (size of the buffer for
> transactions that have not been committed yet)
>
> (5) *innodb_buffer_pool_instances : 1* (The number of buffer pool
> instances. According to the mysql documentation[1], on systems with a large
> amount of memory, we can improve concurrency by dividing the buffer pool
> into multiple buffer pool instances. But couldn't change since it's a read
> only variable)
>
> (6) *key_buffer_size : 38400* (size of the buffer used for index
> blocks)
>
> (7) *table_open_cache : 4000* (The number of open tables for all threads)
>
> (8) *sort_buffer_size : 400* (Each session that must perform a sort
> allocates a buffer of this size)
>
> (9) *read_buffer_size : 100* (Each thread that does a sequential scan
> for a table allocates a buffer of this size for each table it scans. If we
> do many sequential scans, we might want to increase this value)
>
> (10) *query_cache_type : 0 *
>
> (11) *query_cache_limit : 1048576* (Do not cache results that are larger
> than this number of bytes)
>
> (12) *query_cache_size : 1048576* (The amount of memory allocated for
> caching query results)
>
> (13) *thread_stack : 262144* (The stack size for each thread)
>
> (14) *net_buffer_length : 16384* (Each client thread is associated with a
> connection buffer and result buffer. Both begin with a size given by
> net_buffer_length but are dynamically enlarged up to max_allowed_packet
> bytes as needed)
>
> (15) *max_allowed_packet : 4194304* (The maximum size of one packet or
> any generated/intermediate string)
>
> (16) *thread_cache_size : 30* (no of threads the server should cache for
> reuse)
>
>
>
> IS has been configured as follows to optimize the performance.
>
> (1) JVM Heap Settings (-Xms -Xmx) changed as follows:
>
> *Xms : 2g *
>
> *Xmx : 2g *
>
> (2) Removed following entry from
> /repository/conf/tomcat/catalina-server.xml to disable http access
> logs.
>
>  directory="${carbon.home}/repository/logs" prefix="http_access_"
> suffix=".log" pattern="combined" />
>
> (3) Tuned following parameters in axis2client.xml file.
>
> 1000
>
> 3
>
> (4) Added following additional parameters to optimize database connection
> pool.
>
> 6
>
> 600
>
> 20
>
> (5) Tuning Tomcat parameters in
> /repository/conf/tomcat/catalina-server.xml.
>
> *acceptorThreadCount = 8 *
>
> *maxThreads="750" *
>
> *minSpareThreads="150" *
>
> *maxKeepAliveRequests="600" *
>
> *acceptCount="600"*
>
>
>
> JMeter has been configured as follows to optimize the performance.
>
> (1) JVM Heap Settings (-Xms -Xmx) changed as follows:
>
> *Xms : 1g *
>
> *Xmx : 1g *
>
>
> We were able to optimize the environment up to some level. But* currently
> the TPS is dropping from the initial TPS 1139.5/s to 198.1/s in around
> 610 user count.(User Add)*
>
> Appreciate your help on figuring out whether we need to do any
> modifications to the optimizations in MYSQL, IS and JMeter servers or to
> identify the exact issue for this sudden TPS dropping.
>
> [1] http://dev.mysql.com/doc/refman/5.7/en/optimizing-server.html
>
> [2] http://www.askapache.com/mysql/mysql-performance-tuning.html
>
>
> Thanks and Regards
> --
> Indunil Upeksha Rathnayake
> Software Engineer | WSO2 Inc
> Emailindu...@wso2.com
> Mobile   0772182255
>



-- 
Malith Jayasinghe