Re: [Architecture] MB 3.0.0 MQTT Integration Status
While testing with the Interop test suite the following tests failed with the default parameters (TIMEOUT = 500 ms and CONNECTION_TIME_OUT = 1000 ms). where timeout is the amount of time a subscriber awaits for a message and connection time out is the whole time the connection will be maintained with the broker. Test Basics , Test Long Topic, Test OverLapping Topics, Test Dots , Test Active MQ Wild Cards , Test Native MQTT Wild Cards , Test Subs , Test Wild Card Plus , Test LWT, Test Clean Session further, when the parameters were incremented to (TIMEOUT = 1000 ms and CONNECTION_TIME_OUT = 5000 ms). The failing tests were narrowed down to the following, Test Long Topic, Test OverLapping Topic, Test Active MQ Wild Cards, Test Native MQTT Wild Cards, Test Subs, Test Wild Card Plus, Test LWT, Test Clean Session currently we're in progress of fixing the issues related to cause test failures, also will be looking at the possibilities to reduce the latency of the response time so that the tests will be positive for the default parameters. Thanks, Pamod On Sat, Mar 8, 2014 at 4:58 PM, Pamod Sylvester pa...@wso2.com wrote: Hi Paul, +1 to reflect the changes back to the community. Will get in touch with their mailing list. Also next line item is to test the flows using the Interop test suite. Will get back to you with the results ASAP. Thanks, Pamod On Saturday, March 8, 2014, Paul Fremantle p...@wso2.com wrote: Pamod The ideal situation is to aim to get these changes back into Moquette. That means (a) getting them into a shape where they are generically useful to the Moquette community - so that other people wishing to embed Moquette would also look at this and say yes please, and (b) persuading the Moquette group this is a valuable change/addition. Is there any chance of doing an interop test with a beta on the 17th March at EclipseCon? I'll be at the interop day. Paul On 8 March 2014 05:36, Pamod Sylvester pa...@wso2.com wrote: Hi Samisa, Yes we did changes which are local to the Moquette library, However, the changes were done in a separate class to act as an interface between the Andes Kernal and the MQTT library. All the changes which are specific for the kernel integration are embedded in that class, so that when the library updates with a minimum effort we could adopt to the changes. However, it was also considered to totally abstract the library without having us modify any of its content but rather call only its interfaces. Then the challenge was to have a way to resolve circular dependencies. Where, to start the MQTT server MQTT library will depend on Andes and for message exchange Andes will depend on MQTT library. Also, IMO in order to maintain patches which we also need to have the library source code in a separate branch and maintain the way we do for Synapse. Please do correct me if i am wrong. Thanks, Pamod On Sat, Mar 8, 2014 at 10:35 AM, Samisa Abeysinghe sam...@wso2.com wrote: On Sat, Mar 8, 2014 at 10:00 AM, Pamod Sylvester pa...@wso2.com wrote: Hi All, The following is the status of the MQTT Moquette library integration with MB 3.0.0, ~ Library was revamped to fit in with the Andes kernel. Have we done local changes in this re-vamp? How do we plan to maintain those if yes? Attached diagram depicts how the messages are flowed between the components at a high level. ~ Implemented a MQTT client for subscription and publishing. ~ Also the Moquette library was patched, the default behaviour of the library did not support multiple subscribers to receive a published message. How do we plan to maintain these patches? That was due to a bug which had a common ByteBuffer shared between concurrently accessing subscriber connections. The issue was fixed by cloning ByteBuffer instances per subscriber. Note that the wrapped array element was not duplicated or changed. The implemented code and artefacts could be found in [1]. Will commit the code once reviewed. To-Dos ~Need to package the dependencies. - Need to prepare test cases that would cover all the aspects described in the MQTT spec. - Need to test MQTT use cases in a cluster. - Need to load test and perform a long running test. - Need to discuss on the UI aspects. Please do let know if there're amendments to be made. Thanks, Pamod [1] https://svn.wso2.org/repos/wso2/scratch/pamod/mqtt_wrok/ -- Pamod Sylvester Software Engineer Integration Technologies Team, WSO2 Inc.; http://wso2.com email: pa...@wso2.com cell: +94 77 7779495 ___ Architecture mailing list Architecture@wso2.org Paul Fremantle CTO and Co-Founder, WSO2 OASIS WS-RX TC Co-chair, Apache Member UK: +44 207 096 0336 US: +1 646 595 7614 blog: http://pzf.fremantle.org twitter.com/pzfreo p...@wso2.com wso2.com Lean Enterprise Middleware Disclaimer: This
Re: [Architecture] MB 3.0.0 MQTT Integration Status
Pamod. shall we write some test cases right away .. to test it and load test it. Make sure we do all their QOS scenarios How is publishing and receiving throughputs? we need to measure that Also What is end to end latency? --Srinath On Mon, Mar 10, 2014 at 12:38 PM, Pamod Sylvester pa...@wso2.com wrote: While testing with the Interop test suite the following tests failed with the default parameters (TIMEOUT = 500 ms and CONNECTION_TIME_OUT = 1000 ms). where timeout is the amount of time a subscriber awaits for a message and connection time out is the whole time the connection will be maintained with the broker. Test Basics , Test Long Topic, Test OverLapping Topics, Test Dots , Test Active MQ Wild Cards , Test Native MQTT Wild Cards , Test Subs , Test Wild Card Plus , Test LWT, Test Clean Session further, when the parameters were incremented to (TIMEOUT = 1000 ms and CONNECTION_TIME_OUT = 5000 ms). The failing tests were narrowed down to the following, Test Long Topic, Test OverLapping Topic, Test Active MQ Wild Cards, Test Native MQTT Wild Cards, Test Subs, Test Wild Card Plus, Test LWT, Test Clean Session currently we're in progress of fixing the issues related to cause test failures, also will be looking at the possibilities to reduce the latency of the response time so that the tests will be positive for the default parameters. Thanks, Pamod On Sat, Mar 8, 2014 at 4:58 PM, Pamod Sylvester pa...@wso2.com wrote: Hi Paul, +1 to reflect the changes back to the community. Will get in touch with their mailing list. Also next line item is to test the flows using the Interop test suite. Will get back to you with the results ASAP. Thanks, Pamod On Saturday, March 8, 2014, Paul Fremantle p...@wso2.com wrote: Pamod The ideal situation is to aim to get these changes back into Moquette. That means (a) getting them into a shape where they are generically useful to the Moquette community - so that other people wishing to embed Moquette would also look at this and say yes please, and (b) persuading the Moquette group this is a valuable change/addition. Is there any chance of doing an interop test with a beta on the 17th March at EclipseCon? I'll be at the interop day. Paul On 8 March 2014 05:36, Pamod Sylvester pa...@wso2.com wrote: Hi Samisa, Yes we did changes which are local to the Moquette library, However, the changes were done in a separate class to act as an interface between the Andes Kernal and the MQTT library. All the changes which are specific for the kernel integration are embedded in that class, so that when the library updates with a minimum effort we could adopt to the changes. However, it was also considered to totally abstract the library without having us modify any of its content but rather call only its interfaces. Then the challenge was to have a way to resolve circular dependencies. Where, to start the MQTT server MQTT library will depend on Andes and for message exchange Andes will depend on MQTT library. Also, IMO in order to maintain patches which we also need to have the library source code in a separate branch and maintain the way we do for Synapse. Please do correct me if i am wrong. Thanks, Pamod On Sat, Mar 8, 2014 at 10:35 AM, Samisa Abeysinghe sam...@wso2.com wrote: On Sat, Mar 8, 2014 at 10:00 AM, Pamod Sylvester pa...@wso2.com wrote: Hi All, The following is the status of the MQTT Moquette library integration with MB 3.0.0, ~ Library was revamped to fit in with the Andes kernel. Have we done local changes in this re-vamp? How do we plan to maintain those if yes? Attached diagram depicts how the messages are flowed between the components at a high level. ~ Implemented a MQTT client for subscription and publishing. ~ Also the Moquette library was patched, the default behaviour of the library did not support multiple subscribers to receive a published message. How do we plan to maintain these patches? That was due to a bug which had a common ByteBuffer shared between concurrently accessing subscriber connections. The issue was fixed by cloning ByteBuffer instances per subscriber. Note that the wrapped array element was not duplicated or changed. The implemented code and artefacts could be found in [1]. Will commit the code once reviewed. To-Dos ~Need to package the dependencies. - Need to prepare test cases that would cover all the aspects described in the MQTT spec. - Need to test MQTT use cases in a cluster. - Need to load test and perform a long running test. - Need to discuss on the UI aspects. Please do let know if there're amendments to be made. Thanks, Pamod [1] https://svn.wso2.org/repos/wso2/scratch/pamod/mqtt_wrok/ -- Pamod Sylvester Software Engineer Integration Technologies Team, WSO2 Inc.; http://wso2.com email: pa...@wso2.com cell: +94 77 7779495
Re: [Architecture] A few questions about WSO2 CEP/Siddhi
Hi Leo, Let me add couple of words as well On Mon, Mar 10, 2014 at 9:16 AM, Sriskandarajah Suhothayan s...@wso2.comwrote: Thanks for you interest in WSO2 CEP Siddhi Sorry for the late reply, I some how missed the mail On Fri, Mar 7, 2014 at 4:48 AM, Leo Romanoff romix...@yahoo.com wrote: Hi, I've started playing with WSO2 CEP recently and after some experiments with it I collected a few questions, which I cannot answer by reading the docs. Hopefully I'll have them answered by posting them on this list. First of all, some background information. I have some previous experience with Esper and Drools. Currently, I'm mostly interested in using Siddhi in embedded mode, i.e. as a library. I could built a few small test projects using it without any serious problems. But now I'm trying to understand some more complex things about it. So, here are my questions: 1) How many rules/queries can be defined in one engine. How does it affect performance? For example, can I define (tens of) thousands of queries using the same (or multiple) instance of SiddhiManager? Would it make processing much slower? Or is the speed not proportional to the number of queries? E.g. when a new event arrives, does Siddhi test it in a linear fashion against each query or does Siddhi keep an internal state machine that tries to match an event against all rules at once? SiddhiManager can have many queries, and if you chain the queries in a liner fashion then all those queries will be executed one after the other and you might see some performance degradation, but if you have have then parallel then there wont be any issues. 2) Is it possible to easily disable/enable some queries? In my use-cases I have a lot of queries. Actually, I have a lot of tenants and each tenant may have something like 10-100 queries. Rather often (e.g. few times a day), tenants would like to disable/enable some of their queries. What is a proper way to do it? Is it a costly operation, i.e. does Siddhi need to perform a lot of processing to disable or enabled a query? Is it better to keep a dedicated SiddhiManager instance per tenant or is it OK to have one SiddhiManager instance which handles all those tenants with all their queries? The general norm is, you have to use a SiddhiManager per scenario, where each scenario might contain one or more queries, with this modal its easy if any tenant want to add a remove a scenario and it will not affect other queries and tenants. 3) What is the semantics of distributed execution? I have found that Siddhi supports it by means of Hazelcast. But what does distributed execution means? E.g. what happens when I feed in an event at one of the instances? How can this distributed execution be controlled besides enabling/disabling it? Currently Siddhi share its states among its different nodes via Hazelcast. We are currently working on alternative ways to improve its distribution capability. We found hazelcast is too slow for distributed processing. We are working on a storm based model, see WSO2 CEP/Siddhi Storm Integration thread some time back. This should come out Q2/Q3. 4) What is the semantics of async processing? I have found setAsyncProcessing method, but what would be the effect of enabling this kind of processing as compared to the usual way of operation? What are the benefits and what are the drawbacks? When should it be used? By default Siddhi uses the request thread to do the event processing, but if you want to handover the data to another thread process then you can enable async processing. Its useful if you are doing very complex time consuming process using Siddhi. If you do this, event are processed by placing them in a queues between any two queries and assigning threads to queries. However, we found often single thread (async model is faster). 5) I figured out that Siddhi can persist its state and later on restore it. This is a cool feature, but I'd like to understand better what kind of information is being persisted. Is it only events whose processing is not finished yet? Does it include a set of queries currently defined in a given SiddhiManager? Does it include all of SiddhiManager's settings? What kind of information is restored and what kind of information should be provided again in addition to restored one? What is a typical situation when Siddhi persistence could/should be used? It only stores the state information of the processing, E.g the current running Avg of the average calculation. This will be used when server recovers from a failure. I hope that most of my questions are pretty simple to answer for those, who are familiar with Siddhi's architecture and its inner workings. Hope these information is very useful. Regards, Suho Thanks, Leo ___ Architecture mailing list Architecture@wso2.org
Re: [Architecture] A few questions about WSO2 CEP/Siddhi
Hi All, these questions and answers are very educating. Shall we add them to our doc FAQs? Thanks, Samisa... Samisa Abeysinghe Vice President Developer Evangelism WSO2 Inc. http://wso2.com On Mon, Mar 10, 2014 at 1:14 PM, Srinath Perera srin...@wso2.com wrote: Hi Leo, Let me add couple of words as well On Mon, Mar 10, 2014 at 9:16 AM, Sriskandarajah Suhothayan s...@wso2.comwrote: Thanks for you interest in WSO2 CEP Siddhi Sorry for the late reply, I some how missed the mail On Fri, Mar 7, 2014 at 4:48 AM, Leo Romanoff romix...@yahoo.com wrote: Hi, I've started playing with WSO2 CEP recently and after some experiments with it I collected a few questions, which I cannot answer by reading the docs. Hopefully I'll have them answered by posting them on this list. First of all, some background information. I have some previous experience with Esper and Drools. Currently, I'm mostly interested in using Siddhi in embedded mode, i.e. as a library. I could built a few small test projects using it without any serious problems. But now I'm trying to understand some more complex things about it. So, here are my questions: 1) How many rules/queries can be defined in one engine. How does it affect performance? For example, can I define (tens of) thousands of queries using the same (or multiple) instance of SiddhiManager? Would it make processing much slower? Or is the speed not proportional to the number of queries? E.g. when a new event arrives, does Siddhi test it in a linear fashion against each query or does Siddhi keep an internal state machine that tries to match an event against all rules at once? SiddhiManager can have many queries, and if you chain the queries in a liner fashion then all those queries will be executed one after the other and you might see some performance degradation, but if you have have then parallel then there wont be any issues. 2) Is it possible to easily disable/enable some queries? In my use-cases I have a lot of queries. Actually, I have a lot of tenants and each tenant may have something like 10-100 queries. Rather often (e.g. few times a day), tenants would like to disable/enable some of their queries. What is a proper way to do it? Is it a costly operation, i.e. does Siddhi need to perform a lot of processing to disable or enabled a query? Is it better to keep a dedicated SiddhiManager instance per tenant or is it OK to have one SiddhiManager instance which handles all those tenants with all their queries? The general norm is, you have to use a SiddhiManager per scenario, where each scenario might contain one or more queries, with this modal its easy if any tenant want to add a remove a scenario and it will not affect other queries and tenants. 3) What is the semantics of distributed execution? I have found that Siddhi supports it by means of Hazelcast. But what does distributed execution means? E.g. what happens when I feed in an event at one of the instances? How can this distributed execution be controlled besides enabling/disabling it? Currently Siddhi share its states among its different nodes via Hazelcast. We are currently working on alternative ways to improve its distribution capability. We found hazelcast is too slow for distributed processing. We are working on a storm based model, see WSO2 CEP/Siddhi Storm Integration thread some time back. This should come out Q2/Q3. 4) What is the semantics of async processing? I have found setAsyncProcessing method, but what would be the effect of enabling this kind of processing as compared to the usual way of operation? What are the benefits and what are the drawbacks? When should it be used? By default Siddhi uses the request thread to do the event processing, but if you want to handover the data to another thread process then you can enable async processing. Its useful if you are doing very complex time consuming process using Siddhi. If you do this, event are processed by placing them in a queues between any two queries and assigning threads to queries. However, we found often single thread (async model is faster). 5) I figured out that Siddhi can persist its state and later on restore it. This is a cool feature, but I'd like to understand better what kind of information is being persisted. Is it only events whose processing is not finished yet? Does it include a set of queries currently defined in a given SiddhiManager? Does it include all of SiddhiManager's settings? What kind of information is restored and what kind of information should be provided again in addition to restored one? What is a typical situation when Siddhi persistence could/should be used? It only stores the state information of the processing, E.g the current running Avg of the average calculation. This will be used when server recovers from a failure. I hope that most of my questions are pretty simple to answer for those, who are
Re: [Architecture] MB 3.0.0 MQTT Integration Status
ah OK. Can we get the introp tests integrated with our product or integration tests? On Mon, Mar 10, 2014 at 2:03 PM, Pamod Sylvester pa...@wso2.com wrote: Hi Srinath, I have a client written to generate the load, will do a performance round and will let know of the figures ASAP. Mean while there's a Interop test suite which has test cases which will validate all protocol level use cases, it can be found in [1]. Currently i am testing the MB against it to ensure that our broker adheres to the spec. Once the validation is complete will do a performance round. Thanks, Pamod [1] http://wiki.eclipse.org/File:Mqtt-test.zip On Mon, Mar 10, 2014 at 1:01 PM, Srinath Perera srin...@wso2.com wrote: Pamod. shall we write some test cases right away .. to test it and load test it. Make sure we do all their QOS scenarios How is publishing and receiving throughputs? we need to measure that Also What is end to end latency? --Srinath On Mon, Mar 10, 2014 at 12:38 PM, Pamod Sylvester pa...@wso2.com wrote: While testing with the Interop test suite the following tests failed with the default parameters (TIMEOUT = 500 ms and CONNECTION_TIME_OUT = 1000 ms). where timeout is the amount of time a subscriber awaits for a message and connection time out is the whole time the connection will be maintained with the broker. Test Basics , Test Long Topic, Test OverLapping Topics, Test Dots , Test Active MQ Wild Cards , Test Native MQTT Wild Cards , Test Subs , Test Wild Card Plus , Test LWT, Test Clean Session further, when the parameters were incremented to (TIMEOUT = 1000 ms and CONNECTION_TIME_OUT = 5000 ms). The failing tests were narrowed down to the following, Test Long Topic, Test OverLapping Topic, Test Active MQ Wild Cards, Test Native MQTT Wild Cards, Test Subs, Test Wild Card Plus, Test LWT, Test Clean Session currently we're in progress of fixing the issues related to cause test failures, also will be looking at the possibilities to reduce the latency of the response time so that the tests will be positive for the default parameters. Thanks, Pamod On Sat, Mar 8, 2014 at 4:58 PM, Pamod Sylvester pa...@wso2.com wrote: Hi Paul, +1 to reflect the changes back to the community. Will get in touch with their mailing list. Also next line item is to test the flows using the Interop test suite. Will get back to you with the results ASAP. Thanks, Pamod On Saturday, March 8, 2014, Paul Fremantle p...@wso2.com wrote: Pamod The ideal situation is to aim to get these changes back into Moquette. That means (a) getting them into a shape where they are generically useful to the Moquette community - so that other people wishing to embed Moquette would also look at this and say yes please, and (b) persuading the Moquette group this is a valuable change/addition. Is there any chance of doing an interop test with a beta on the 17th March at EclipseCon? I'll be at the interop day. Paul On 8 March 2014 05:36, Pamod Sylvester pa...@wso2.com wrote: Hi Samisa, Yes we did changes which are local to the Moquette library, However, the changes were done in a separate class to act as an interface between the Andes Kernal and the MQTT library. All the changes which are specific for the kernel integration are embedded in that class, so that when the library updates with a minimum effort we could adopt to the changes. However, it was also considered to totally abstract the library without having us modify any of its content but rather call only its interfaces. Then the challenge was to have a way to resolve circular dependencies. Where, to start the MQTT server MQTT library will depend on Andes and for message exchange Andes will depend on MQTT library. Also, IMO in order to maintain patches which we also need to have the library source code in a separate branch and maintain the way we do for Synapse. Please do correct me if i am wrong. Thanks, Pamod On Sat, Mar 8, 2014 at 10:35 AM, Samisa Abeysinghe sam...@wso2.com wrote: On Sat, Mar 8, 2014 at 10:00 AM, Pamod Sylvester pa...@wso2.com wrote: Hi All, The following is the status of the MQTT Moquette library integration with MB 3.0.0, ~ Library was revamped to fit in with the Andes kernel. Have we done local changes in this re-vamp? How do we plan to maintain those if yes? Attached diagram depicts how the messages are flowed between the components at a high level. ~ Implemented a MQTT client for subscription and publishing. ~ Also the Moquette library was patched, the default behaviour of the library did not support multiple subscribers to receive a published message. How do we plan to maintain these patches? That was due to a bug which had a common ByteBuffer shared between concurrently accessing subscriber connections. The issue was fixed by cloning ByteBuffer instances per subscriber. Note that the wrapped array element was not
Re: [Architecture] MB 3.0.0 MQTT Integration Status
Yes. They are Junit test cases. We could integrate them. Thanks, Pamod On Mon, Mar 10, 2014 at 2:15 PM, Srinath Perera srin...@wso2.com wrote: ah OK. Can we get the introp tests integrated with our product or integration tests? On Mon, Mar 10, 2014 at 2:03 PM, Pamod Sylvester pa...@wso2.com wrote: Hi Srinath, I have a client written to generate the load, will do a performance round and will let know of the figures ASAP. Mean while there's a Interop test suite which has test cases which will validate all protocol level use cases, it can be found in [1]. Currently i am testing the MB against it to ensure that our broker adheres to the spec. Once the validation is complete will do a performance round. Thanks, Pamod [1] http://wiki.eclipse.org/File:Mqtt-test.zip On Mon, Mar 10, 2014 at 1:01 PM, Srinath Perera srin...@wso2.com wrote: Pamod. shall we write some test cases right away .. to test it and load test it. Make sure we do all their QOS scenarios How is publishing and receiving throughputs? we need to measure that Also What is end to end latency? --Srinath On Mon, Mar 10, 2014 at 12:38 PM, Pamod Sylvester pa...@wso2.comwrote: While testing with the Interop test suite the following tests failed with the default parameters (TIMEOUT = 500 ms and CONNECTION_TIME_OUT = 1000 ms). where timeout is the amount of time a subscriber awaits for a message and connection time out is the whole time the connection will be maintained with the broker. Test Basics , Test Long Topic, Test OverLapping Topics, Test Dots , Test Active MQ Wild Cards , Test Native MQTT Wild Cards , Test Subs , Test Wild Card Plus , Test LWT, Test Clean Session further, when the parameters were incremented to (TIMEOUT = 1000 ms and CONNECTION_TIME_OUT = 5000 ms). The failing tests were narrowed down to the following, Test Long Topic, Test OverLapping Topic, Test Active MQ Wild Cards, Test Native MQTT Wild Cards, Test Subs, Test Wild Card Plus, Test LWT, Test Clean Session currently we're in progress of fixing the issues related to cause test failures, also will be looking at the possibilities to reduce the latency of the response time so that the tests will be positive for the default parameters. Thanks, Pamod On Sat, Mar 8, 2014 at 4:58 PM, Pamod Sylvester pa...@wso2.com wrote: Hi Paul, +1 to reflect the changes back to the community. Will get in touch with their mailing list. Also next line item is to test the flows using the Interop test suite. Will get back to you with the results ASAP. Thanks, Pamod On Saturday, March 8, 2014, Paul Fremantle p...@wso2.com wrote: Pamod The ideal situation is to aim to get these changes back into Moquette. That means (a) getting them into a shape where they are generically useful to the Moquette community - so that other people wishing to embed Moquette would also look at this and say yes please, and (b) persuading the Moquette group this is a valuable change/addition. Is there any chance of doing an interop test with a beta on the 17th March at EclipseCon? I'll be at the interop day. Paul On 8 March 2014 05:36, Pamod Sylvester pa...@wso2.com wrote: Hi Samisa, Yes we did changes which are local to the Moquette library, However, the changes were done in a separate class to act as an interface between the Andes Kernal and the MQTT library. All the changes which are specific for the kernel integration are embedded in that class, so that when the library updates with a minimum effort we could adopt to the changes. However, it was also considered to totally abstract the library without having us modify any of its content but rather call only its interfaces. Then the challenge was to have a way to resolve circular dependencies. Where, to start the MQTT server MQTT library will depend on Andes and for message exchange Andes will depend on MQTT library. Also, IMO in order to maintain patches which we also need to have the library source code in a separate branch and maintain the way we do for Synapse. Please do correct me if i am wrong. Thanks, Pamod On Sat, Mar 8, 2014 at 10:35 AM, Samisa Abeysinghe sam...@wso2.com wrote: On Sat, Mar 8, 2014 at 10:00 AM, Pamod Sylvester pa...@wso2.com wrote: Hi All, The following is the status of the MQTT Moquette library integration with MB 3.0.0, ~ Library was revamped to fit in with the Andes kernel. Have we done local changes in this re-vamp? How do we plan to maintain those if yes? Attached diagram depicts how the messages are flowed between the components at a high level. ~ Implemented a MQTT client for subscription and publishing. ~ Also the Moquette library was patched, the default behaviour of the library did not support multiple subscribers to receive a published message. How do we plan to maintain these patches? That was due to a bug which had a common ByteBuffer shared between concurrently
Re: [Architecture] A few questions about WSO2 CEP/Siddhi
Hi all, First of all, thank you very much for your explanations and clarifications! It is very interesting and useful! Let me ask a few more questions and provide a few comments. Hi All, these questions and answers are very educating. Shall we add them to our doc FAQs? I think it would be a very good idea to add something like this to the FAQs or to create some sort of an architecture and implementation overview document. 1) How many rules/queries can be defined in one engine. How does it affect performance? For example, can I define (tens of) thousands of queries using the same (or multiple) instance of SiddhiManager? Would it make processing much slower? Or is the speed not proportional to the number of queries? E.g. when a new event arrives, does Siddhi test it in a linear fashion against each query or does Siddhi keep an internal state machine that tries to match an event against all rules at once? SiddhiManager can have many queries, and if you chain the queries in a liner fashion then all those queries will be executed one after the other and you might see some performance degradation, but if you have have then parallel then there wont be any issues. Well, before I got this answer, I created a few test-cases to check experimentally how it behaves. I created a single instance of a SiddhiManager, added 1 queries that all read from the same input stream, check if a specific attribute (namely, price) of an event is inside a given random interval ( [ price = random_low and price = random_high] ) and output into randomly into one of 100 streams. Then I measured the time required to process 100 events using this setup. I also did exactly the same experiment with Esper. My findings were that Siddhi is much slower than Esper in this setup. After looking into the internal implementations of both, I realized the reason. Siddhi processes all queries that read from the same input stream in a linear fashion, sequentially. Even if many of the queries have almost the same condition, no optimization attempts are done by Siddhi. Esper detects that many queries have a condition on the same variable and create some sort of a decision tree. As a result, their running time in log N, where as Siddhi needs O(n). I'm not saying that this test-case if very typical or important, but may be Siddhi should try to analyze the complete set of queries and try to apply some optimizations, when it is possible? I.e. it is a bit of a global optimization applied. It could detect some common sub-expressions or sub-conditions in the queries and evaluate them only once, instead of doing it over and over again by evaluating each query separately. After getting these first results, I changed the setup, so that each query uses one of many input streams (e.g. one of 300) instead of using the same one. This greatly improved the situation, because now the number of queries per input stream was much smaller and thus processing was way faster. But even in this setup it is still about 5-6 times slower than Esper in this situation. 2) Is it possible to easily disable/enable some queries? In my use-cases I have a lot of queries. Actually, I have a lot of tenants and each tenant may have something like 10-100 queries. Rather often (e.g. few times a day), tenants would like to disable/enable some of their queries. What is a proper way to do it? Is it a costly operation, i.e. does Siddhi need to perform a lot of processing to disable or enabled a query? Is it better to keep a dedicated SiddhiManager instance per tenant or is it OK to have one SiddhiManager instance which handles all those tenants with all their queries? The general norm is, you have to use a SiddhiManager per scenario, where each scenario might contain one or more queries, with this modal its easy if any tenant want to add a remove a scenario and it will not affect other queries and tenants. If I have tens of thousands of tenants, then having a dedicated SiddhiManager per tenant is probably not very practical or even possible, as it will get pretty heave weight, I guess. Therefore, having the ability to enable/disable to query could be very practical. In fact, it could be probably implemented very easily. Imagine that each query object has a boolean flag that indicates if it is enabled or not. If the condition matches and before Siddhi tries to perform the insert, i.e. the action, it could check if the query is disabled. If it is disabled, no action (i.e. insert) is performed at all. Of course, there is still some overhead when matching the query. But may be even this can be skipped if query is disabled? I.e. conditions are immediately evaluated to false and thus never trigger? BTW, Esper has this feature. You can disable/enable any query without removing and later adding it again. When it comes to Siddhi persistent stores, you write: It only stores the state information of the
Re: [Architecture] SSO IDP Proxy Application + SDK
Hi Gayan, Here the IDP proxy app is only used to get the authorization code from the WSO2 IS and pass it to the SDK. After which the SDK is communicates directly with the WSO2 IS to get the access token and manage the access token and refresh token. Just a small clarification why we can't use the IDP proxy app to do this, .i.e, let the IDP proxy app manage the access token and refresh token for each app. Therefore cutting off the connection between the SDK and the WSO2 IS. Here if the access token expires then the SDK will call the IDP proxy app to get the token refreshed. On Mon, Mar 10, 2014 at 3:58 PM, Gayan Gunawardana ga...@wso2.com wrote: Image attached On Mon, Mar 10, 2014 at 3:51 PM, Gayan Gunawardana ga...@wso2.com wrote: Hi All, Problem: Implement SSO for enterprise mobile apps The idea is to provide SDK for mobile apps developers within the organization, then they can integrate SDK inside the application and implement SSO across required applications. Provide (SDK + Mobile IDP proxy app) To achieve above purpose we plan to utilize oauth 2.0 with *Authorization code* grant type. Briefly Explaining message flow : Initially new application has to be registered in WSO2 IS under Oauth management and obtain client_key, client_secret, Access Token Url and Authorize Url 1. SDK initiate the process by sending client_key, redirect_url and scope to mobile IDP proxy app 2. IDP proxy app obtain Authorization code 3. SDK (in side mobile app) receive Authorization code 4. SDK send second request directly to WSO2 IS with Authorization code, client secret and redirect_url 5. SDK obtain access token 6. Mobile app pass access token to resource server 7. Resource server contact IPD and validate access token This is much similar to Facebook approach where facebook application act as mobile IDP proxy app and they provide SDK to develop apps. All your suggestions are welcome. -- Gayan Gunawardana Software Engineer; WSO2 Inc.; http://wso2.com/ Email: ga...@wso2.com Mobile: +94 (71) 8020933 Blog: http://gayanj2ee.blogspot.com/ -- Gayan Gunawardana Software Engineer; WSO2 Inc.; http://wso2.com/ Email: ga...@wso2.com Mobile: +94 (71) 8020933 Blog: http://gayanj2ee.blogspot.com/ ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- *Niranjan Karunanandham* Senior Software Engineer - WSO2 Inc. WSO2 Inc.: http://www.wso2.com M: +94 777 749 661 http:/// ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] A few questions about WSO2 CEP/Siddhi
One more thing: It could be very, very useful, if Siddhi/WSO2 CEP would have a description/document like this: http://esper.codehaus.org/tutorials/solution_patterns/solution_patterns.html This document covers many generic CEP use-cases and then shows how they can be solved using Esper. May be Siddhi could take this document as a basis and provide a Siddhi-specific version of it reflecting which of described CEP use-cases could be expressed/implemented with Siddhi and how? Also stating what is not possible currently or what is planned could be very useful. I think it would be very valuable for users, because right now due to lack of information it is pretty difficult to figure out what is possible to do with Siddhi and what are its current limitations. It could be also useful for Siddhi developers, because it may highlight certain features that are missing currently, but are essential for implementing certain classes of scenarios. I hope this proposal makes sense. Best Regards, -Leo Leo Romanoff romix...@yahoo.com schrieb am 11:23 Montag, 10.März 2014: Hi all, First of all, thank you very much for your explanations and clarifications! It is very interesting and useful! Let me ask a few more questions and provide a few comments. Hi All, these questions and answers are very educating. Shall we add them to our doc FAQs? I think it would be a very good idea to add something like this to the FAQs or to create some sort of an architecture and implementation overview document. 1) How many rules/queries can be defined in one engine. How does it affect performance? For example, can I define (tens of) thousands of queries using the same (or multiple) instance of SiddhiManager? Would it make processing much slower? Or is the speed not proportional to the number of queries? E.g. when a new event arrives, does Siddhi test it in a linear fashion against each query or does Siddhi keep an internal state machine that tries to match an event against all rules at once? SiddhiManager can have many queries, and if you chain the queries in a liner fashion then all those queries will be executed one after the other and you might see some performance degradation, but if you have have then parallel then there wont be any issues. Well, before I got this answer, I created a few test-cases to check experimentally how it behaves. I created a single instance of a SiddhiManager, added 1 queries that all read from the same input stream, check if a specific attribute (namely, price) of an event is inside a given random interval ( [ price = random_low and price = random_high] ) and output into randomly into one of 100 streams. Then I measured the time required to process 100 events using this setup. I also did exactly the same experiment with Esper. My findings were that Siddhi is much slower than Esper in this setup. After looking into the internal implementations of both, I realized the reason. Siddhi processes all queries that read from the same input stream in a linear fashion, sequentially. Even if many of the queries have almost the same condition, no optimization attempts are done by Siddhi. Esper detects that many queries have a condition on the same variable and create some sort of a decision tree. As a result, their running time in log N, where as Siddhi needs O(n). I'm not saying that this test-case if very typical or important, but may be Siddhi should try to analyze the complete set of queries and try to apply some optimizations, when it is possible? I.e. it is a bit of a global optimization applied. It could detect some common sub-expressions or sub-conditions in the queries and evaluate them only once, instead of doing it over and over again by evaluating each query separately. After getting these first results, I changed the setup, so that each query uses one of many input streams (e.g. one of 300) instead of using the same one. This greatly improved the situation, because now the number of queries per input stream was much smaller and thus processing was way faster. But even in this setup it is still about 5-6 times slower than Esper in this situation. 2) Is it possible to easily disable/enable some queries? In my use-cases I have a lot of queries. Actually, I have a lot of tenants and each tenant may have something like 10-100 queries. Rather often (e.g. few times a day), tenants would like to disable/enable some of their queries. What is a proper way to do it? Is it a costly operation, i.e. does Siddhi need to perform a lot of processing to disable or enabled a query? Is it better to keep a dedicated SiddhiManager instance per tenant or is it OK to have one SiddhiManager instance which handles all those tenants with all their queries? The general norm is, you have to use a SiddhiManager per scenario, where each scenario might contain one or more queries, with this modal its easy if any tenant
Re: [Architecture] SSO IDP Proxy Application + SDK
Hi all, How do we store client secret and access tokens in mobile application? Have we encrypted the client secret? In case of mobile device is lost, how do we remove the mobile application subscription from OAuth server without affecting to other mobile devices which uses same application? Do we generate the applicationId together with a unique mobile Id? Is the mobile IDP app code signed by a trusted cert? How does the trust relationship works with mobile IDP and WSO2IS? thank you. On Mon, Mar 10, 2014 at 4:37 PM, Gayan Gunawardana ga...@wso2.com wrote: Hi Nira, Reason to do that way is normally client secret does not share with any other party On Mon, Mar 10, 2014 at 4:24 PM, Niranjan Karunanandham niran...@wso2.com wrote: Hi Gayan, Here the IDP proxy app is only used to get the authorization code from the WSO2 IS and pass it to the SDK. After which the SDK is communicates directly with the WSO2 IS to get the access token and manage the access token and refresh token. Just a small clarification why we can't use the IDP proxy app to do this, .i.e, let the IDP proxy app manage the access token and refresh token for each app. Therefore cutting off the connection between the SDK and the WSO2 IS. Here if the access token expires then the SDK will call the IDP proxy app to get the token refreshed. On Mon, Mar 10, 2014 at 3:58 PM, Gayan Gunawardana ga...@wso2.comwrote: Image attached On Mon, Mar 10, 2014 at 3:51 PM, Gayan Gunawardana ga...@wso2.comwrote: Hi All, Problem: Implement SSO for enterprise mobile apps The idea is to provide SDK for mobile apps developers within the organization, then they can integrate SDK inside the application and implement SSO across required applications. Provide (SDK + Mobile IDP proxy app) To achieve above purpose we plan to utilize oauth 2.0 with *Authorization code* grant type. Briefly Explaining message flow : Initially new application has to be registered in WSO2 IS under Oauth management and obtain client_key, client_secret, Access Token Url and Authorize Url 1. SDK initiate the process by sending client_key, redirect_url and scope to mobile IDP proxy app 2. IDP proxy app obtain Authorization code 3. SDK (in side mobile app) receive Authorization code 4. SDK send second request directly to WSO2 IS with Authorization code, client secret and redirect_url 5. SDK obtain access token 6. Mobile app pass access token to resource server 7. Resource server contact IPD and validate access token This is much similar to Facebook approach where facebook application act as mobile IDP proxy app and they provide SDK to develop apps. All your suggestions are welcome. -- Gayan Gunawardana Software Engineer; WSO2 Inc.; http://wso2.com/ Email: ga...@wso2.com Mobile: +94 (71) 8020933 Blog: http://gayanj2ee.blogspot.com/ -- Gayan Gunawardana Software Engineer; WSO2 Inc.; http://wso2.com/ Email: ga...@wso2.com Mobile: +94 (71) 8020933 Blog: http://gayanj2ee.blogspot.com/ ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- *Niranjan Karunanandham* Senior Software Engineer - WSO2 Inc. WSO2 Inc.: http://www.wso2.com M: +94 777 749 661 http:/// ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- Gayan Gunawardana Software Engineer; WSO2 Inc.; http://wso2.com/ Email: ga...@wso2.com Mobile: +94 (71) 8020933 Blog: http://gayanj2ee.blogspot.com/ ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- Manjula Rathnayaka Software Engineer WSO2, Inc. Mobile:+94 77 743 1987 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
[Architecture] Fwd: [Dev] Moving Workflow configurations to Registry
Moving to Architecture Hi, Currently the configuration for Workflow Executors are kept in the api-manager.xml due to which all the tenants living in a APIM Deployment have to use the same executors defined by the Super Tenant Admin. To allow Tenants configure their own executors, Workflow Configurations will be moved to the Registry. Following places will be changed when accommodating this; 1. With the existing implementation, Workflow Executor Factory reads the config and maintains a map for different executors needed by each Workflow Type. This factory will be changed to hold Workflows defined by each different tenant. 2. At present, the configuration is read only at the start up. When moving those to the registry, users will be able to change them at any moment and the data structure created out of the configs will have to be updated reflecting the changes. This will be done using a Registry Handler. 3. To reduce frequent accesses of the registry resource, the data structure created out by parsing the config will be cached. The new registry handler which gets called upon updating the config will be responsible for invalidating the cache. Comments are welcome. -- *Amila De Silva* *Software Engineer* WSO2 Inc. mobile :(+94) 775119302 -- *Amila De Silva* *Software Engineer* WSO2 Inc. mobile :(+94) 775119302 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] SSO IDP Proxy Application + SDK
Hi Manjula, Let me answer inline, On Mon, Mar 10, 2014 at 4:54 PM, Manjula Rathnayake manju...@wso2.comwrote: Hi all, How do we store client secret and access tokens in mobile application? Have we encrypted the client secret? We can let the mobile app developer to implement his own mechanism for this, or if we are supporting this at the SDK, we can use a password to encrypt the client secrete. In case of mobile device is lost, how do we remove the mobile application subscription from OAuth server without affecting to other mobile devices which uses same application? Do we generate the applicationId together with a unique mobile Id? User can always revoke the tokens issued for the application. We can let each application to have its own client-key, client-secrete as well using dynamic client registration. Is the mobile IDP app code signed by a trusted cert? How does the trust relationship works with mobile IDP and WSO2IS? WSO2IS does not have to trust the proxy IDP in the mobile. IS will always validate client-key, client-secrete and will check user authentication at logins. thank you. On Mon, Mar 10, 2014 at 4:37 PM, Gayan Gunawardana ga...@wso2.com wrote: Hi Nira, Reason to do that way is normally client secret does not share with any other party On Mon, Mar 10, 2014 at 4:24 PM, Niranjan Karunanandham niran...@wso2.com wrote: Hi Gayan, Here the IDP proxy app is only used to get the authorization code from the WSO2 IS and pass it to the SDK. After which the SDK is communicates directly with the WSO2 IS to get the access token and manage the access token and refresh token. Just a small clarification why we can't use the IDP proxy app to do this, .i.e, let the IDP proxy app manage the access token and refresh token for each app. Therefore cutting off the connection between the SDK and the WSO2 IS. Here if the access token expires then the SDK will call the IDP proxy app to get the token refreshed. On Mon, Mar 10, 2014 at 3:58 PM, Gayan Gunawardana ga...@wso2.comwrote: Image attached On Mon, Mar 10, 2014 at 3:51 PM, Gayan Gunawardana ga...@wso2.comwrote: Hi All, Problem: Implement SSO for enterprise mobile apps The idea is to provide SDK for mobile apps developers within the organization, then they can integrate SDK inside the application and implement SSO across required applications. Provide (SDK + Mobile IDP proxy app) To achieve above purpose we plan to utilize oauth 2.0 with *Authorization code* grant type. Briefly Explaining message flow : Initially new application has to be registered in WSO2 IS under Oauth management and obtain client_key, client_secret, Access Token Url and Authorize Url 1. SDK initiate the process by sending client_key, redirect_url and scope to mobile IDP proxy app 2. IDP proxy app obtain Authorization code 3. SDK (in side mobile app) receive Authorization code 4. SDK send second request directly to WSO2 IS with Authorization code, client secret and redirect_url 5. SDK obtain access token 6. Mobile app pass access token to resource server 7. Resource server contact IPD and validate access token This is much similar to Facebook approach where facebook application act as mobile IDP proxy app and they provide SDK to develop apps. All your suggestions are welcome. -- Gayan Gunawardana Software Engineer; WSO2 Inc.; http://wso2.com/ Email: ga...@wso2.com Mobile: +94 (71) 8020933 Blog: http://gayanj2ee.blogspot.com/ -- Gayan Gunawardana Software Engineer; WSO2 Inc.; http://wso2.com/ Email: ga...@wso2.com Mobile: +94 (71) 8020933 Blog: http://gayanj2ee.blogspot.com/ ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- *Niranjan Karunanandham* Senior Software Engineer - WSO2 Inc. WSO2 Inc.: http://www.wso2.com M: +94 777 749 661 http:/// ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- Gayan Gunawardana Software Engineer; WSO2 Inc.; http://wso2.com/ Email: ga...@wso2.com Mobile: +94 (71) 8020933 Blog: http://gayanj2ee.blogspot.com/ ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- Manjula Rathnayaka Software Engineer WSO2, Inc. Mobile:+94 77 743 1987 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- Suresh Attanayake Senior Software Engineer; WSO2 Inc. http://wso2.com/ Blog : http://sureshatt.blogspot.com/ Web : http://www.ssoarcade.com/ Facebook : https://www.facebook.com/IdentityWorld Twitter : https://twitter.com/sureshatt LinkedIn : http://lk.linkedin.com/in/sureshatt Mobile :
[Architecture] [C5] Jetty integration with carbon kernel
We have done $subject which can be used for servlet transport in C5 kernel [1]. The embedded jetty instance uses a default jetty.xml configuration file located at $CARBON_HOME/repository/conf/jetty/jetty.xml. The main advantage of jetty is that it is OSGi friendly [2]. It can also expose the OSGi HttpSevice for servlet registrations, etc. We are thinking of using this for the UI framework of carbon. Other servlet container can also be plugged in (Eg: Tomcat with AS) using the plugable run-time concept, but they will use different ports to avoid collisions. Thanks, Kishanthan. [1] https://wso2.org/jira/browse/CARBON-14716 [2] http://wiki.eclipse.org/Jetty/Feature/Jetty_OSGi -- *Kishanthan Thangarajah* Senior Software Engineer, Platform Technologies Team, WSO2, Inc. lean.enterprise.middleware Mobile - +94773426635 Blog - *http://kishanthan.wordpress.com http://kishanthan.wordpress.com* Twitter - *http://twitter.com/kishanthan http://twitter.com/kishanthan* ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] G-Reg vs ES for artefacts
Hi Guys, Will these limitations get fixed in the upcoming release? If so do we have a timeline for that? Thanks, Janaka On Sat, Mar 8, 2014 at 10:43 AM, Chandana Napagoda chand...@wso2.comwrote: Hi Samisa In addition to above issue, ES is still not supporting all the RXT configuration elements(ex: unbounded table option) which are currently supported by G-Reg. We have reported JIRAs to get those added to coming up releases. Regards, Chandana On Mar 8, 2014 7:09 AM, Janaka Ranabahu jan...@wso2.com wrote: Hi Samisa, IMO, it should be ES. But the current ES architecture does not pick new RXT types automatically and some configurations to jaggery config files are required to show the new asset in both Store and Publisher[1]. Until we have a more flexible way of generating the Store and Publisher UI, I believe we have to stick with G-Reg. Thanks, Janaka [1] https://docs.wso2.org/display/ES100/Adding+a+New+Asset+Type On Fri, Mar 7, 2014 at 8:08 PM, Samisa Abeysinghe sam...@wso2.comwrote: We used to position G-Reg as a store of anything with RXT. Now what ES is there, and is more capable, do we propose the users use ES or G-Reg RXT like use cases? Thanks, Samisa... Samisa Abeysinghe Vice President Developer Evangelism WSO2 Inc. http://wso2.com ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- *Janaka Ranabahu* Senior Software Engineer; WSO2 Inc.; http://wso2.com * E-mail: jan...@wso2.com http://wso2.com**M: **+94 718370861 %2B94%20718370861* Lean . Enterprise . Middleware ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- *Janaka Ranabahu* Senior Software Engineer; WSO2 Inc.; http://wso2.com * E-mail: jan...@wso2.com http://wso2.com**M: **+94 718370861* Lean . Enterprise . Middleware ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] A few questions about WSO2 CEP/Siddhi
Hi Leo, Please see my comments inline. First of all, thank you very much for your explanations and clarifications! It is very interesting and useful! Let me ask a few more questions and provide a few comments. Hi All, these questions and answers are very educating. Shall we add them to our doc FAQs? I think it would be a very good idea to add something like this to the FAQs or to create some sort of an architecture and implementation overview document. 1) How many rules/queries can be defined in one engine. How does it affect performance? For example, can I define (tens of) thousands of queries using the same (or multiple) instance of SiddhiManager? Would it make processing much slower? Or is the speed not proportional to the number of queries? E.g. when a new event arrives, does Siddhi test it in a linear fashion against each query or does Siddhi keep an internal state machine that tries to match an event against all rules at once? SiddhiManager can have many queries, and if you chain the queries in a liner fashion then all those queries will be executed one after the other and you might see some performance degradation, but if you have have then parallel then there wont be any issues. Well, before I got this answer, I created a few test-cases to check experimentally how it behaves. I created a single instance of a SiddhiManager, added 1 queries that all read from the same input stream, check if a specific attribute (namely, price) of an event is inside a given random interval ( [ price = random_low and price = random_high] ) and output into randomly into one of 100 streams. Then I measured the time required to process 100 events using this setup. I also did exactly the same experiment with Esper. My findings were that Siddhi is much slower than Esper in this setup. After looking into the internal implementations of both, I realized the reason. Siddhi processes all queries that read from the same input stream in a linear fashion, sequentially. Even if many of the queries have almost the same condition, no optimization attempts are done by Siddhi. Esper detects that many queries have a condition on the same variable and create some sort of a decision tree. As a result, their running time in log N, where as Siddhi needs O(n). I'm not saying that this test-case if very typical or important, but may be Siddhi should try to analyze the complete set of queries and try to apply some optimizations, when it is possible? I.e. it is a bit of a global optimization applied. It could detect some common sub-expressions or sub-conditions in the queries and evaluate them only once, instead of doing it over and over again by evaluating each query separately. After getting these first results, I changed the setup, so that each query uses one of many input streams (e.g. one of 300) instead of using the same one. This greatly improved the situation, because now the number of queries per input stream was much smaller and thus processing was way faster. But even in this setup it is still about 5-6 times slower than Esper in this situation. Could you share your testcases?, and we can have a look. Yes we have not much worked with 1000s of queries much, but likely it is something we can fix without much trouble. 2) Is it possible to easily disable/enable some queries? In my use-cases I have a lot of queries. Actually, I have a lot of tenants and each tenant may have something like 10-100 queries. Rather often (e.g. few times a day), tenants would like to disable/enable some of their queries. What is a proper way to do it? Is it a costly operation, i.e. does Siddhi need to perform a lot of processing to disable or enabled a query? Is it better to keep a dedicated SiddhiManager instance per tenant or is it OK to have one SiddhiManager instance which handles all those tenants with all their queries? The general norm is, you have to use a SiddhiManager per scenario, where each scenario might contain one or more queries, with this modal its easy if any tenant want to add a remove a scenario and it will not affect other queries and tenants. If I have tens of thousands of tenants, then having a dedicated SiddhiManager per tenant is probably not very practical or even possible, as it will get pretty heave weight, I guess. Therefore, having the ability to enable/disable to query could be very practical. In fact, it could be probably implemented very easily. Imagine that each query object has a boolean flag that indicates if it is enabled or not. If the condition matches and before Siddhi tries to perform the insert, i.e. the action, it could check if the query is disabled. If it is disabled, no action (i.e. insert) is performed at all. Of course, there is still some overhead when matching the query. But may be even this can be skipped if query is disabled? I.e. conditions are immediately evaluated to false and thus never
Re: [Architecture] WSO2 Products Tenant Creation Process calls the Reset Password
Any ideas on this? Thanks Regards Danushka Fernando Software Engineer WSO2 inc. http://wso2.com/ Mobile : +94716332729 On Wed, Mar 5, 2014 at 7:38 PM, Danushka Fernando danush...@wso2.comwrote: Hi all I found that our tenant creation process is calling reset password call inside tenant creation process. When we call tenant creation it goes through *persistTenant* call in *TenantPersistor* class. And it calls *persistTenantInUserStore*. In the end of this call it calls for *updateTenantAdminPassword*. By the time Tenant Manager is created the tenant admin and have added the password to the LDAP. So is there a particular reason that we should do this? I cant see any reason that we call the update/reset password at this moment. So IMO we should remove this if no such reason. WDYT? Thanks Regards Danushka Fernando Software Engineer WSO2 inc. http://wso2.com/ Mobile : +94716332729 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] WSO2 Products Tenant Creation Process calls the Reset Password
Hi Danushka, I can not recall why this is done. But we might need to consider other tenant manager implementations as well not only the LDAP based one. AFAIR, we create the tenant admin with a dummy password and then later reset it after hashing the password with hashing algorithm defined in user-mgt.xml. If you have tested this functionality, then we can remove password reset. thank you. On Tue, Mar 11, 2014 at 10:22 AM, Danushka Fernando danush...@wso2.comwrote: Any ideas on this? Thanks Regards Danushka Fernando Software Engineer WSO2 inc. http://wso2.com/ Mobile : +94716332729 On Wed, Mar 5, 2014 at 7:38 PM, Danushka Fernando danush...@wso2.comwrote: Hi all I found that our tenant creation process is calling reset password call inside tenant creation process. When we call tenant creation it goes through *persistTenant* call in *TenantPersistor* class. And it calls *persistTenantInUserStore*. In the end of this call it calls for *updateTenantAdminPassword*. By the time Tenant Manager is created the tenant admin and have added the password to the LDAP. So is there a particular reason that we should do this? I cant see any reason that we call the update/reset password at this moment. So IMO we should remove this if no such reason. WDYT? Thanks Regards Danushka Fernando Software Engineer WSO2 inc. http://wso2.com/ Mobile : +94716332729 -- Manjula Rathnayaka Software Engineer WSO2, Inc. Mobile:+94 77 743 1987 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] WSO2 Products Tenant Creation Process calls the Reset Password
What are the issues of having this? On Thu, Mar 6, 2014 at 9:08 AM, Danushka Fernando danush...@wso2.comwrote: Hi all I found that our tenant creation process is calling reset password call inside tenant creation process. When we call tenant creation it goes through *persistTenant* call in *TenantPersistor* class. And it calls *persistTenantInUserStore*. In the end of this call it calls for *updateTenantAdminPassword*. By the time Tenant Manager is created the tenant admin and have added the password to the LDAP. So is there a particular reason that we should do this? I cant see any reason that we call the update/reset password at this moment. So IMO we should remove this if no such reason. WDYT? Thanks Regards Danushka Fernando Software Engineer WSO2 inc. http://wso2.com/ Mobile : +94716332729 -- *Amila Maharachchi* Senior Technical Lead WSO2, Inc.; http://wso2.com Blog: http://maharachchi.blogspot.com Mobile: +94719371446 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
