from:"Pumudu Ruhunage"

Re: [Dev] [MB] ValueEvent exception occurred on disruptor for MQTT

2017-04-28 Thread Pumudu Ruhunage

Hi Sumedha,

As pamod mention this is a generic exception[1][2][3] which requires more
info to identify the exact cause.
We had long running test(mixed QOS test) for around 5 days(120h+) and
during that we didn't observe this particular issue as well.
Further, we have done several comparisons tests[4] and added few
integration mqtt load tests[5] to verify mqtt stability.

[1] https://wso2.org/jira/browse/MB-1211
[2] https://wso2.org/jira/browse/MB-1837
[3] https://wso2.org/jira/browse/MB-1328

[4]
https://docs.google.com/a/wso2.com/spreadsheets/d/1C50DqgBqXB8_-SND_YJpmXE-ArhxN8Fbg6-K0kQAZZc/edit?usp=sharing
[5]
https://github.com/wso2/product-mb/tree/v3.1.0/modules/integration/tests-integration/tests-mqtt/src/test/java/org/wso2/mb/integration/tests/mqtt/load

Thanks,

On Thu, Apr 27, 2017 at 11:45 PM, Pamod Sylvester <pa...@wso2.com> wrote:

> Hi Sumedha,
>
> With the amount of information available it would be hard to conclude
> where the error occurs.
>
> Also it would be ideal if there's a way to get disruptor to pin-point the
> exact location of the Nullpointer when it occurs in the stack trace in it's
> log exception handler. Will update this thread if i find any.
>
> Let's also make sure to monitor the packets exchanged between the clients
> since this issue occurs during the protocol handling and converting the
> bytes[] to a message, so that we could identify whether it's due to a
> result of malformed packet etc just to make sure.
>
> @Pumudu please add your thoughts. Also could you share the details of the
> long running tests we did (the type of tests we carried out etc)
>
> Thanks,
> Pamod
>
> On Thu, Apr 27, 2017 at 9:51 PM, Sumedha Rubasinghe <sume...@wso2.com>
> wrote:
>
>> Pamod,
>> What is the most longest running MB deployment we have?
>>
>> Is this case, we have had 8 devices pushing data to MB several time
>> within a minute and a DAS listener picking them up. We had this setup
>> running for more than 1 month.
>>
>> On Thu, Apr 27, 2017 at 8:18 PM, Pamod Sylvester <pa...@wso2.com> wrote:
>>
>>> We've not come across this issue before.
>>>
>>> it seems, this has occurred in one of the disruptor handlers in
>>> Moquette. Shall we enable debug/tracing logs in Moquette side to further
>>> drill down the issue,
>>>
>>> The following logs will be useful,
>>>
>>>  1. org.dna.mqtt.moquette.messaging.spi.impl.SimpleMessaging
>>>  2. org.dna.mqtt.moquette.messaging.spi.impl.ProtocolProcessor
>>>
>>> Thanks,
>>> Pamod
>>>
>>> On Thu, Apr 27, 2017 at 6:59 PM, Sumedha Rubasinghe <sume...@wso2.com>
>>> wrote:
>>>
>>>> MB guys,
>>>> You can consider this as a real production issue. We have been running
>>>> the server for few weeks now.
>>>>
>>>> On Thu, Apr 27, 2017 at 4:22 PM, Ayyoob Hamza <ayy...@wso2.com> wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> We did a long running test for a couple of weeks and in MB we see
>>>>> following logs getting printed continuously and devices are failing to
>>>>> publish data. Is this a known issue?
>>>>>
>>>>> TID: [] [] [2017-04-27 16:03:21,390] ERROR
>>>>> {org.dna.mqtt.wso2.MQTTLogExceptionHandler} -  ValueEvent exception
>>>>> occurred on disruptor. {org.dna.mqtt.wso2.MQTTLogExceptionHandler}
>>>>> java.lang.NullPointerException
>>>>>
>>>>>
>>>>> Thanks,
>>>>> *Ayyoob Hamza*
>>>>> *Senior Software Engineer*
>>>>> WSO2 Inc.; http://wso2.com
>>>>> email: ayy...@wso2.com cell: +94 77 1681010 <%2B94%2077%207779495>
>>>>>
>>>>> ___
>>>>> Dev mailing list
>>>>> Dev@wso2.org
>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> /sumedha
>>>> m: +94 773017743 <077%20301%207743>
>>>> b :  bit.ly/sumedha
>>>>
>>>
>>>
>>>
>>> --
>>> *Pamod Sylvester *
>>>
>>> *WSO2 Inc.; http://wso2.com <http://wso2.com>*
>>> cell: +94 77 7779495 <077%20777%209495>
>>>
>>
>>
>>
>> --
>> /sumedha
>> m: +94 773017743 <077%20301%207743>
>> b :  bit.ly/sumedha
>>
>
>
>
> --
> *Pamod Sylvester *
>
> *WSO2 Inc.; http://wso2.com <http://wso2.com>*
> cell: +94 77 7779495 <+94%2077%20777%209495>
>



-- 
Pumudu Ruhunage
Software Engineer | WSO2 Inc
M: +94 779 664493  | http://wso2.com
<https://wso2.com/signature>
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] WSO2 Message Broker 3.2.0 RC2 Released !!

2017-04-21 Thread Pumudu Ruhunage

Hi All,
Tested MQTT standalone/clustered basic test scenarios and execute eclipse
Interop test suite.

No issues found.
[+] Stable - go ahead and release

Thanks,

On Thu, Apr 20, 2017 at 6:47 PM, Pamod Sylvester <pa...@wso2.com> wrote:

> Hi Devs,
>
> This is the 2nd release candidate of WSO2 Message Broker 3.2.0.
>
> *This release fixes the following issues:*
> *https://wso2.org/jira/browse/MB-1910?filter=13675
> <https://wso2.org/jira/browse/MB-1910?filter=13675>*
>
> *Known Issues:*
> *https://wso2.org/jira/browse/MB-1757?filter=13676
> <https://wso2.org/jira/browse/MB-1757?filter=13676>*
>
> <https://wso2.org/jira/issues/?filter=12422>
> Please download MB 3.2.0 RC2 and test the functionality and vote. The
> vote will be open for 72 hours or as needed.
>
> *Source & binary distribution files:*
> https://github.com/wso2/product-mb/releases/tag/v3.2.0-RC2
>
> *The tag to be voted upon:*
> https://github.com/wso2/product-mb/tree/v3.2.0-RC2
>
>
> [+] Stable - go ahead and release
> [-]  Broken - do not release (explain why)
>
> Thanks and Regards,
>
> --
> *Pamod Sylvester *
>
> *WSO2 Inc.; http://wso2.com <http://wso2.com>*
> cell: +94 77 7779495 <077%20777%209495>
>
> ___
> Dev mailing list
> Dev@wso2.org
> http://wso2.org/cgi-bin/mailman/listinfo/dev
>
>


-- 
Pumudu Ruhunage
Software Engineer | WSO2 Inc
M: +94 779 664493  | http://wso2.com
<https://wso2.com/signature>
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] Is there a hard and fast rule to chose java 8 streams over for loops

2017-04-06 Thread Pumudu Ruhunage

Hi, IMO for less number of iterations/elements for loops are much
efficient, and when number gets higher both are performing with similar
efficiency. Advantage of parallel streams shows up when number of elements
increases and cost of each iteration increases.
Therefore, usage of streams/loops is highly dependent of the scenario and
can not suggest either without proper knowledge of the scenario, especially
for critical paths.

Regards,

On Thu, Apr 6, 2017 at 5:19 PM, Tharika Madurapperuma <thar...@wso2.com>
wrote:

> Hi Rajith,
>
> IMO Java streams should not be used always as a practice. It is not a
> silver bullet. There can be issues in performance in some cases as you have
> pointed out.
>
> One of the main advantage of using streams is when the logic of
> looping is quite complex and involves different operations on collections
> together, so that it can improve readability and understandability. But at
> the same time it can result in new and subtle bugs if we are not careful
> enough [1].
>
>  The performance aspect of streams vs loops depends on the type of the
> collection and the situation they deal with. This performance difference
> can sometimes be negligible. Hence we cannot completely go with one against
> another.
>
>  So IMHO we can consider using streams only when necessary based on
> our requirement and use loops where it deserves.
>
>
> [1] https://blog.jooq.org/2014/06/13/java-8-friday-10-subtle-
> mistakes-when-using-the-streams-api/
>
>
> Thanks,
> Tharika.
>
>
> On Thu, Mar 30, 2017 at 12:01 PM, Rajith Roshan <raji...@wso2.com> wrote:
>
>> Hi all,
>>
>> Most of our C5 development have used java8 streams api frequently more
>> than the for loops.
>> But there are some research have done to compare the performance of
>> streams api compared to for loops. They have shown that stream API is slow
>> compared to for loops in many scenarios (even for parallel streams). [1][2]
>>
>> Is there any proper guidance to use which one over the other or are we
>> using streams as a practice in C5 development
>>
>> [1] - https://blog.jooq.org/2015/12/08/3-reasons-why-you-shouldn
>> t-replace-your-for-loops-by-stream-foreach/
>> [2] - https://jaxenter.com/java-performance-tutorial-how-fast-ar
>> e-the-java-8-streams-118830.html
>>
>> Thanks!
>> Rajith
>>
>> --
>> Rajith Roshan
>> Software Engineer, WSO2 Inc.
>> Mobile: +94-72-642-8350 <%2B94-71-554-8430>
>>
>> ___
>> Dev mailing list
>> Dev@wso2.org
>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>
>>
>
>
> --
> *Tharika Madurapperuma*
> Software Engineer | WSO2, Inc.
>
> Email : thar...@wso2.com
> Mobile : +94777875624 <+94%2077%20787%205624>
> Web : http://wso2.com
>
> <http://wso2.com/signature>
>
> ___
> Dev mailing list
> Dev@wso2.org
> http://wso2.org/cgi-bin/mailman/listinfo/dev
>
>


-- 
Pumudu Ruhunage
Software Engineer | WSO2 Inc
M: +94 779 664493  | http://wso2.com
<https://wso2.com/signature>
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

[Dev] [Architecture] WSO2 Message Broker 3.2.0-ALPHA-2 Released!

2017-02-03 Thread Pumudu Ruhunage

Hi All,

WSO2 Message Broker team is pleased to announce the alpha-2 release of WSO2
Message Broker (MB) 3.2.0.

Source & binary distribution files of WSO2 Message Broker can be found in:
https://github.com/wso2/product-mb/releases/tag/v3.2.0-ALPHA2

All the known issues identified are listed at
https://wso2.org/jira/issues/?filter=13645

Thanks,
-- 
Pumudu Ruhunage
Software Engineer | WSO2 Inc
M: +94 779 664493  | http://wso2.com
<https://wso2.com/signature>
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

[Dev] [MB] MB 3.1.0 goes OOM with compression enabled with MSSQL as message store

2016-06-09 Thread Pumudu Ruhunage

Hi all,

we saw that in MB 3.1.0, for several scenarios (ex: 10KB messages, with
compression enabled) servers went OOM half way to 1M messages[1]. Please
note that this happens with default flow control settings.

AFAIU this is what happened.

1. With MySQL, database was the overall bottleneck in both compression and
non-compression scenarios.

2. MSSQL is faster than MySQL and with compression enabled DB is not the
bottleneck anymore in inbound message flow.
Compression was taking more execution time than db read/writes. Therefore,
compression logic becomes the bottleneck in MSSQL scenario.

3. In "contentChunckHandler" (which compress chunks if compression enabled)
has executed before[2] "messagePreprocessor" which has the flow control
logic[3] implemented. Therefore, lag in compression has not counted in flow
control logic. This leads inbound path accept messages from publishers more
than server can handle.

4. This might have lead to the OOM.

WDYT?


[1] https://wso2.org/jira/browse/MB-1683

[2]
https://github.com/wso2/andes/blob/v3.1.0/modules/andes-core/broker/src/main/java/org/wso2/andes/kernel/disruptor/inbound/InboundEventManager.java#L154

[3]
https://github.com/wso2/andes/blob/v3.1.0/modules/andes-core/broker/src/main/java/org/wso2/andes/kernel/disruptor/inbound/MessagePreProcessor.java#L148


Thanks,
-- 
Pumudu Ruhunage
Software Engineer | WSO2 Inc
http://wso2.com
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] [Docker][Puppet] [MB] Setting 'thriftServerHost' in WSO2 MB broker.xml file

2016-05-09 Thread Pumudu Ruhunage

re.init.CarbonServerManager.initializeCarbon(CarbonServerManager.java:514)
>>> at
>>> org.wso2.carbon.core.init.CarbonServerManager.removePendingItem(CarbonServerManager.java:290)
>>> at
>>> org.wso2.carbon.core.init.PreAxis2ConfigItemListener.bundleChanged(PreAxis2ConfigItemListener.java:118)
>>> at
>>> org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:847)
>>> at
>>> org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
>>> at
>>> org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:340)
>>> Caused by: org.wso2.andes.kernel.AndesException: Cannot start Thrift
>>> server on port 7611 on host wso2mb-default
>>> at org.wso2.andes.thrift.MBThriftServer.start(MBThriftServer.java:87)
>>> at
>>> org.wso2.andes.kernel.AndesKernelBoot.startThriftServer(AndesKernelBoot.java:440)
>>> at
>>> org.wso2.andes.kernel.AndesKernelBoot.initializeComponents(AndesKernelBoot.java:93)
>>> at org.wso2.andes.server.Broker.startupImpl(Broker.java:293)
>>> ... 70 more
>>> Caused by: org.apache.thrift.transport.TTransportException: Could not
>>> create ServerSocket on address wso2mb-default/10.100.186.164:7611.
>>> at
>>> org.apache.thrift.transport.TServerSocket.(TServerSocket.java:109)
>>> at
>>> org.apache.thrift.transport.TServerSocket.(TServerSocket.java:91)
>>> at
>>> org.apache.thrift.transport.TServerSocket.(TServerSocket.java:87)
>>> at org.wso2.andes.thrift.MBThriftServer.start(MBThriftServer.java:75)
>>> ... 73 more
>>> TID: [-1234] [] [2016-05-09 05:39:06,816]  INFO
>>> {org.wso2.andes.server.registry.ApplicationRegistry} -  Shutting down
>>> ApplicationRegistry(org.wso2.andes.server.registry.ConfigurationFileApplicationRegistry@70194966)
>>> {org.wso2.andes.server.registry.ApplicationRegistry}
>>>
>>>
>>> It's because currently  'thriftServerHost' is set to hostname in
>>> broker.xml [1] and according to MB clustering guide [2] , this should be
>>> set to MB node (docker container) IP which hosts the Thrift Server.
>>>
>>> If it's correct, what will be the best approach to update the
>>> 'thriftServerHost' value with container IP?
>>>
>>> Please give your suggestions.
>>>
>>> [1].
>>> https://github.com/wso2/puppet-modules/blob/master/modules/wso2mb/templates/3.1.0/repository/conf/broker.xml.erb#L37
>>> [2].
>>> https://docs.wso2.com/display/CLUSTER44x/Configuring+the+Broker+Nodes
>>>
>>>
>>> Thanks.
>>>
>>> --
>>> Thanuja Uruththirakodeeswaran
>>> Software Engineer
>>> WSO2 Inc.;http://wso2.com
>>> lean.enterprise.middleware
>>>
>>> mobile: +94 774363167
>>>
>>
>>
>>
>> --
>> Thanks and Regards,
>>
>> Isuru H.
>> +94 716 358 048* <http://wso2.com/>*
>>
>>
>>
>> ___
>> Dev mailing list
>> Dev@wso2.org
>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>
>>
>
>
> --
> *Darshana Akalanka Pagoda Arachchi,*
> *Software Engineer, WSO2*
> *+94777118016 <%2B94777118016>*
>
> ___
> Dev mailing list
> Dev@wso2.org
> http://wso2.org/cgi-bin/mailman/listinfo/dev
>
>


-- 
Pumudu Ruhunage
Associate Software Engineer | WSO2 Inc
M: +94 779 664493  | http://wso2.com
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] [VOTE] Release WSO2 Message Broker 3.0.0 RC 2

2015-11-07 Thread Pumudu Ruhunage

Hi all,

I tested following MQTT scenarios in a MB cluster.

   - Multiple subscriber scenarios
   - Multiple publisher scenarios
   - Mixed Multiple publisher/subscriber scenarios
  - Single topic 5 subscribers and 5 publishers
  - 5 topics 5 subscribers and 5 publishers (1 topic per subscriber and
  publisher)
  - Multiple topics 5 subscribers and 5 publishers
   - Clean session true scenarios
   - Clean session false scenarios
   - QOS levels basic test scenarios. Following QOS levels covered.
  - QOS 0
  - QOS 1
  - QOS 2
   - Hierarchical topics and Wildcard scenarios
   - Tenant level security - basic scenarios
   - Retain topic scenarios

Did not encounter issues.
Thus +1.  Stable - go ahead and release.

Thanks,
Pumudu

On Sat, Nov 7, 2015 at 2:21 PM, Hasitha Hiranya <hasit...@wso2.com> wrote:

> Hi all,
>
> I tested the following scenarios in a MB cluster
>
>
>- multiple publisher multiple consumer  - queues
>- multiple publisher multiple consumer  - topics
>- multiple publisher multiple consumer  - durable topic shared
>- multiple publisher multiple consumer  - queues -  with subscription
>break in middle
>- hierarchical topic subs - subscribers on multiple nodes
>- topics with wildcards (* and #) - subscribers on multiple nodes
>
>
> Did not encounter issues
> Thus +1.  Stable - go ahead and release
>
> Thanks
>
> On Sat, Nov 7, 2015 at 3:28 AM, Akalanka Pagoda Arachchi <
> darsha...@wso2.com> wrote:
>
>> Hi Devs,
>>
>> This is the second release candidate of WSO2 Message Broker 3.0.0.
>>
>> This release fixes the following issues:
>> https://wso2.org/jira/issues/?filter=12508
>>
>> <https://wso2.org/jira/issues/?filter=12422>
>> Please download, test and vote.
>>
>> Source & binary distribution files:
>> https://github.com/wso2/product-mb/releases/tag/v3.0.0-RC2
>>
>> Maven staging repo:
>> http://maven.wso2.org/nexus/content/repositories/orgwso2mb-118/
>>
>> The tag to be voted upon:
>> https://github.com/wso2/product-mb/tree/v3.0.0-RC2
>>
>>
>> [+] Stable - go ahead and release
>> [-]  Broken - do not release (explain why)
>>
>> Thanks and Regards,
>> MB Team.
>>
>> --
>> *Darshana Akalanka Pagoda Arachchi,*
>> *Software Engineer*
>> *078-4721791*
>>
>> ___
>> Dev mailing list
>> Dev@wso2.org
>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>
>>
>
>
> --
> *Hasitha Abeykoon*
> Senior Software Engineer; WSO2, Inc.; http://wso2.com
> *cell:* *+94 719363063*
> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com>
>
>
> ___
> Dev mailing list
> Dev@wso2.org
> http://wso2.org/cgi-bin/mailman/listinfo/dev
>
>


-- 
Pumudu Ruhunage
Associate Software Engineer | WSO2 Inc
M: +94 779 664493  | http://wso2.com
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

[Dev] [ESB][MB] SSL handshake Exception when try to send queue message from ESB 4.8.0 to MB 2.2.0

2015-10-08 Thread Pumudu Ruhunage

} -  [/10.
<http://10.100.5.165:54107>111 <http://10.100.5.165:54107/>.5.165:54107]
 Data Read: org.apache.mina.filter.support.SSLHandler@213b0eef
(HeapBuffer[pos=0 lim=8 cap=1024: 41 4D 51 50 00 00 09 01])
[2015-10-08 18:07:22,228] DEBUG
{org.wso2.andes.transport.network.mina.MinaNetworkHandler} -  [/10.
<http://10.100.5.165:54107>111 <http://10.100.5.165:54107/>.5.165:54107]
 doHandshake()
[2015-10-08 18:07:22,229] DEBUG
{org.wso2.andes.transport.network.mina.MinaNetworkHandler} -  [/10.
<http://10.100.5.165:54107>111 <http://10.100.5.165:54107/>.5.165:54107
<http://10.100.5.165:54107>]   handshakeStatus=NEED_UNWRAP
[2015-10-08 18:07:22,230] DEBUG
{org.wso2.andes.transport.network.mina.MinaNetworkHandler} -  [/10.
<http://10.100.5.165:54107>111 <http://10.100.5.165:54107/>.5.165:54107
<http://10.100.5.165:54107>]  *unwrapHandshake()*
[2015-10-08 18:07:22,230] DEBUG
{org.wso2.andes.transport.network.mina.MinaNetworkHandler} -  [/10.
<http://10.100.5.165:54107>111 <http://10.100.5.165:54107/>.5.165:54107
<http://10.100.5.165:54107>]inNetBuffer:
java.nio.DirectByteBuffer[pos=0 lim=8 cap=16665]
[2015-10-08 18:07:22,230] DEBUG
{org.wso2.andes.transport.network.mina.MinaNetworkHandler} -  [/10.
<http://10.100.5.165:54107>111 <http://10.100.5.165:54107/>.5.165:54107
<http://10.100.5.165:54107>]appBuffer: java.nio.DirectByteBuffer[pos=0
lim=0 cap=0]
[2015-10-08 18:07:22,231] ERROR
{org.wso2.andes.transport.network.mina.MinaNetworkHandler} -  Exception
caught by Mina
javax.net.ssl.SSLHandshakeException: SSL handshake failed.


Thanks,
Pumudu
-- 
Pumudu Ruhunage
Associate Software Engineer | WSO2 Inc
M: +94 779 664493  | http://wso2.com
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] [DEV][MB][MQTT] How to handle permission denied to subscribe a topic due to tenancy

2015-10-03 Thread Pumudu Ruhunage

Hi All,

@pamod, I also agree disconnecting is not the ideal solution. just thought
from client side, since it's not getting any indication of not subscribing
to the topic.

Then implementation will be as follows.

MQTT 3.1- No error code send to client.
MQTT 3.1.1 - Send error code 0x80 to client.

Thanks,
Pumudu

On Sat, Oct 3, 2015 at 3:44 PM, Ramith Jayasinghe <ram...@wso2.com> wrote:

> I think whats more relevant is to make v 3.1.1 case correct.
>
>
> On Sat, Oct 3, 2015 at 2:03 PM, Pamod Sylvester <pa...@wso2.com> wrote:
>
>> Both versions of the protocol are supported at the moment AFAIK.
>>
>> So for 3.1.1 sending the error code is adequate. As mentioned in the
>> previous mail there can be several topic subscriptions in one connection.
>> So disconnection might not be the ideal solution IMO. WDYT ?
>>
>>
>> On Saturday, October 3, 2015, Pumudu Ruhunage <pum...@wso2.com> wrote:
>>
>>> Hi All,
>>>
>>> Since we have to support both protocols we need implement both behaviors
>>> based on protocols? right ?
>>> IMO for 3.1 we need to indicate that it's not subscribed to the topic by
>>> disconnection if it's not subscribed. otherwise it gives false impression
>>> that client successfully subscribed to given topic.
>>>
>>> So can we agree on following implementation?
>>> MQTT 3.1- No error code send to client. Disconnect client
>>> from broker.
>>> MQTT 3.1.1 - Send error code 0x80 to client. (Is error code enough ?
>>> should we disconnect client from broker ?)
>>>
>>> WDYT ?
>>>
>>> Thanks,
>>> Pumudu
>>>
>>>
>>>
>>> On Sat, Oct 3, 2015 at 10:18 AM, Pamod Sylvester <pa...@wso2.com> wrote:
>>>
>>>> The CONNECT and SUBSCRIBE are two different command messages. The
>>>> subscription phase comes after the connection.
>>>>
>>>> Also the other fact is a single connection could involve multiple topic
>>>> subscriptions. So lets say if a given tenant sends a list of topic
>>>> subscriptions there could be subscriptions which the tenant could be
>>>> authorised among that list. So closing/refusing the entire connection might
>>>> not be appropriate in this case.
>>>>
>>>> - 0x05 (Connection Refused: not authorized) is sent as an
>>>> acknowledgement for CONNECT message (CONNACK).  i.e a given client has
>>>> provided unauthorised credentials etc.
>>>>
>>>> During this phase the subscription will not occur, we only could
>>>> validate whether a given tenants credentials are correct or not.
>>>>
>>>> - When the subscription happens based on my understanding of the spec
>>>> it has passed the CONNACK stage. The only way we could indicate an error
>>>> related to subscription is through SUBACK.
>>>>
>>>>   # The MQTT 3.1 specification has a limitation where error codes
>>>> cannot be sent with SUBACK. Hence for 3.1 spec having subscription failed
>>>> in silent mode is expected behaviour.
>>>>
>>>> You could find a similar discussion on [1]. Based on [1] what they've
>>>> done is hacked the client and the server to accept error codes for 3.1
>>>> specification. This will not be clean and will not interop with other
>>>> clients (even if we could hack the server to send error codes, the clients
>>>> which has implemented according to 3.1 specification will not accept the
>>>> error message).
>>>>
>>>> Another option is to initiate a disconnection from the broker to avoid
>>>> stale connections. But then again as mentioned in the beginning if there're
>>>> list of subscriptions for a connection, even the authorised subscriptions
>>>> would not be able to consume.
>>>>
>>>>   # The MQTT 3.1.1 specification there's an error code introduced 0x80
>>>> for forbidden subscriptions. So that should be the way we should ideally
>>>> use for indication of an error IMHO. But it'll be available in 3.1.1.
>>>>
>>>>
>>>> [1] https://issues.jboss.org/browse/ENTMQ-990
>>>>
>>>>
>>>>
>>>>
>>>> On Sat, Oct 3, 2015 at 7:06 AM, Hasitha Hiranya <hasit...@wso2.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Yes, Connection Refused, not authorized is the best we can do.
>>>>>
>>>>>

Re: [Dev] [DEV][MB][MQTT] How to handle permission denied to subscribe a topic due to tenancy

2015-10-03 Thread Pumudu Ruhunage

Hi All,

Since we have to support both protocols we need implement both behaviors
based on protocols? right ?
IMO for 3.1 we need to indicate that it's not subscribed to the topic by
disconnection if it's not subscribed. otherwise it gives false impression
that client successfully subscribed to given topic.

So can we agree on following implementation?
MQTT 3.1- No error code send to client. Disconnect client from broker.
MQTT 3.1.1 - Send error code 0x80 to client. (Is error code enough ? should
we disconnect client from broker ?)

WDYT ?

Thanks,
Pumudu



On Sat, Oct 3, 2015 at 10:18 AM, Pamod Sylvester <pa...@wso2.com> wrote:

> The CONNECT and SUBSCRIBE are two different command messages. The
> subscription phase comes after the connection.
>
> Also the other fact is a single connection could involve multiple topic
> subscriptions. So lets say if a given tenant sends a list of topic
> subscriptions there could be subscriptions which the tenant could be
> authorised among that list. So closing/refusing the entire connection might
> not be appropriate in this case.
>
> - 0x05 (Connection Refused: not authorized) is sent as an acknowledgement
> for CONNECT message (CONNACK).  i.e a given client has provided
> unauthorised credentials etc.
>
> During this phase the subscription will not occur, we only could validate
> whether a given tenants credentials are correct or not.
>
> - When the subscription happens based on my understanding of the spec it
> has passed the CONNACK stage. The only way we could indicate an error
> related to subscription is through SUBACK.
>
>   # The MQTT 3.1 specification has a limitation where error codes cannot
> be sent with SUBACK. Hence for 3.1 spec having subscription failed in
> silent mode is expected behaviour.
>
> You could find a similar discussion on [1]. Based on [1] what they've done
> is hacked the client and the server to accept error codes for 3.1
> specification. This will not be clean and will not interop with other
> clients (even if we could hack the server to send error codes, the clients
> which has implemented according to 3.1 specification will not accept the
> error message).
>
> Another option is to initiate a disconnection from the broker to avoid
> stale connections. But then again as mentioned in the beginning if there're
> list of subscriptions for a connection, even the authorised subscriptions
> would not be able to consume.
>
>   # The MQTT 3.1.1 specification there's an error code introduced 0x80 for
> forbidden subscriptions. So that should be the way we should ideally use
> for indication of an error IMHO. But it'll be available in 3.1.1.
>
>
> [1] https://issues.jboss.org/browse/ENTMQ-990
>
>
>
>
> On Sat, Oct 3, 2015 at 7:06 AM, Hasitha Hiranya <hasit...@wso2.com> wrote:
>
>> Hi,
>>
>> Yes, Connection Refused, not authorized is the best we can do.
>>
>> @Pamod, any batter alternative?
>>
>> Thanks!
>>
>> On Fri, Oct 2, 2015 at 10:55 PM, Yasassri Ratnayake <yasas...@wso2.com>
>> wrote:
>>
>>> Hi Pumudu,
>>>
>>> MQTT spec doesn't talk about a permission model, So IMO the closest
>>> error we can propagate to the client is "0x05 Connection Refused, not
>>> authorized".
>>>
>>>
>>> With Regards,
>>>
>>> On Fri, Oct 2, 2015 at 6:17 PM, Pumudu Ruhunage <pum...@wso2.com> wrote:
>>>
>>>> Hi All,
>>>>
>>>> If a tenant don't have permission to subscribe to a given topic (tenant
>>>> 'a.com' try to subscribe to tenant 'b.com') currently it will keep
>>>> connected even though it's not subscribed, because user authentication was
>>>> successful. This is misleading to subscriber as no return code sent and
>>>> it's actively connected to broker.
>>>> AFAIK mqtt don't define how to handle tenancy scenarios in it's spec.
>>>> Couldn't find a definition in MQTT spec 3.1.0/3.1.1 on how to reject if
>>>> permission denied to subscribe to a given topic(due to tenancy).
>>>>
>>>> One option to handle this scenario is to send return code 5 (Connection
>>>> Refused, not authorized) to client if it doesn't have permission to
>>>> subscribe to given topic. This way client will disconnect if it don't have
>>>> permission to subscribe. Is this acceptable or are there any alternatives
>>>> to handle this scenario?
>>>>
>>>> jira https://wso2.org/jira/browse/MB-1409
>>>>
>>>> Thanks,
>>>> Pumudu
>>>> --
&g

[Dev] [DEV][MB][MQTT] How to handle permission denied to subscribe a topic due to tenancy

2015-10-02 Thread Pumudu Ruhunage

Hi All,

If a tenant don't have permission to subscribe to a given topic (tenant '
a.com' try to subscribe to tenant 'b.com') currently it will keep connected
even though it's not subscribed, because user authentication was
successful. This is misleading to subscriber as no return code sent and
it's actively connected to broker.
AFAIK mqtt don't define how to handle tenancy scenarios in it's spec.
Couldn't find a definition in MQTT spec 3.1.0/3.1.1 on how to reject if
permission denied to subscribe to a given topic(due to tenancy).

One option to handle this scenario is to send return code 5 (Connection
Refused, not authorized) to client if it doesn't have permission to
subscribe to given topic. This way client will disconnect if it don't have
permission to subscribe. Is this acceptable or are there any alternatives
to handle this scenario?

jira https://wso2.org/jira/browse/MB-1409

Thanks,
Pumudu
-- 
Pumudu Ruhunage
Associate Software Engineer | WSO2 Inc
M: +94 779 664493  | http://wso2.com
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] [DEV] [Automation] How to get a value from Automation.xml file to test classes

2015-09-28 Thread Pumudu Ruhunage

Hi,

Have you tried something like this?

AutomationContext automationContext;
automationContext = new AutomationContext();
String maximumWaitingTime =
automationContext.getInstance().getProperty("maximumWaitingTime");

It's better if you can check how other integration tests were written and
follow the same pattern to refer automation.xml. (without creating new
automationContext and use already initialized objects etc.)

Regards,
Pumudu

On Mon, Sep 28, 2015 at 2:05 PM, Mushthaq Rumy <musht...@wso2.com> wrote:

> Hi Rajeenthani,
>
> I think you need to create an object instance of AutomationContext within
> your java class and proceed with it. The below link[1] might be useful for
> you.
>
> [1]
> http://malinthas.blogspot.com/2014/02/how-to-obtain-configuration-details.html
> .
>
>
>
> On Mon, Sep 28, 2015 at 1:44 PM, Rajeenthini Satkunam <
> rajeenth...@wso2.com> wrote:
>
>> Hi all,
>>
>> I am currently working on writing UI integration tests for product UES
>> using selenium.I have to make maximum waiting time for search element by()
>> as default value.As per discussion it can be configured through
>> automation.xml file. I have added 30> maximumWaitingTime> at automation.xml file.So I need to get the value of
>> it for test java classes.Can anyone please give any suggestion on how to
>> get that value?
>>
>> --
>>
>> *Thank You.*
>>
>> *Rajeenthini Satkunam*
>>
>> *Associate Software Engineer | WSO2*
>>
>>
>> *E:rajeenth...@wso2.com <rajeenth...@wso2.com>*
>>
>> *M :+94770832823 <%2B94770832823>   *
>>
>>
>> ___
>> Dev mailing list
>> Dev@wso2.org
>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>
>>
>
> Thanks & Regards,
> --
> Mushthaq Rumy
> *Software Engineer*
> Mobile : +94 (0) 779 492140 <%2B94%20%280%29%20773%20451194>
> Email : musht...@wso2.com
> WSO2, Inc.; http://wso2.com/
> lean . enterprise . middleware.
>
> ___
> Dev mailing list
> Dev@wso2.org
> http://wso2.org/cgi-bin/mailman/listinfo/dev
>
>


-- 
Pumudu Ruhunage
Associate Software Engineer | WSO2 Inc
M: +94 779 664493  | http://wso2.com
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

[Dev] [MB] com.hazelcast.core.OperationTimeoutException when testing MQTT

2015-08-24 Thread Pumudu Ruhunage

Hi All,

I came across attached intermittent hazelcast error[1] when doing some
tests on MQTT, 2-node cluster with latest code base. All messages are 1KB.
Slot storage mode has set to RDBMS.
After this error all operations on that node halted.

Following is the scenario I tested.

nodeA (coordinator)
publish to 'topicA' with QOS-1 100TPS
publish to 'topicC' with QOS-1 100TPS
publish to 'topic1' with QOS-0 100TPS
publish to 'topic3' with QOS-0 100TPS
subscribe to 'topicB' with 5 QOS-1 subscribers.
subscribe to 'topicD' with 5 QOS-1 subscribers.
subscribe to 'topic2' with 20 QOS-0 subscribers.
subscribe to 'topic4' with 20 QOS-0 subscribers.

nodeB
publish to 'topicB' with QOS-1 100TPS
publish to 'topicD' with QOS-1 100TPS
publish to 'topic2' with QOS-0 100TPS
publish to 'topic4' with QOS-0 100TPS
subscribe to 'topicA' with 5 QOS-1 subscribers.
subscribe to 'topicC' with 5 QOS-1 subscribers.
subscribe to 'topic1' with 20 QOS-0 subscribers.
subscribe to 'topic3' with 20 QOS-0 subscribers.

Have anyone faced this exception with hazelcast? What can be the cause of
the issue?

[1] https://wso2.org/jira/browse/MB-1193


Thanks,
Pumudu
-- 
Pumudu Ruhunage
Associate Software Engineer | WSO2 Inc
M: +94 779 664493  | http://wso2.com


hazelcast.core.OperationTimeoutException
Description: Binary data
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] [MB] How to clean up non-durable topic subscriptions if MB node killed in cluster setup

2015-08-23 Thread Pumudu Ruhunage

Hi All,
We have implemented it with following PR's[1][2] using newly introduced
methods by HasithaH.

Following subscription removing scenarios will be covered with the fix.
1. Remove non-durable subscriptions during startup and shuting down the
server for it's node id.
2. Remove non-durable subscriptions when member removed for it's node id
(by coordinator node).

[1] https://github.com/wso2/andes/pull/288
[2] https://github.com/wso2/andes/pull/286

Thanks,
Pumudu

On Mon, Aug 24, 2015 at 9:40 AM, Hasitha Hiranya hasit...@wso2.com wrote:

Hi Pumudu,

LocalSubscriptions are local to the node. You can deliver messages only to
the Localsubscriptions. Clustersubscription is the object carrying basic
info about subscriptions which does not have an actual channel associated
with it.

To solve the problem I have introduce a method to get a mock local
subscription using the cluster subscription.

Thanks

On Tue, Aug 18, 2015 at 6:22 PM, Pumudu Ruhunage pum...@wso2.com wrote:

Adding some info to first question.
As I understood there are two types of andes subscriptions for cluster
and local. When I try to call removeLocalSubscription() inside
createDisconnectOrRemoveClusterSubscription() to remove subscriptions from
local database i'm getting following exception[1].
This prevents removing redundant subscriptions from database during
startup while initializing in cluster mode.

Why do we need two type of andes subscriptions? can't we keep same
subscription type for both cluster and local in MB?

[1] Caused by: java.lang.ClassCastException:
org.wso2.andes.subscription.BasicSubscription cannot be cast to
org.wso2.andes.kernel.LocalSubscription

at
org.wso2.andes.subscription.SubscriptionStore.removeLocalSubscription(SubscriptionStore.java:645)

at
org.wso2.andes.subscription.SubscriptionStore.createDisconnectOrRemoveClusterSubscription(SubscriptionStore.java:487)

at
org.wso2.andes.kernel.AndesSubscriptionManager.updateClusterSubscriptionMaps(AndesSubscriptionManager.java:234)

at
org.wso2.andes.server.cluster.coordination.ClusterCoordinationHandler.handleClusterSubscriptionsChanged(ClusterCoordinationHandler.java:134)

at
org.wso2.andes.kernel.AndesSubscriptionManager.notifyClusterSubscriptionHasChanged(AndesSubscriptionManager.java:298)

at
org.wso2.andes.kernel.AndesSubscriptionManager.reloadSubscriptionsFromStorage(AndesSubscriptionManager.java:275)

at
org.wso2.andes.kernel.AndesRecoveryTask.reloadSubscriptions(AndesRecoveryTask.java:188)

at
org.wso2.andes.kernel.AndesRecoveryTask.recoverExchangesQueuesBindingsSubscriptions(AndesRecoveryTask.java:94)

at
org.wso2.andes.kernel.AndesKernelBoot.syncNodeWithClusterState(AndesKernelBoot.java:443)

at
org.wso2.andes.kernel.AndesKernelBoot.bootAndesKernel(AndesKernelBoot.java:116)

at org.wso2.andes.server.Broker.startupImpl(Broker.java:290)

Thanks,
Pumudu

On Tue, Aug 18, 2015 at 4:43 PM, Pumudu Ruhunage pum...@wso2.com wrote:

Hi All,

I faced two issues while fixing this.

1. There is a method for clear cluster subscriptions
('closeAllClusterSubscriptionsOfNode()' in 'andesSubscriptionManager')
while initializing a cluster node. But it dosen't clean up database when
executed. Is there a specific reason not to delete non-durable
subscriptions from database when call this method?

2. Currently all subscriptions(durable/non-durable) are stored in
MB_DURABLE_SUBSCRIPTION table. Is there a reason for this? Can't we change
this table name to something meaningful if we continue to keep all
subscriptions in same table?

WDYT?

Thanks,
Pumudu

On Thu, Aug 13, 2015 at 2:41 PM, Pumudu Ruhunage pum...@wso2.com
wrote:

Hi All,

Thanks for the inputs. +1 for implementing both cases.
will do both implementations.

Regards,
Pumudu

On Thu, Aug 13, 2015 at 11:40 AM, Hemika Kodikara hem...@wso2.com
wrote:

Hi All,

We have already done a similar implementation for durable topics on
member removed.

+1 for Asanka's suggestion to remove non-topic subscriptions on both
occasions.

Regards,
Hemika

Hemika Kodikara
Software Engineer
WSO2 Inc.
lean . enterprise . middleware
http://wso2.com

Mobile : +9477762

On Thu, Aug 13, 2015 at 11:09 AM, Asitha Nanayakkara asi...@wso2.com
wrote:

On Thu, Aug 13, 2015 at 3:17 PM, Asanka Abeyweera asank...@wso2.com
wrote:

Hi Asitha,

I don't think we can achieve this by only doing the cleanup at the
coordinator selection point. We have to clean the topic subscriptions
for a
non-coordinator node when it leaves (In this case a new coordinator will
not be elected).

Oh! yes agreed!

On Thu, Aug 13, 2015 at 5:04 AM, Asitha Nanayakkara asi...@wso2.com
wrote:

On Wed, Aug 12, 2015 at 10:53 PM, Asanka Abeyweera
asank...@wso2.com wrote:

Hi Pumudu,

I think we have to do the clean up in both places. Case 1 is
required when the only node in the cluster leaves and come back. Case
2 is
required when a single node leaves in a multi node cluster

Re: [Dev] [MB] How to clean up non-durable topic subscriptions if MB node killed in cluster setup

2015-08-18 Thread Pumudu Ruhunage

Hi All,

I faced two issues while fixing this.

1. There is a method for clear cluster subscriptions
('closeAllClusterSubscriptionsOfNode()' in 'andesSubscriptionManager')
while initializing a cluster node. But it dosen't clean up database when
executed. Is there a specific reason not to delete non-durable
subscriptions from database when call this method?

2. Currently all subscriptions(durable/non-durable) are stored in
MB_DURABLE_SUBSCRIPTION table. Is there a reason for this? Can't we change
this table name to something meaningful if we continue to keep all
subscriptions in same table?

WDYT?

Thanks,
Pumudu

On Thu, Aug 13, 2015 at 2:41 PM, Pumudu Ruhunage pum...@wso2.com wrote:

 Hi All,

 Thanks for the inputs. +1 for implementing both cases.
 will do both implementations.

 Regards,
 Pumudu

 On Thu, Aug 13, 2015 at 11:40 AM, Hemika Kodikara hem...@wso2.com wrote:

 Hi All,

 We have already done a similar implementation for durable topics on
 member removed.

 +1 for Asanka's suggestion to remove non-topic subscriptions on both
 occasions.

 Regards,
 Hemika

 Hemika Kodikara
 Software Engineer
 WSO2 Inc.
 lean . enterprise . middleware
 http://wso2.com

 Mobile : +9477762

 On Thu, Aug 13, 2015 at 11:09 AM, Asitha Nanayakkara asi...@wso2.com
 wrote:


 On Thu, Aug 13, 2015 at 3:17 PM, Asanka Abeyweera asank...@wso2.com
 wrote:

 Hi Asitha,

 I don't think we can achieve this by only doing the cleanup at the
 coordinator selection point. We have to clean the topic subscriptions for a
 non-coordinator node when it leaves (In this case a new coordinator will
 not be elected).

 Oh! yes agreed!


 On Thu, Aug 13, 2015 at 5:04 AM, Asitha Nanayakkara asi...@wso2.com
 wrote:



 On Wed, Aug 12, 2015 at 10:53 PM, Asanka Abeyweera asank...@wso2.com
 wrote:

 Hi Pumudu,

 I think we have to do the clean up in both places. Case 1 is required
 when the only node in the cluster leaves and come back. Case 2 is 
 required
 when a single node leaves in a multi node cluster.

 On Wed, Aug 12, 2015 at 5:38 PM, Pumudu Ruhunage pum...@wso2.com
 wrote:

 Hi,

 Currently if MB node got killed unexpectedly non-durable topic
 subscriptions bind to that node will remain in cluster. This can cause
 unexpected behaviors in cluster since that node has already removed.

 IMO these topic subscriptions can clean up in two methods.

 1. During each MB node start up, remove all non-durable topic
 subscriptions for it's local node id. (This will ensure newly started 
 node
 won't receive any non-durable topic messages related to previously 
 killed
 MB node with same node id)


 I think  we can do this in org.wso2.andes.kernel.AndesKernelBoot or
 in a similar class. May be we can do this similar to the way we do the 
 slot
 recovery task.



 2. Cluster coordinator should remove all non-durable topic
 subscriptions when a member left from cluster. (This can be a problem if
 coordinator node killed. If there's a method to get previous coordinator
 node id by newly elected coordinator this method can be implemented)


 For this case we can do the clean up in
 org.wso2.andes.server.cluster.ClusterManager#memberRemoved method after
 checking if the current node is the coordinator. This will work even if 
 the
 coordinator node get killed.


 When a new coordinator is elected can't we let that newly elected
 coordinator do this ? Even at startup there is a coordinator election
 right? if that's the case we can cover both scenarios IMO?



 How should we do the clean up. WDYT?

 Regards,
 Pumudu
 --
 Pumudu Ruhunage
 Associate Software Engineer | WSO2 Inc
 M: +94 779 664493  | http://wso2.com

 ___
 Dev mailing list
 Dev@wso2.org
 http://wso2.org/cgi-bin/mailman/listinfo/dev




 --
 Asanka Abeyweera
 Software Engineer
 WSO2 Inc.

 Phone: +94 712228648
 Blog: a5anka.github.io

 ___
 Dev mailing list
 Dev@wso2.org
 http://wso2.org/cgi-bin/mailman/listinfo/dev




 --
 *Asitha Nanayakkara*
 Software Engineer
 WSO2, Inc. http://wso2.com/
 Mob: + 94 77 85 30 682




 --
 Asanka Abeyweera
 Software Engineer
 WSO2 Inc.

 Phone: +94 712228648
 Blog: a5anka.github.io




 --
 *Asitha Nanayakkara*
 Software Engineer
 WSO2, Inc. http://wso2.com/
 Mob: + 94 77 85 30 682


 ___
 Dev mailing list
 Dev@wso2.org
 http://wso2.org/cgi-bin/mailman/listinfo/dev



 ___
 Dev mailing list
 Dev@wso2.org
 http://wso2.org/cgi-bin/mailman/listinfo/dev




 --
 Pumudu Ruhunage
 Associate Software Engineer | WSO2 Inc
 M: +94 779 664493  | http://wso2.com




-- 
Pumudu Ruhunage
Associate Software Engineer | WSO2 Inc
M: +94 779 664493  | http://wso2.com
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] [MB] How to clean up non-durable topic subscriptions if MB node killed in cluster setup

2015-08-18 Thread Pumudu Ruhunage

Adding some info to first question.
As I understood there are two types of andes subscriptions for cluster and
local. When I try to call removeLocalSubscription() inside
createDisconnectOrRemoveClusterSubscription() to remove subscriptions from
local database i'm getting following exception[1].
This prevents removing redundant subscriptions from database during startup
while initializing in cluster mode.