Re: [Dev] [MB] ValueEvent exception occurred on disruptor for MQTT

[Pumudu Ruhunage]

Hi Sumedha,

As pamod mention this is a generic exception[1][2][3] which requires more
info to identify the exact cause.
We had long running test(mixed QOS test) for around 5 days(120h+) and
during that we didn't observe this particular issue as well.
Further, we have done several comparisons tests[4] and added few
integration mqtt load tests[5] to verify mqtt stability.

[1] https://wso2.org/jira/browse/MB-1211
[2] https://wso2.org/jira/browse/MB-1837
[3] https://wso2.org/jira/browse/MB-1328

[4]
https://docs.google.com/a/wso2.com/spreadsheets/d/1C50DqgBqXB8_-SND_YJpmXE-ArhxN8Fbg6-K0kQAZZc/edit?usp=sharing
[5]
https://github.com/wso2/product-mb/tree/v3.1.0/modules/integration/tests-integration/tests-mqtt/src/test/java/org/wso2/mb/integration/tests/mqtt/load

Thanks,

On Thu, Apr 27, 2017 at 11:45 PM, Pamod Sylvester  wrote:

> Hi Sumedha,
>
> With the amount of information available it would be hard to conclude
> where the error occurs.
>
> Also it would be ideal if there's a way to get disruptor to pin-point the
> exact location of the Nullpointer when it occurs in the stack trace in it's
> log exception handler. Will update this thread if i find any.
>
> Let's also make sure to monitor the packets exchanged between the clients
> since this issue occurs during the protocol handling and converting the
> bytes[] to a message, so that we could identify whether it's due to a
> result of malformed packet etc just to make sure.
>
> @Pumudu please add your thoughts. Also could you share the details of the
> long running tests we did (the type of tests we carried out etc)
>
> Thanks,
> Pamod
>
> On Thu, Apr 27, 2017 at 9:51 PM, Sumedha Rubasinghe 
> wrote:
>
>> Pamod,
>> What is the most longest running MB deployment we have?
>>
>> Is this case, we have had 8 devices pushing data to MB several time
>> within a minute and a DAS listener picking them up. We had this setup
>> running for more than 1 month.
>>
>> On Thu, Apr 27, 2017 at 8:18 PM, Pamod Sylvester  wrote:
>>
>>> We've not come across this issue before.
>>>
>>> it seems, this has occurred in one of the disruptor handlers in
>>> Moquette. Shall we enable debug/tracing logs in Moquette side to further
>>> drill down the issue,
>>>
>>> The following logs will be useful,
>>>
>>>  1. org.dna.mqtt.moquette.messaging.spi.impl.SimpleMessaging
>>>  2. org.dna.mqtt.moquette.messaging.spi.impl.ProtocolProcessor
>>>
>>> Thanks,
>>> Pamod
>>>
>>> On Thu, Apr 27, 2017 at 6:59 PM, Sumedha Rubasinghe 
>>> wrote:
>>>
 MB guys,
 You can consider this as a real production issue. We have been running
 the server for few weeks now.

 On Thu, Apr 27, 2017 at 4:22 PM, Ayyoob Hamza  wrote:

> Hi All,
>
> We did a long running test for a couple of weeks and in MB we see
> following logs getting printed continuously and devices are failing to
> publish data. Is this a known issue?
>
> TID: [] [] [2017-04-27 16:03:21,390] ERROR
> {org.dna.mqtt.wso2.MQTTLogExceptionHandler} -  ValueEvent exception
> occurred on disruptor. {org.dna.mqtt.wso2.MQTTLogExceptionHandler}
> java.lang.NullPointerException
>
>
> Thanks,
> *Ayyoob Hamza*
> *Senior Software Engineer*
> WSO2 Inc.; http://wso2.com
> email: ayy...@wso2.com cell: +94 77 1681010 <%2B94%2077%207779495>
>
> ___
> Dev mailing list
> Dev@wso2.org
> http://wso2.org/cgi-bin/mailman/listinfo/dev
>
>


 --
 /sumedha
 m: +94 773017743 <077%20301%207743>
 b :  bit.ly/sumedha

>>>
>>>
>>>
>>> --
>>> *Pamod Sylvester *
>>>
>>> *WSO2 Inc.; http://wso2.com *
>>> cell: +94 77 7779495 <077%20777%209495>
>>>
>>
>>
>>
>> --
>> /sumedha
>> m: +94 773017743 <077%20301%207743>
>> b :  bit.ly/sumedha
>>
>
>
>
> --
> *Pamod Sylvester *
>
> *WSO2 Inc.; http://wso2.com *
> cell: +94 77 7779495 <+94%2077%20777%209495>
>



-- 
Pumudu Ruhunage
Software Engineer | WSO2 Inc
M: +94 779 664493  | http://wso2.com

___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] [MB] ValueEvent exception occurred on disruptor for MQTT

[Pamod Sylvester]

Hi Sumedha,

With the amount of information available it would be hard to conclude where
the error occurs.

Also it would be ideal if there's a way to get disruptor to pin-point the
exact location of the Nullpointer when it occurs in the stack trace in it's
log exception handler. Will update this thread if i find any.

Let's also make sure to monitor the packets exchanged between the clients
since this issue occurs during the protocol handling and converting the
bytes[] to a message, so that we could identify whether it's due to a
result of malformed packet etc just to make sure.

@Pumudu please add your thoughts. Also could you share the details of the
long running tests we did (the type of tests we carried out etc)

Thanks,
Pamod

On Thu, Apr 27, 2017 at 9:51 PM, Sumedha Rubasinghe 
wrote:

> Pamod,
> What is the most longest running MB deployment we have?
>
> Is this case, we have had 8 devices pushing data to MB several time within
> a minute and a DAS listener picking them up. We had this setup running for
> more than 1 month.
>
> On Thu, Apr 27, 2017 at 8:18 PM, Pamod Sylvester  wrote:
>
>> We've not come across this issue before.
>>
>> it seems, this has occurred in one of the disruptor handlers in Moquette.
>> Shall we enable debug/tracing logs in Moquette side to further drill down
>> the issue,
>>
>> The following logs will be useful,
>>
>>  1. org.dna.mqtt.moquette.messaging.spi.impl.SimpleMessaging
>>  2. org.dna.mqtt.moquette.messaging.spi.impl.ProtocolProcessor
>>
>> Thanks,
>> Pamod
>>
>> On Thu, Apr 27, 2017 at 6:59 PM, Sumedha Rubasinghe 
>> wrote:
>>
>>> MB guys,
>>> You can consider this as a real production issue. We have been running
>>> the server for few weeks now.
>>>
>>> On Thu, Apr 27, 2017 at 4:22 PM, Ayyoob Hamza  wrote:
>>>
 Hi All,

 We did a long running test for a couple of weeks and in MB we see
 following logs getting printed continuously and devices are failing to
 publish data. Is this a known issue?

 TID: [] [] [2017-04-27 16:03:21,390] ERROR
 {org.dna.mqtt.wso2.MQTTLogExceptionHandler} -  ValueEvent exception
 occurred on disruptor. {org.dna.mqtt.wso2.MQTTLogExceptionHandler}
 java.lang.NullPointerException


 Thanks,
 *Ayyoob Hamza*
 *Senior Software Engineer*
 WSO2 Inc.; http://wso2.com
 email: ayy...@wso2.com cell: +94 77 1681010 <%2B94%2077%207779495>

 ___
 Dev mailing list
 Dev@wso2.org
 http://wso2.org/cgi-bin/mailman/listinfo/dev


>>>
>>>
>>> --
>>> /sumedha
>>> m: +94 773017743 <077%20301%207743>
>>> b :  bit.ly/sumedha
>>>
>>
>>
>>
>> --
>> *Pamod Sylvester *
>>
>> *WSO2 Inc.; http://wso2.com *
>> cell: +94 77 7779495 <077%20777%209495>
>>
>
>
>
> --
> /sumedha
> m: +94 773017743 <077%20301%207743>
> b :  bit.ly/sumedha
>



-- 
*Pamod Sylvester *

*WSO2 Inc.; http://wso2.com *
cell: +94 77 7779495
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] [MB] ValueEvent exception occurred on disruptor for MQTT

[Sumedha Rubasinghe]

Pamod,
What is the most longest running MB deployment we have?

Is this case, we have had 8 devices pushing data to MB several time within
a minute and a DAS listener picking them up. We had this setup running for
more than 1 month.

On Thu, Apr 27, 2017 at 8:18 PM, Pamod Sylvester  wrote:

> We've not come across this issue before.
>
> it seems, this has occurred in one of the disruptor handlers in Moquette.
> Shall we enable debug/tracing logs in Moquette side to further drill down
> the issue,
>
> The following logs will be useful,
>
>  1. org.dna.mqtt.moquette.messaging.spi.impl.SimpleMessaging
>  2. org.dna.mqtt.moquette.messaging.spi.impl.ProtocolProcessor
>
> Thanks,
> Pamod
>
> On Thu, Apr 27, 2017 at 6:59 PM, Sumedha Rubasinghe 
> wrote:
>
>> MB guys,
>> You can consider this as a real production issue. We have been running
>> the server for few weeks now.
>>
>> On Thu, Apr 27, 2017 at 4:22 PM, Ayyoob Hamza  wrote:
>>
>>> Hi All,
>>>
>>> We did a long running test for a couple of weeks and in MB we see
>>> following logs getting printed continuously and devices are failing to
>>> publish data. Is this a known issue?
>>>
>>> TID: [] [] [2017-04-27 16:03:21,390] ERROR 
>>> {org.dna.mqtt.wso2.MQTTLogExceptionHandler}
>>> -  ValueEvent exception occurred on disruptor.
>>> {org.dna.mqtt.wso2.MQTTLogExceptionHandler}
>>> java.lang.NullPointerException
>>>
>>>
>>> Thanks,
>>> *Ayyoob Hamza*
>>> *Senior Software Engineer*
>>> WSO2 Inc.; http://wso2.com
>>> email: ayy...@wso2.com cell: +94 77 1681010 <%2B94%2077%207779495>
>>>
>>> ___
>>> Dev mailing list
>>> Dev@wso2.org
>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>
>>>
>>
>>
>> --
>> /sumedha
>> m: +94 773017743 <077%20301%207743>
>> b :  bit.ly/sumedha
>>
>
>
>
> --
> *Pamod Sylvester *
>
> *WSO2 Inc.; http://wso2.com *
> cell: +94 77 7779495 <077%20777%209495>
>



-- 
/sumedha
m: +94 773017743
b :  bit.ly/sumedha
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] [MB] ValueEvent exception occurred on disruptor for MQTT

[Ayyoob Hamza]

Hi Pamod,

I'll enable those logs and will do another run, But it might take couple of
weeks to reproduce this.

Thanks,
Ayyoob

*Ayyoob Hamza*
*Senior Software Engineer*
WSO2 Inc.; http://wso2.com
email: ayy...@wso2.com cell: +94 77 1681010 <%2B94%2077%207779495>

On Thu, Apr 27, 2017 at 8:18 PM, Pamod Sylvester  wrote:

> We've not come across this issue before.
>
> it seems, this has occurred in one of the disruptor handlers in Moquette.
> Shall we enable debug/tracing logs in Moquette side to further drill down
> the issue,
>
> The following logs will be useful,
>
>  1. org.dna.mqtt.moquette.messaging.spi.impl.SimpleMessaging
>  2. org.dna.mqtt.moquette.messaging.spi.impl.ProtocolProcessor
>
> Thanks,
> Pamod
>
> On Thu, Apr 27, 2017 at 6:59 PM, Sumedha Rubasinghe 
> wrote:
>
>> MB guys,
>> You can consider this as a real production issue. We have been running
>> the server for few weeks now.
>>
>> On Thu, Apr 27, 2017 at 4:22 PM, Ayyoob Hamza  wrote:
>>
>>> Hi All,
>>>
>>> We did a long running test for a couple of weeks and in MB we see
>>> following logs getting printed continuously and devices are failing to
>>> publish data. Is this a known issue?
>>>
>>> TID: [] [] [2017-04-27 16:03:21,390] ERROR 
>>> {org.dna.mqtt.wso2.MQTTLogExceptionHandler}
>>> -  ValueEvent exception occurred on disruptor.
>>> {org.dna.mqtt.wso2.MQTTLogExceptionHandler}
>>> java.lang.NullPointerException
>>>
>>>
>>> Thanks,
>>> *Ayyoob Hamza*
>>> *Senior Software Engineer*
>>> WSO2 Inc.; http://wso2.com
>>> email: ayy...@wso2.com cell: +94 77 1681010 <%2B94%2077%207779495>
>>>
>>> ___
>>> Dev mailing list
>>> Dev@wso2.org
>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>
>>>
>>
>>
>> --
>> /sumedha
>> m: +94 773017743 <077%20301%207743>
>> b :  bit.ly/sumedha
>>
>
>
>
> --
> *Pamod Sylvester *
>
> *WSO2 Inc.; http://wso2.com *
> cell: +94 77 7779495 <+94%2077%20777%209495>
>
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] [MB] ValueEvent exception occurred on disruptor for MQTT

[Pamod Sylvester]

We've not come across this issue before.

it seems, this has occurred in one of the disruptor handlers in Moquette.
Shall we enable debug/tracing logs in Moquette side to further drill down
the issue,

The following logs will be useful,

 1. org.dna.mqtt.moquette.messaging.spi.impl.SimpleMessaging
 2. org.dna.mqtt.moquette.messaging.spi.impl.ProtocolProcessor

Thanks,
Pamod

On Thu, Apr 27, 2017 at 6:59 PM, Sumedha Rubasinghe 
wrote:

> MB guys,
> You can consider this as a real production issue. We have been running the
> server for few weeks now.
>
> On Thu, Apr 27, 2017 at 4:22 PM, Ayyoob Hamza  wrote:
>
>> Hi All,
>>
>> We did a long running test for a couple of weeks and in MB we see
>> following logs getting printed continuously and devices are failing to
>> publish data. Is this a known issue?
>>
>> TID: [] [] [2017-04-27 16:03:21,390] ERROR 
>> {org.dna.mqtt.wso2.MQTTLogExceptionHandler}
>> -  ValueEvent exception occurred on disruptor.
>> {org.dna.mqtt.wso2.MQTTLogExceptionHandler}
>> java.lang.NullPointerException
>>
>>
>> Thanks,
>> *Ayyoob Hamza*
>> *Senior Software Engineer*
>> WSO2 Inc.; http://wso2.com
>> email: ayy...@wso2.com cell: +94 77 1681010 <%2B94%2077%207779495>
>>
>> ___
>> Dev mailing list
>> Dev@wso2.org
>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>
>>
>
>
> --
> /sumedha
> m: +94 773017743 <077%20301%207743>
> b :  bit.ly/sumedha
>



-- 
*Pamod Sylvester *

*WSO2 Inc.; http://wso2.com *
cell: +94 77 7779495
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] [MB] ValueEvent exception occurred on disruptor for MQTT

[Sumedha Rubasinghe]

MB guys,
You can consider this as a real production issue. We have been running the
server for few weeks now.

On Thu, Apr 27, 2017 at 4:22 PM, Ayyoob Hamza  wrote:

> Hi All,
>
> We did a long running test for a couple of weeks and in MB we see
> following logs getting printed continuously and devices are failing to
> publish data. Is this a known issue?
>
> TID: [] [] [2017-04-27 16:03:21,390] ERROR 
> {org.dna.mqtt.wso2.MQTTLogExceptionHandler}
> -  ValueEvent exception occurred on disruptor.
> {org.dna.mqtt.wso2.MQTTLogExceptionHandler}
> java.lang.NullPointerException
>
>
> Thanks,
> *Ayyoob Hamza*
> *Senior Software Engineer*
> WSO2 Inc.; http://wso2.com
> email: ayy...@wso2.com cell: +94 77 1681010 <%2B94%2077%207779495>
>
> ___
> Dev mailing list
> Dev@wso2.org
> http://wso2.org/cgi-bin/mailman/listinfo/dev
>
>


-- 
/sumedha
m: +94 773017743
b :  bit.ly/sumedha
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

[Dev] [MB] ValueEvent exception occurred on disruptor for MQTT

[Ayyoob Hamza]

Hi All,

We did a long running test for a couple of weeks and in MB we see following
logs getting printed continuously and devices are failing to publish data.
Is this a known issue?

TID: [] [] [2017-04-27 16:03:21,390] ERROR
{org.dna.mqtt.wso2.MQTTLogExceptionHandler}
-  ValueEvent exception occurred on disruptor. {org.dna.mqtt.wso2.
MQTTLogExceptionHandler}
java.lang.NullPointerException


Thanks,
*Ayyoob Hamza*
*Senior Software Engineer*
WSO2 Inc.; http://wso2.com
email: ayy...@wso2.com cell: +94 77 1681010 <%2B94%2077%207779495>
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] [MB] ValueEvent exception occurred on disruptor for MQTT

[Ayyoob Hamza]

Please note this works after restarting the server.

*Ayyoob Hamza*
*Senior Software Engineer*
WSO2 Inc.; http://wso2.com
email: ayy...@wso2.com cell: +94 77 1681010 <%2B94%2077%207779495>

On Thu, Apr 27, 2017 at 4:22 PM, Ayyoob Hamza  wrote:

> Hi All,
>
> We did a long running test for a couple of weeks and in MB we see
> following logs getting printed continuously and devices are failing to
> publish data. Is this a known issue?
>
> TID: [] [] [2017-04-27 16:03:21,390] ERROR 
> {org.dna.mqtt.wso2.MQTTLogExceptionHandler}
> -  ValueEvent exception occurred on disruptor.
> {org.dna.mqtt.wso2.MQTTLogExceptionHandler}
> java.lang.NullPointerException
>
>
> Thanks,
> *Ayyoob Hamza*
> *Senior Software Engineer*
> WSO2 Inc.; http://wso2.com
> email: ayy...@wso2.com cell: +94 77 1681010 <%2B94%2077%207779495>
>
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev