Re: [Dev] [IoTS] IoTS 3.1.0-Update11 throw intermittent exception due to concurrency issue.

[Sumedha Rubasinghe]

Guys,
While discussing the technical reasoning, please create a git issue for
things like this. These are public releases and someone else could be
facing the same thing across the world.

Having a issue created and that resolved with a fixed version would help
 some other user down the lane.

On Wed, Jan 24, 2018 at 7:38 AM SajithAR Ariyarathna 
wrote:

> Hi Rasika,
>
> Regarding your code snippet:
>
>> try {
>> operationMappingDAO.updateOperationMapping(operationId, enrolmentId, 
>> org.wso2.carbon.device.mgt.core.dto.operation.mgt.Operation.PushNotificationStatus.COMPLETED);
>> notificationStrategy.execute(new NotificationContext(deviceId, 
>> operation));
>> } catch (Throwable e) {
>> log.error("Error occurred while sending push notifications to " +
>>   deviceId.getType() + " device carrying id '" +
>>   deviceId + "'", e);
>> // Reschedule if push notification failed.
>> operationMappingDAO.updateOperationMapping(operationId, enrolmentId, 
>> org.wso2.carbon.device.mgt.core.dto.operation.mgt.Operation.PushNotificationStatus.SCHEDULED);
>> }
>>
>> Catching Throwable means that you are catching Error as well, and that is
> not recommended [1]. The rule of thumb that need to follow when catching
> Error is whether we can fully recover from an Error. If the answer is yes,
> then its fine to catch Error (catch Throwable), otherwise catching
> Exception will be enough.
>
> AFAIU, only  notificationStrategy.execute(new NotificationContext(deviceId,
> operation)); line is the concern here. Seems like operationMappingDAO.
> updateOperationMapping is a DB operation. If so, then my suggestion is to
> have 2 try-catch blocks. One for the operationMappingDAO.
> updateOperationMapping line with catching appropriate DB operation
> exception (SQLException). Another one for the notificationStrategy
> .execute(new NotificationContext(deviceId, operation)); catching
> Throwable.
>
> [1] https://stackoverflow.com/a/11018879/1577286
>
> Thanks.
>
> On Tue, Jan 23, 2018 at 8:23 PM, Geeth Munasinghe  wrote:
>
>> Hi Charitha/Rasika
>>
>>
>> On Tue, Jan 23, 2018 at 8:17 PM, Rasika Perera  wrote:
>>
>>> Good catch Charitha! As you explained this code might raise unforeseen
>>> issues.
>>>
>>> However; if we are bringing the update operation mapping before the FCM
>>> notification; as per [1] status of the push notification will be marked as
>>> *COMPLETED* even before executing the notification sending process. And
>>> this will cause another issue of an inconsistent state.
>>>
>>> Most of the time, these notification providers involves calling external
>>> 3rd party libraries(eg. MQTT, FCM...etc). Suppose there's a
>>> RuntimeException thrown while sending the push notification. Notification
>>> status still will be marked as COMPLETED and will not try ever again since
>>> it is not tracked with the current try-catch block(it only catches
>>> PushNotificationExecutionFailedException).
>>>
>>> Thus, If we are changing the order as suggested please consider adding a
>>> Throwable catch block(since notificationStrategy.execute() calls 3rd party
>>> or untrusted libraries);
>>>
>>> try {
>>> operationMappingDAO.updateOperationMapping(operationId, enrolmentId, 
>>> org.wso2.carbon.device.mgt.core.dto.operation.mgt.Operation.PushNotificationStatus.COMPLETED);
>>> notificationStrategy.execute(new NotificationContext(deviceId, 
>>> operation));
>>> } catch (Throwable e) {
>>> log.error("Error occurred while sending push notifications to " +
>>>   deviceId.getType() + " device carrying id '" +
>>>   deviceId + "'", e);
>>> // Reschedule if push notification failed.
>>> operationMappingDAO.updateOperationMapping(operationId, enrolmentId, 
>>> org.wso2.carbon.device.mgt.core.dto.operation.mgt.Operation.PushNotificationStatus.SCHEDULED);
>>> }
>>>
>>>
>> +1 for this approach.
>>
>>>
>>> [1]
>>> https://github.com/wso2/carbon-device-mgt/blob/e56fb5f4de5d8f08af879a39a412ff501cba35fc/components/device-mgt/org.wso2.carbon.device.mgt.core/src/main/java/org/wso2/carbon/device/mgt/core/operation/mgt/OperationManagerImpl.java#L196
>>>
>>> On Tue, Jan 23, 2018 at 7:16 PM, Charitha Goonetilleke <
>>> charit...@wso2.com> wrote:
>>>
 Hi All,

 I have implemented state life cycle to prevent concurrent API call from
 agent to pending operations endpoint. However issue is still appearing. I
 have tested the same scenario with Update 10 and was able to reproduce the
 issue with Update 10 pack also.

 However as I figured out, we are sending FCM notification before
 updating operation mapping table[1]. Due to that device received FCM
 notification in realtime as same as it is being added to the Operations DB
 in the server. So there is chance to a raise condition and as a result, we
 are observing above exception. So we might need to 

Re: [Dev] [IoTS] IoTS 3.1.0-Update11 throw intermittent exception due to concurrency issue.

[Rasika Perera]

Hi Sajith,

Thanks for the suggestions.

Catching Throwable means that you are catching Error as well, and that is
> not recommended [1]. The rule of thumb that need to follow when catching
> Error is whether we can fully recover from an Error. If the answer is yes,
> then its fine to catch Error (catch Throwable), otherwise catching
> Exception will be enough.

Yes, I am aware of it. My intention was to correct the push notification
status at all cost when whatever throwable occurs since we mark it as
*COMPLETED* prior to calling notification send method.

AFAIU, only  notificationStrategy.execute(new NotificationContext(deviceId,
> operation)); line is the concern here. Seems like operationMappingDAO.
> updateOperationMapping is a DB operation. If so, then my suggestion is to
> have 2 try-catch blocks. One for the operationMappingDAO.update
> OperationMapping line with catching appropriate DB operation exception
> (SQLException). Another one for the notificationStrategy.execute(new 
> NotificationContext(deviceId,
> operation)); catching Throwable.

Agree. We'll consider adding two try-catch blocks.

Best Regards,
~Rasika

On Wed, Jan 24, 2018 at 7:36 AM, SajithAR Ariyarathna 
wrote:

> Hi Rasika,
>
> Regarding your code snippet:
>
>> try {
>> operationMappingDAO.updateOperationMapping(operationId, enrolmentId, 
>> org.wso2.carbon.device.mgt.core.dto.operation.mgt.Operation.PushNotificationStatus.COMPLETED);
>> notificationStrategy.execute(new NotificationContext(deviceId, 
>> operation));
>> } catch (Throwable e) {
>> log.error("Error occurred while sending push notifications to " +
>>   deviceId.getType() + " device carrying id '" +
>>   deviceId + "'", e);
>> // Reschedule if push notification failed.
>> operationMappingDAO.updateOperationMapping(operationId, enrolmentId, 
>> org.wso2.carbon.device.mgt.core.dto.operation.mgt.Operation.PushNotificationStatus.SCHEDULED);
>> }
>>
>> Catching Throwable means that you are catching Error as well, and that is
> not recommended [1]. The rule of thumb that need to follow when catching
> Error is whether we can fully recover from an Error. If the answer is yes,
> then its fine to catch Error (catch Throwable), otherwise catching
> Exception will be enough.
>
> AFAIU, only  notificationStrategy.execute(new NotificationContext(deviceId,
> operation)); line is the concern here. Seems like operationMappingDAO.
> updateOperationMapping is a DB operation. If so, then my suggestion is to
> have 2 try-catch blocks. One for the operationMappingDAO.update
> OperationMapping line with catching appropriate DB operation exception
> (SQLException). Another one for the notificationStrategy.execute(new 
> NotificationContext(deviceId,
> operation)); catching Throwable.
>
> [1] https://stackoverflow.com/a/11018879/1577286
>
> Thanks.
>
> On Tue, Jan 23, 2018 at 8:23 PM, Geeth Munasinghe  wrote:
>
>> Hi Charitha/Rasika
>>
>>
>> On Tue, Jan 23, 2018 at 8:17 PM, Rasika Perera  wrote:
>>
>>> Good catch Charitha! As you explained this code might raise unforeseen
>>> issues.
>>>
>>> However; if we are bringing the update operation mapping before the FCM
>>> notification; as per [1] status of the push notification will be marked as
>>> *COMPLETED* even before executing the notification sending process. And
>>> this will cause another issue of an inconsistent state.
>>>
>>> Most of the time, these notification providers involves calling external
>>> 3rd party libraries(eg. MQTT, FCM...etc). Suppose there's a
>>> RuntimeException thrown while sending the push notification. Notification
>>> status still will be marked as COMPLETED and will not try ever again since
>>> it is not tracked with the current try-catch block(it only catches
>>> PushNotificationExecutionFailedException).
>>>
>>> Thus, If we are changing the order as suggested please consider adding a
>>> Throwable catch block(since notificationStrategy.execute() calls 3rd party
>>> or untrusted libraries);
>>>
>>> try {
>>> operationMappingDAO.updateOperationMapping(operationId, enrolmentId, 
>>> org.wso2.carbon.device.mgt.core.dto.operation.mgt.Operation.PushNotificationStatus.COMPLETED);
>>> notificationStrategy.execute(new NotificationContext(deviceId, 
>>> operation));
>>> } catch (Throwable e) {
>>> log.error("Error occurred while sending push notifications to " +
>>>   deviceId.getType() + " device carrying id '" +
>>>   deviceId + "'", e);
>>> // Reschedule if push notification failed.
>>> operationMappingDAO.updateOperationMapping(operationId, enrolmentId, 
>>> org.wso2.carbon.device.mgt.core.dto.operation.mgt.Operation.PushNotificationStatus.SCHEDULED);
>>> }
>>>
>>>
>> +1 for this approach.
>>
>>>
>>> [1] https://github.com/wso2/carbon-device-mgt/blob/e56fb5f4d
>>> e5d8f08af879a39a412ff501cba35fc/components/device-mgt/org.ws
>>> 

Re: [Dev] [IoTS] IoTS 3.1.0-Update11 throw intermittent exception due to concurrency issue.

[SajithAR Ariyarathna]

Hi Rasika,

Regarding your code snippet:

> try {
> operationMappingDAO.updateOperationMapping(operationId, enrolmentId, 
> org.wso2.carbon.device.mgt.core.dto.operation.mgt.Operation.PushNotificationStatus.COMPLETED);
> notificationStrategy.execute(new NotificationContext(deviceId, 
> operation));
> } catch (Throwable e) {
> log.error("Error occurred while sending push notifications to " +
>   deviceId.getType() + " device carrying id '" +
>   deviceId + "'", e);
> // Reschedule if push notification failed.
> operationMappingDAO.updateOperationMapping(operationId, enrolmentId, 
> org.wso2.carbon.device.mgt.core.dto.operation.mgt.Operation.PushNotificationStatus.SCHEDULED);
> }
>
> Catching Throwable means that you are catching Error as well, and that is
not recommended [1]. The rule of thumb that need to follow when catching
Error is whether we can fully recover from an Error. If the answer is yes,
then its fine to catch Error (catch Throwable), otherwise catching
Exception will be enough.

AFAIU, only  notificationStrategy.execute(new NotificationContext(deviceId,
operation)); line is the concern here. Seems like operationMappingDAO.
updateOperationMapping is a DB operation. If so, then my suggestion is to
have 2 try-catch blocks. One for the operationMappingDAO.
updateOperationMapping line with catching appropriate DB operation
exception (SQLException). Another one for the notificationStrategy.execute(new
NotificationContext(deviceId, operation)); catching Throwable.

[1] https://stackoverflow.com/a/11018879/1577286

Thanks.

On Tue, Jan 23, 2018 at 8:23 PM, Geeth Munasinghe  wrote:

> Hi Charitha/Rasika
>
>
> On Tue, Jan 23, 2018 at 8:17 PM, Rasika Perera  wrote:
>
>> Good catch Charitha! As you explained this code might raise unforeseen
>> issues.
>>
>> However; if we are bringing the update operation mapping before the FCM
>> notification; as per [1] status of the push notification will be marked as
>> *COMPLETED* even before executing the notification sending process. And
>> this will cause another issue of an inconsistent state.
>>
>> Most of the time, these notification providers involves calling external
>> 3rd party libraries(eg. MQTT, FCM...etc). Suppose there's a
>> RuntimeException thrown while sending the push notification. Notification
>> status still will be marked as COMPLETED and will not try ever again since
>> it is not tracked with the current try-catch block(it only catches
>> PushNotificationExecutionFailedException).
>>
>> Thus, If we are changing the order as suggested please consider adding a
>> Throwable catch block(since notificationStrategy.execute() calls 3rd party
>> or untrusted libraries);
>>
>> try {
>> operationMappingDAO.updateOperationMapping(operationId, enrolmentId, 
>> org.wso2.carbon.device.mgt.core.dto.operation.mgt.Operation.PushNotificationStatus.COMPLETED);
>> notificationStrategy.execute(new NotificationContext(deviceId, 
>> operation));
>> } catch (Throwable e) {
>> log.error("Error occurred while sending push notifications to " +
>>   deviceId.getType() + " device carrying id '" +
>>   deviceId + "'", e);
>> // Reschedule if push notification failed.
>> operationMappingDAO.updateOperationMapping(operationId, enrolmentId, 
>> org.wso2.carbon.device.mgt.core.dto.operation.mgt.Operation.PushNotificationStatus.SCHEDULED);
>> }
>>
>>
> +1 for this approach.
>
>>
>> [1] https://github.com/wso2/carbon-device-mgt/blob/e56fb5f4d
>> e5d8f08af879a39a412ff501cba35fc/components/device-mgt/org.
>> wso2.carbon.device.mgt.core/src/main/java/org/wso2/carbon/
>> device/mgt/core/operation/mgt/OperationManagerImpl.java#L196
>>
>> On Tue, Jan 23, 2018 at 7:16 PM, Charitha Goonetilleke <
>> charit...@wso2.com> wrote:
>>
>>> Hi All,
>>>
>>> I have implemented state life cycle to prevent concurrent API call from
>>> agent to pending operations endpoint. However issue is still appearing. I
>>> have tested the same scenario with Update 10 and was able to reproduce the
>>> issue with Update 10 pack also.
>>>
>>> However as I figured out, we are sending FCM notification before
>>> updating operation mapping table[1]. Due to that device received FCM
>>> notification in realtime as same as it is being added to the Operations DB
>>> in the server. So there is chance to a raise condition and as a result, we
>>> are observing above exception. So we might need to send FCM notification
>>> after updating Operation Mapping in order to prevent that. WDYT?
>>>
>>> [1] https://github.com/wso2/carbon-device-mgt/blob/master/co
>>> mponents/device-mgt/org.wso2.carbon.device.mgt.core/src/main
>>> /java/org/wso2/carbon/device/mgt/core/operation/mgt/Operati
>>> onManagerImpl.java#L195
>>>
>>> Thanks & Regards,
>>> /charithag
>>>
>>> On Tue, Jan 23, 2018 at 2:34 PM, Charitha Goonetilleke <
>>> charit...@wso2.com> wrote:
>>>
 Hi All,

 I 

Re: [Dev] [IoTS] IoTS 3.1.0-Update11 throw intermittent exception due to concurrency issue.

[Geeth Munasinghe]

Hi Charitha/Rasika


On Tue, Jan 23, 2018 at 8:17 PM, Rasika Perera  wrote:

> Good catch Charitha! As you explained this code might raise unforeseen
> issues.
>
> However; if we are bringing the update operation mapping before the FCM
> notification; as per [1] status of the push notification will be marked as
> *COMPLETED* even before executing the notification sending process. And
> this will cause another issue of an inconsistent state.
>
> Most of the time, these notification providers involves calling external
> 3rd party libraries(eg. MQTT, FCM...etc). Suppose there's a
> RuntimeException thrown while sending the push notification. Notification
> status still will be marked as COMPLETED and will not try ever again since
> it is not tracked with the current try-catch block(it only catches
> PushNotificationExecutionFailedException).
>
> Thus, If we are changing the order as suggested please consider adding a
> Throwable catch block(since notificationStrategy.execute() calls 3rd party
> or untrusted libraries);
>
> try {
> operationMappingDAO.updateOperationMapping(operationId, enrolmentId, 
> org.wso2.carbon.device.mgt.core.dto.operation.mgt.Operation.PushNotificationStatus.COMPLETED);
> notificationStrategy.execute(new NotificationContext(deviceId, 
> operation));
> } catch (Throwable e) {
> log.error("Error occurred while sending push notifications to " +
>   deviceId.getType() + " device carrying id '" +
>   deviceId + "'", e);
> // Reschedule if push notification failed.
> operationMappingDAO.updateOperationMapping(operationId, enrolmentId, 
> org.wso2.carbon.device.mgt.core.dto.operation.mgt.Operation.PushNotificationStatus.SCHEDULED);
> }
>
>
+1 for this approach.

>
> [1] https://github.com/wso2/carbon-device-mgt/blob/
> e56fb5f4de5d8f08af879a39a412ff501cba35fc/components/device-
> mgt/org.wso2.carbon.device.mgt.core/src/main/java/org/
> wso2/carbon/device/mgt/core/operation/mgt/OperationManagerImpl.java#L196
>
> On Tue, Jan 23, 2018 at 7:16 PM, Charitha Goonetilleke  > wrote:
>
>> Hi All,
>>
>> I have implemented state life cycle to prevent concurrent API call from
>> agent to pending operations endpoint. However issue is still appearing. I
>> have tested the same scenario with Update 10 and was able to reproduce the
>> issue with Update 10 pack also.
>>
>> However as I figured out, we are sending FCM notification before updating
>> operation mapping table[1]. Due to that device received FCM notification in
>> realtime as same as it is being added to the Operations DB in the server.
>> So there is chance to a raise condition and as a result, we are observing
>> above exception. So we might need to send FCM notification after updating
>> Operation Mapping in order to prevent that. WDYT?
>>
>> [1] https://github.com/wso2/carbon-device-mgt/blob/master/co
>> mponents/device-mgt/org.wso2.carbon.device.mgt.core/src/
>> main/java/org/wso2/carbon/device/mgt/core/operation/mgt/Oper
>> ationManagerImpl.java#L195
>>
>> Thanks & Regards,
>> /charithag
>>
>> On Tue, Jan 23, 2018 at 2:34 PM, Charitha Goonetilleke <
>> charit...@wso2.com> wrote:
>>
>>> Hi All,
>>>
>>> I was able to figure out the root cause for the issue. As Geeth and
>>> Rasika pointed out this is happening due to concurrent insert and update
>>> query executions. The root cause for this is calling pending operations
>>> endpoint more than once from a single android device at a given time. Due
>>> to that concurrent insert and update happen in the DB level. Ideally a
>>> device should not call pending operations endpoint simultaneously. But as
>>> device may receive multiple FCM notifications within fractions of seconds,
>>> it is triggering pending operation calls per notification.
>>>
>>> So I'm now implementing a state machine to avoid that raise condition.
>>>
>>> Thanks & Regards,
>>> /charithag
>>>
>>> On Tue, Jan 23, 2018 at 2:14 PM, Rasika Perera  wrote:
>>>
 ​Hi All,

 This error has raised since two threads trying to ​update the DB with
 following requirements;

 Thread #1
 waiting to lock PUBLIC.*DM_ENROLMENT_OP_MAPPING* while
 locking PUBLIC.*DM_OPERATION* (exclusive), PUBLIC.
 *DM_COMMAND_OPERATION* (exclusive).

 Thread #2
 waiting to lock PUBLIC.DM_OPERATION while
 locking PUBLIC.*DM_ENROLMENT_OP_MAPPING* (exclusive), PUBLIC.
 *DM_DEVICE_OPERATION_RESPONSE* (exclusive).";

 I noticed we are already using ";LOCK_TIMEOUT=6" (60sec) H2 DB
 parameter which would allow sometime to continue the thread and do the DB
 update.

 Hence, this might happen when you are holding the debug pointer in
 updateOperation() method or in a state where device responses burst occurs.
 However the latter case is very rare on other DBs(other than embedded H2).
 We can even minimize the chance for a such issue implementing a LOCK for
 

Re: [Dev] [IoTS] IoTS 3.1.0-Update11 throw intermittent exception due to concurrency issue.

[Rasika Perera]

Good catch Charitha! As you explained this code might raise unforeseen
issues.

However; if we are bringing the update operation mapping before the FCM
notification; as per [1] status of the push notification will be marked as
*COMPLETED* even before executing the notification sending process. And
this will cause another issue of an inconsistent state.

Most of the time, these notification providers involves calling external
3rd party libraries(eg. MQTT, FCM...etc). Suppose there's a
RuntimeException thrown while sending the push notification. Notification
status still will be marked as COMPLETED and will not try ever again since
it is not tracked with the current try-catch block(it only catches
PushNotificationExecutionFailedException).

Thus, If we are changing the order as suggested please consider adding a
Throwable catch block(since notificationStrategy.execute() calls 3rd party
or untrusted libraries);

try {
operationMappingDAO.updateOperationMapping(operationId,
enrolmentId, 
org.wso2.carbon.device.mgt.core.dto.operation.mgt.Operation.PushNotificationStatus.COMPLETED);
notificationStrategy.execute(new NotificationContext(deviceId, operation));
} catch (Throwable e) {
log.error("Error occurred while sending push notifications to " +
  deviceId.getType() + " device carrying id '" +
  deviceId + "'", e);
// Reschedule if push notification failed.
operationMappingDAO.updateOperationMapping(operationId,
enrolmentId, 
org.wso2.carbon.device.mgt.core.dto.operation.mgt.Operation.PushNotificationStatus.SCHEDULED);
}


[1]
https://github.com/wso2/carbon-device-mgt/blob/e56fb5f4de5d8f08af879a39a412ff501cba35fc/components/device-mgt/org.wso2.carbon.device.mgt.core/src/main/java/org/wso2/carbon/device/mgt/core/operation/mgt/OperationManagerImpl.java#L196

On Tue, Jan 23, 2018 at 7:16 PM, Charitha Goonetilleke 
wrote:

> Hi All,
>
> I have implemented state life cycle to prevent concurrent API call from
> agent to pending operations endpoint. However issue is still appearing. I
> have tested the same scenario with Update 10 and was able to reproduce the
> issue with Update 10 pack also.
>
> However as I figured out, we are sending FCM notification before updating
> operation mapping table[1]. Due to that device received FCM notification in
> realtime as same as it is being added to the Operations DB in the server.
> So there is chance to a raise condition and as a result, we are observing
> above exception. So we might need to send FCM notification after updating
> Operation Mapping in order to prevent that. WDYT?
>
> [1] https://github.com/wso2/carbon-device-mgt/blob/master/
> components/device-mgt/org.wso2.carbon.device.mgt.core/
> src/main/java/org/wso2/carbon/device/mgt/core/operation/mgt/
> OperationManagerImpl.java#L195
>
> Thanks & Regards,
> /charithag
>
> On Tue, Jan 23, 2018 at 2:34 PM, Charitha Goonetilleke  > wrote:
>
>> Hi All,
>>
>> I was able to figure out the root cause for the issue. As Geeth and
>> Rasika pointed out this is happening due to concurrent insert and update
>> query executions. The root cause for this is calling pending operations
>> endpoint more than once from a single android device at a given time. Due
>> to that concurrent insert and update happen in the DB level. Ideally a
>> device should not call pending operations endpoint simultaneously. But as
>> device may receive multiple FCM notifications within fractions of seconds,
>> it is triggering pending operation calls per notification.
>>
>> So I'm now implementing a state machine to avoid that raise condition.
>>
>> Thanks & Regards,
>> /charithag
>>
>> On Tue, Jan 23, 2018 at 2:14 PM, Rasika Perera  wrote:
>>
>>> ​Hi All,
>>>
>>> This error has raised since two threads trying to ​update the DB with
>>> following requirements;
>>>
>>> Thread #1
>>> waiting to lock PUBLIC.*DM_ENROLMENT_OP_MAPPING* while
>>> locking PUBLIC.*DM_OPERATION* (exclusive), PUBLIC.*DM_COMMAND_OPERATION*
>>> (exclusive).
>>>
>>> Thread #2
>>> waiting to lock PUBLIC.DM_OPERATION while
>>> locking PUBLIC.*DM_ENROLMENT_OP_MAPPING* (exclusive), PUBLIC.
>>> *DM_DEVICE_OPERATION_RESPONSE* (exclusive).";
>>>
>>> I noticed we are already using ";LOCK_TIMEOUT=6" (60sec) H2 DB
>>> parameter which would allow sometime to continue the thread and do the DB
>>> update.
>>>
>>> Hence, this might happen when you are holding the debug pointer in
>>> updateOperation() method or in a state where device responses burst occurs.
>>> However the latter case is very rare on other DBs(other than embedded H2).
>>> We can even minimize the chance for a such issue implementing a LOCK for
>>> calling updateOperation method(might affect the performance). WDYT?
>>>
>>> Best Regards,
>>> ~Rasika
>>>
>>> On Tue, Jan 23, 2018 at 12:39 PM, Charitha Goonetilleke <
>>> charit...@wso2.com> wrote:
>>>
 Hi All,

 I have encountered following exception 

Re: [Dev] [IoTS] IoTS 3.1.0-Update11 throw intermittent exception due to concurrency issue.

[Charitha Goonetilleke]

Hi All,

I have implemented state life cycle to prevent concurrent API call from
agent to pending operations endpoint. However issue is still appearing. I
have tested the same scenario with Update 10 and was able to reproduce the
issue with Update 10 pack also.

However as I figured out, we are sending FCM notification before updating
operation mapping table[1]. Due to that device received FCM notification in
realtime as same as it is being added to the Operations DB in the server.
So there is chance to a raise condition and as a result, we are observing
above exception. So we might need to send FCM notification after updating
Operation Mapping in order to prevent that. WDYT?

[1]
https://github.com/wso2/carbon-device-mgt/blob/master/components/device-mgt/org.wso2.carbon.device.mgt.core/src/main/java/org/wso2/carbon/device/mgt/core/operation/mgt/OperationManagerImpl.java#L195

Thanks & Regards,
/charithag

On Tue, Jan 23, 2018 at 2:34 PM, Charitha Goonetilleke 
wrote:

> Hi All,
>
> I was able to figure out the root cause for the issue. As Geeth and Rasika
> pointed out this is happening due to concurrent insert and update query
> executions. The root cause for this is calling pending operations endpoint
> more than once from a single android device at a given time. Due to that
> concurrent insert and update happen in the DB level. Ideally a device
> should not call pending operations endpoint simultaneously. But as device
> may receive multiple FCM notifications within fractions of seconds, it is
> triggering pending operation calls per notification.
>
> So I'm now implementing a state machine to avoid that raise condition.
>
> Thanks & Regards,
> /charithag
>
> On Tue, Jan 23, 2018 at 2:14 PM, Rasika Perera  wrote:
>
>> ​Hi All,
>>
>> This error has raised since two threads trying to ​update the DB with
>> following requirements;
>>
>> Thread #1
>> waiting to lock PUBLIC.*DM_ENROLMENT_OP_MAPPING* while
>> locking PUBLIC.*DM_OPERATION* (exclusive), PUBLIC.*DM_COMMAND_OPERATION*
>> (exclusive).
>>
>> Thread #2
>> waiting to lock PUBLIC.DM_OPERATION while
>> locking PUBLIC.*DM_ENROLMENT_OP_MAPPING* (exclusive), PUBLIC.
>> *DM_DEVICE_OPERATION_RESPONSE* (exclusive).";
>>
>> I noticed we are already using ";LOCK_TIMEOUT=6" (60sec) H2 DB
>> parameter which would allow sometime to continue the thread and do the DB
>> update.
>>
>> Hence, this might happen when you are holding the debug pointer in
>> updateOperation() method or in a state where device responses burst occurs.
>> However the latter case is very rare on other DBs(other than embedded H2).
>> We can even minimize the chance for a such issue implementing a LOCK for
>> calling updateOperation method(might affect the performance). WDYT?
>>
>> Best Regards,
>> ~Rasika
>>
>> On Tue, Jan 23, 2018 at 12:39 PM, Charitha Goonetilleke <
>> charit...@wso2.com> wrote:
>>
>>> Hi All,
>>>
>>> I have encountered following exception intermittently on IoTS Update 11,
>>> when testing with Android device.
>>>
>>> [2018-01-23 12:19:51,077] [IoT-Core] ERROR -
>>> {org.wso2.carbon.mdm.services.android.services.impl.DeviceManagementServiceImpl}
>>> Issue in retrieving operation management service instance
>>> org.wso2.carbon.device.mgt.common.operation.mgt.OperationManagementException:
>>> Error occurred while updating the operation: 90 status:COMPLETED
>>> at org.wso2.carbon.device.mgt.core.operation.mgt.OperationManag
>>> erImpl.updateOperation(OperationManagerImpl.java:551)
>>> at org.wso2.carbon.device.mgt.core.service.DeviceManagementProv
>>> iderServiceImpl.updateOperation(DeviceManagementProviderServ
>>> iceImpl.java:1426)
>>> at org.wso2.carbon.mdm.services.android.util.AndroidDeviceUtils
>>> .updateOperation(AndroidDeviceUtils.java:211)
>>> at org.wso2.carbon.mdm.services.android.services.impl.DeviceMan
>>> agementServiceImpl.updateOperations(DeviceManagementServiceI
>>> mpl.java:184)
>>> at org.wso2.carbon.mdm.services.android.services.impl.DeviceMan
>>> agementServiceImpl.getPendingOperations(DeviceManagementServ
>>> iceImpl.java:139)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>>> ssorImpl.java:62)
>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>> thodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>> at org.apache.cxf.service.invoker.AbstractInvoker.performInvoca
>>> tion(AbstractInvoker.java:188)
>>> at org.apache.cxf.service.invoker.AbstractInvoker.invoke(Abstra
>>> ctInvoker.java:104)
>>> at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:204)
>>> at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:101)
>>> at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(S
>>> erviceInvokerInterceptor.java:58)
>>> at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleM
>>> essage(ServiceInvokerInterceptor.java:94)
>>> at 

Re: [Dev] [IoTS] IoTS 3.1.0-Update11 throw intermittent exception due to concurrency issue.

[Charitha Goonetilleke]

Hi All,

I was able to figure out the root cause for the issue. As Geeth and Rasika
pointed out this is happening due to concurrent insert and update query
executions. The root cause for this is calling pending operations endpoint
more than once from a single android device at a given time. Due to that
concurrent insert and update happen in the DB level. Ideally a device
should not call pending operations endpoint simultaneously. But as device
may receive multiple FCM notifications within fractions of seconds, it is
triggering pending operation calls per notification.

So I'm now implementing a state machine to avoid that raise condition.

Thanks & Regards,
/charithag

On Tue, Jan 23, 2018 at 2:14 PM, Rasika Perera  wrote:

> ​Hi All,
>
> This error has raised since two threads trying to ​update the DB with
> following requirements;
>
> Thread #1
> waiting to lock PUBLIC.*DM_ENROLMENT_OP_MAPPING* while
> locking PUBLIC.*DM_OPERATION* (exclusive), PUBLIC.*DM_COMMAND_OPERATION*
> (exclusive).
>
> Thread #2
> waiting to lock PUBLIC.DM_OPERATION while
> locking PUBLIC.*DM_ENROLMENT_OP_MAPPING* (exclusive), PUBLIC.
> *DM_DEVICE_OPERATION_RESPONSE* (exclusive).";
>
> I noticed we are already using ";LOCK_TIMEOUT=6" (60sec) H2 DB
> parameter which would allow sometime to continue the thread and do the DB
> update.
>
> Hence, this might happen when you are holding the debug pointer in
> updateOperation() method or in a state where device responses burst occurs.
> However the latter case is very rare on other DBs(other than embedded H2).
> We can even minimize the chance for a such issue implementing a LOCK for
> calling updateOperation method(might affect the performance). WDYT?
>
> Best Regards,
> ~Rasika
>
> On Tue, Jan 23, 2018 at 12:39 PM, Charitha Goonetilleke <
> charit...@wso2.com> wrote:
>
>> Hi All,
>>
>> I have encountered following exception intermittently on IoTS Update 11,
>> when testing with Android device.
>>
>> [2018-01-23 12:19:51,077] [IoT-Core] ERROR -
>> {org.wso2.carbon.mdm.services.android.services.impl.DeviceManagementServiceImpl}
>> Issue in retrieving operation management service instance
>> org.wso2.carbon.device.mgt.common.operation.mgt.OperationManagementException:
>> Error occurred while updating the operation: 90 status:COMPLETED
>> at org.wso2.carbon.device.mgt.core.operation.mgt.OperationManag
>> erImpl.updateOperation(OperationManagerImpl.java:551)
>> at org.wso2.carbon.device.mgt.core.service.DeviceManagementProv
>> iderServiceImpl.updateOperation(DeviceManagementProviderServ
>> iceImpl.java:1426)
>> at org.wso2.carbon.mdm.services.android.util.AndroidDeviceUtils
>> .updateOperation(AndroidDeviceUtils.java:211)
>> at org.wso2.carbon.mdm.services.android.services.impl.DeviceMan
>> agementServiceImpl.updateOperations(DeviceManagementServiceImpl.java:184)
>> at org.wso2.carbon.mdm.services.android.services.impl.DeviceMan
>> agementServiceImpl.getPendingOperations(DeviceManagementServ
>> iceImpl.java:139)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>> ssorImpl.java:62)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>> thodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at org.apache.cxf.service.invoker.AbstractInvoker.performInvoca
>> tion(AbstractInvoker.java:188)
>> at org.apache.cxf.service.invoker.AbstractInvoker.invoke(Abstra
>> ctInvoker.java:104)
>> at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:204)
>> at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:101)
>> at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(S
>> erviceInvokerInterceptor.java:58)
>> at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleM
>> essage(ServiceInvokerInterceptor.java:94)
>> at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(Phase
>> InterceptorChain.java:272)
>> at org.apache.cxf.transport.ChainInitiationObserver.onMessage(C
>> hainInitiationObserver.java:121)
>> at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke
>> (AbstractHTTPDestination.java:249)
>> at org.apache.cxf.transport.servlet.ServletController.invokeDes
>> tination(ServletController.java:248)
>> at org.apache.cxf.transport.servlet.ServletController.invoke(Se
>> rvletController.java:222)
>> at org.apache.cxf.transport.servlet.ServletController.invoke(Se
>> rvletController.java:153)
>> at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(
>> CXFNonSpringServlet.java:171)
>> at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleR
>> equest(AbstractHTTPServlet.java:289)
>> at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPut(A
>> bstractHTTPServlet.java:226)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:653)
>> at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service
>> (AbstractHTTPServlet.java:265)
>> at 

Re: [Dev] [IoTS] IoTS 3.1.0-Update11 throw intermittent exception due to concurrency issue.

[Rasika Perera]

​Hi All,

This error has raised since two threads trying to ​update the DB with
following requirements;

Thread #1
waiting to lock PUBLIC.*DM_ENROLMENT_OP_MAPPING* while
locking PUBLIC.*DM_OPERATION* (exclusive), PUBLIC.*DM_COMMAND_OPERATION*
(exclusive).

Thread #2
waiting to lock PUBLIC.DM_OPERATION while
locking PUBLIC.*DM_ENROLMENT_OP_MAPPING* (exclusive), PUBLIC.
*DM_DEVICE_OPERATION_RESPONSE* (exclusive).";

I noticed we are already using ";LOCK_TIMEOUT=6" (60sec) H2 DB
parameter which would allow sometime to continue the thread and do the DB
update.

Hence, this might happen when you are holding the debug pointer in
updateOperation() method or in a state where device responses burst occurs.
However the latter case is very rare on other DBs(other than embedded H2).
We can even minimize the chance for a such issue implementing a LOCK for
calling updateOperation method(might affect the performance). WDYT?

Best Regards,
~Rasika

On Tue, Jan 23, 2018 at 12:39 PM, Charitha Goonetilleke 
wrote:

> Hi All,
>
> I have encountered following exception intermittently on IoTS Update 11,
> when testing with Android device.
>
> [2018-01-23 12:19:51,077] [IoT-Core] ERROR - {org.wso2.carbon.mdm.services.
> android.services.impl.DeviceManagementServiceImpl} Issue in retrieving
> operation management service instance
> org.wso2.carbon.device.mgt.common.operation.mgt.OperationManagementException:
> Error occurred while updating the operation: 90 status:COMPLETED
> at org.wso2.carbon.device.mgt.core.operation.mgt.OperationManag
> erImpl.updateOperation(OperationManagerImpl.java:551)
> at org.wso2.carbon.device.mgt.core.service.DeviceManagementProv
> iderServiceImpl.updateOperation(DeviceManagementProviderServiceImpl.java:
> 1426)
> at org.wso2.carbon.mdm.services.android.util.AndroidDeviceUtils
> .updateOperation(AndroidDeviceUtils.java:211)
> at org.wso2.carbon.mdm.services.android.services.impl.DeviceMan
> agementServiceImpl.updateOperations(DeviceManagementServiceImpl.java:184)
> at org.wso2.carbon.mdm.services.android.services.impl.DeviceMan
> agementServiceImpl.getPendingOperations(DeviceManagementServ
> iceImpl.java:139)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
> ssorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
> thodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.cxf.service.invoker.AbstractInvoker.performInvoca
> tion(AbstractInvoker.java:188)
> at org.apache.cxf.service.invoker.AbstractInvoker.invoke(
> AbstractInvoker.java:104)
> at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:204)
> at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:101)
> at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(
> ServiceInvokerInterceptor.java:58)
> at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleM
> essage(ServiceInvokerInterceptor.java:94)
> at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(Phase
> InterceptorChain.java:272)
> at org.apache.cxf.transport.ChainInitiationObserver.onMessage(C
> hainInitiationObserver.java:121)
> at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke
> (AbstractHTTPDestination.java:249)
> at org.apache.cxf.transport.servlet.ServletController.invokeDes
> tination(ServletController.java:248)
> at org.apache.cxf.transport.servlet.ServletController.invoke(
> ServletController.java:222)
> at org.apache.cxf.transport.servlet.ServletController.invoke(
> ServletController.java:153)
> at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(
> CXFNonSpringServlet.java:171)
> at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleR
> equest(AbstractHTTPServlet.java:289)
> at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPut(
> AbstractHTTPServlet.java:226)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:653)
> at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service
> (AbstractHTTPServlet.java:265)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFi
> lter(ApplicationFilterChain.java:303)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(App
> licationFilterChain.java:208)
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFi
> lter(ApplicationFilterChain.java:241)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(App
> licationFilterChain.java:208)
> at org.wso2.carbon.mdm.services.android.util.ApiOriginFilter.do
> Filter(ApiOriginFilter.java:33)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFi
> lter(ApplicationFilterChain.java:241)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(App
> licationFilterChain.java:208)
> at org.apache.catalina.core.StandardWrapperValve.invoke(Standar
> dWrapperValve.java:218)
> at 

Re: [Dev] [IoTS] IoTS 3.1.0-Update11 throw intermittent exception due to concurrency issue.

[Geeth Munasinghe]

Hi Charitha,

When an operation response get stored DM_ENROLMENT_OP_MAPPING table gets an
"update" while DM_DEVICE_OPERATION_RESPONSE gets an "insert" from the
response data. AFAIK when an update happens it will lock the table,
depending on the transactions handling [1 - Transaction Isolation] of H2
database.  And if there is a response burst, this error could happen. I
think H2 database may not be able to handle the load.

[1] http://www.h2database.com/html/advanced.html


Thanks
Geeth


On Tue, Jan 23, 2018 at 12:39 PM, Charitha Goonetilleke 
wrote:

> Hi All,
>
> I have encountered following exception intermittently on IoTS Update 11,
> when testing with Android device.
>
> [2018-01-23 12:19:51,077] [IoT-Core] ERROR - {org.wso2.carbon.mdm.services.
> android.services.impl.DeviceManagementServiceImpl} Issue in retrieving
> operation management service instance
> org.wso2.carbon.device.mgt.common.operation.mgt.OperationManagementException:
> Error occurred while updating the operation: 90 status:COMPLETED
> at org.wso2.carbon.device.mgt.core.operation.mgt.OperationManagerImpl.
> updateOperation(OperationManagerImpl.java:551)
> at org.wso2.carbon.device.mgt.core.service.DeviceManagementProviderServic
> eImpl.updateOperation(DeviceManagementProviderServiceImpl.java:1426)
> at org.wso2.carbon.mdm.services.android.util.AndroidDeviceUtils.
> updateOperation(AndroidDeviceUtils.java:211)
> at org.wso2.carbon.mdm.services.android.services.impl.
> DeviceManagementServiceImpl.updateOperations(DeviceManagementServiceImpl.
> java:184)
> at org.wso2.carbon.mdm.services.android.services.impl.
> DeviceManagementServiceImpl.getPendingOperations(
> DeviceManagementServiceImpl.java:139)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(
> AbstractInvoker.java:188)
> at org.apache.cxf.service.invoker.AbstractInvoker.
> invoke(AbstractInvoker.java:104)
> at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:204)
> at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:101)
> at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.
> run(ServiceInvokerInterceptor.java:58)
> at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(
> ServiceInvokerInterceptor.java:94)
> at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(
> PhaseInterceptorChain.java:272)
> at org.apache.cxf.transport.ChainInitiationObserver.onMessage(
> ChainInitiationObserver.java:121)
> at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(
> AbstractHTTPDestination.java:249)
> at org.apache.cxf.transport.servlet.ServletController.invokeDestination(
> ServletController.java:248)
> at org.apache.cxf.transport.servlet.ServletController.
> invoke(ServletController.java:222)
> at org.apache.cxf.transport.servlet.ServletController.
> invoke(ServletController.java:153)
> at org.apache.cxf.transport.servlet.CXFNonSpringServlet.
> invoke(CXFNonSpringServlet.java:171)
> at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(
> AbstractHTTPServlet.java:289)
> at org.apache.cxf.transport.servlet.AbstractHTTPServlet.
> doPut(AbstractHTTPServlet.java:226)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:653)
> at org.apache.cxf.transport.servlet.AbstractHTTPServlet.
> service(AbstractHTTPServlet.java:265)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
> ApplicationFilterChain.java:303)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(
> ApplicationFilterChain.java:208)
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
> ApplicationFilterChain.java:241)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(
> ApplicationFilterChain.java:208)
> at org.wso2.carbon.mdm.services.android.util.ApiOriginFilter.
> doFilter(ApiOriginFilter.java:33)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
> ApplicationFilterChain.java:241)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(
> ApplicationFilterChain.java:208)
> at org.apache.catalina.core.StandardWrapperValve.invoke(
> StandardWrapperValve.java:218)
> at org.apache.catalina.core.StandardContextValve.invoke(
> StandardContextValve.java:110)
> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(
> AuthenticatorBase.java:506)
> at org.apache.catalina.core.StandardHostValve.invoke(
> StandardHostValve.java:169)
> at org.apache.catalina.valves.ErrorReportValve.invoke(
> ErrorReportValve.java:103)
> at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(
> CompositeValve.java:99)
> at 

[Dev] [IoTS] IoTS 3.1.0-Update11 throw intermittent exception due to concurrency issue.

[Charitha Goonetilleke]

Hi All,

I have encountered following exception intermittently on IoTS Update 11,
when testing with Android device.

[2018-01-23 12:19:51,077] [IoT-Core] ERROR -
{org.wso2.carbon.mdm.services.android.services.impl.DeviceManagementServiceImpl}
Issue in retrieving operation management service instance
org.wso2.carbon.device.mgt.common.operation.mgt.OperationManagementException:
Error occurred while updating the operation: 90 status:COMPLETED
at
org.wso2.carbon.device.mgt.core.operation.mgt.OperationManagerImpl.updateOperation(OperationManagerImpl.java:551)
at
org.wso2.carbon.device.mgt.core.service.DeviceManagementProviderServiceImpl.updateOperation(DeviceManagementProviderServiceImpl.java:1426)
at
org.wso2.carbon.mdm.services.android.util.AndroidDeviceUtils.updateOperation(AndroidDeviceUtils.java:211)
at
org.wso2.carbon.mdm.services.android.services.impl.DeviceManagementServiceImpl.updateOperations(DeviceManagementServiceImpl.java:184)
at
org.wso2.carbon.mdm.services.android.services.impl.DeviceManagementServiceImpl.getPendingOperations(DeviceManagementServiceImpl.java:139)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:188)
at
org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:104)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:204)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:101)
at
org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58)
at
org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:94)
at
org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:272)
at
org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
at
org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:249)
at
org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:248)
at
org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:222)
at
org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:153)
at
org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:171)
at
org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:289)
at
org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPut(AbstractHTTPServlet.java:226)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:653)
at
org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:265)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.wso2.carbon.mdm.services.android.util.ApiOriginFilter.doFilter(ApiOriginFilter.java:33)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:506)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at
org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:99)
at
org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
at
org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:57)
at
org.wso2.carbon.webapp.authenticator.framework.WebappAuthenticationValve.processRequest(WebappAuthenticationValve.java:151)
at
org.wso2.carbon.webapp.authenticator.framework.WebappAuthenticationValve.invoke(WebappAuthenticationValve.java:69)
at
org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
at
org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:62)
at
org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:159)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
at