Hao Zhang created HELIX-690:
-------------------------------
Summary: Batch message should not share same NotificationContext
object to update CurrentState
Key: HELIX-690
URL: https://issues.apache.org/jira/browse/HELIX-690
Project: Apache Helix
Issue Type: Bug
Reporter: Hao Zhang
Currently batch message has bugs:
1. Batch message is triggering a lot of duplicated state transition messages
sent from controller, result in "state does not match" error on participant
side. This will further create a lot of ERROR znodes in ZK, which adds up both
read/write workload in participant and controller
2. We see a lot of concurrent update exceptions as well
{noformat}
9909348:[2018-03-30 18:59:55,025] [ERROR] [pool-1-thread-1917]
[org.apache.helix.messaging.handling.HelixTask:113] - Exception while executing
a message. java.util.ConcurrentModificat
ionException msgId: fbdc37d4-ec95-47cb-950c-f9d3d224bbb3 type: STATE_TRANSITION
9909349-java.util.ConcurrentModificationException
9909350- at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1115)
9909351- at java.util.TreeMap$KeyIterator.next(TreeMap.java:1169)
9909352- at org.apache.helix.ZNRecord.merge(ZNRecord.java:497)
9909353- at org.apache.helix.GroupCommit.commit(GroupCommit.java:121)
9909354- at
org.apache.helix.manager.zk.ZKHelixDataAccessor.updateProperty(ZKHelixDataAccessor.java:182)
9909355- at
org.apache.helix.manager.zk.ZKHelixDataAccessor.updateProperty(ZKHelixDataAccessor.java:170)
9909356- at
org.apache.helix.messaging.handling.BatchMessageHandler.postHandleMessage(BatchMessageHandler.java:118)
9909357- at
org.apache.helix.messaging.handling.BatchMessageHandler.handleMessage(BatchMessageHandler.java:203)
9909358- at
org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:96)
{noformat}
The above 2 errors are resulted in the fact that in HelixTaskExecutor, all
HelixTask objects from same batch of messages are sharing the same
changeContext object. For batch message, HelixTask will create current state
update map to record current state updates, and therefore result in a racing
condition in current state recording - it is very normal that due to such bug,
resource's current state is changed on participant side, current state is not
updated in ZK, and after message is removed, controller still think that state
transition is not finished, and send duplicated state transition message.
To fix the issue, we will create hard copies of NotificationContext object for
each HelixTask in HelixTaskExecutor
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
