[kudu-CR] [Java] Fix concurrent problem while traversing the traces of KuduRpc.
Alexey Serbin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21041 ) Change subject: [Java] Fix concurrent problem while traversing the traces of KuduRpc. .. [Java] Fix concurrent problem while traversing the traces of KuduRpc. When the RPC request which is sent by java client times out, the RpcTimeoutTask is called by HashedWheelTimer. During the running of this task, KuduRpc#toString() is called. This method will traverse the traces(a synchronizedList) without any protection, which leads to exception in timeout task and then the callback will not be triggered. The exception message is as follow: An exception was thrown by TimerTask. java.util.ConcurrentModificationException: null at java.util.ArrayList$Itr.checkForComodification at java.util.ArrayList$Itr.next at org.apache.kudu.client.RpcTraceFrame .getHumanReadableSummaryStringForTraces at org.apache.kudu.client.KuduRpc.toString at java.lang.String.valueOf at java.lang.StringBuilder.append at org.apache.kudu.client.KuduRpc$RpcTimeoutTask.run at org.apache.kudu.shaded.io.netty.util .HashedWheelTimer$HashedWheelTimeout.expire So make a copy of the traces and then traverse the copy. This method is mostly called when some exceptions are raised, so the influence on performance should be acceptable. Change-Id: I1a642d93d01ebcfa05b01fe263023b5580d2542b Reviewed-on: http://gerrit.cloudera.org:8080/21041 Reviewed-by: Alexey Serbin Tested-by: Alexey Serbin --- M java/kudu-client/src/main/java/org/apache/kudu/client/KuduRpc.java 1 file changed, 10 insertions(+), 2 deletions(-) Approvals: Alexey Serbin: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/21041 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I1a642d93d01ebcfa05b01fe263023b5580d2542b Gerrit-Change-Number: 21041 Gerrit-PatchSet: 10 Gerrit-Owner: Song Jiacheng <599042...@qq.com> Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Song Jiacheng <599042...@qq.com>
[kudu-CR] [Java] Fix concurrent problem while traversing the traces of KuduRpc.
Alexey Serbin has removed a vote on this change. Change subject: [Java] Fix concurrent problem while traversing the traces of KuduRpc. .. Removed Verified-1 by Kudu Jenkins (120) -- To view, visit http://gerrit.cloudera.org:8080/21041 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I1a642d93d01ebcfa05b01fe263023b5580d2542b Gerrit-Change-Number: 21041 Gerrit-PatchSet: 9 Gerrit-Owner: Song Jiacheng <599042...@qq.com> Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Song Jiacheng <599042...@qq.com>
[kudu-CR] [Java] Fix concurrent problem while traversing the traces of KuduRpc.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/21041 ) Change subject: [Java] Fix concurrent problem while traversing the traces of KuduRpc. .. Patch Set 9: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/21041 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1a642d93d01ebcfa05b01fe263023b5580d2542b Gerrit-Change-Number: 21041 Gerrit-PatchSet: 9 Gerrit-Owner: Song Jiacheng <599042...@qq.com> Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Song Jiacheng <599042...@qq.com> Gerrit-Comment-Date: Fri, 23 Feb 2024 01:46:09 + Gerrit-HasComments: No
[kudu-CR] [Java] Fix concurrent problem while traversing the traces of KuduRpc.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/21041 ) Change subject: [Java] Fix concurrent problem while traversing the traces of KuduRpc. .. Patch Set 9: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21041 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1a642d93d01ebcfa05b01fe263023b5580d2542b Gerrit-Change-Number: 21041 Gerrit-PatchSet: 9 Gerrit-Owner: Song Jiacheng <599042...@qq.com> Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Song Jiacheng <599042...@qq.com> Gerrit-Comment-Date: Fri, 23 Feb 2024 01:45:50 + Gerrit-HasComments: No
[kudu-CR] [Java] Fix concurrent problem while traversing the traces of KuduRpc.
Song Jiacheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/21041 ) Change subject: [Java] Fix concurrent problem while traversing the traces of KuduRpc. .. Patch Set 9: (2 comments) > Patch Set 8: > > (2 comments) > > Thank you for the fix! Thanks for the review! http://gerrit.cloudera.org:8080/#/c/21041/8//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21041/8//COMMIT_MSG@10 PS8, Line 10: HashedWheelTimer. > HashedWheelTimer Done http://gerrit.cloudera.org:8080/#/c/21041/8//COMMIT_MSG@12 PS8, Line 12: which leads to : exception in timeout task > Could you add an example of such a trace into the commit description? It c Sure, i have added the stacktrace into the commit message. I think it should be the late response. I took a quick look at the method addTrace() and it could only be called by response at that time. -- To view, visit http://gerrit.cloudera.org:8080/21041 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1a642d93d01ebcfa05b01fe263023b5580d2542b Gerrit-Change-Number: 21041 Gerrit-PatchSet: 9 Gerrit-Owner: Song Jiacheng Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Song Jiacheng Gerrit-Comment-Date: Thu, 22 Feb 2024 06:29:31 + Gerrit-HasComments: Yes
[kudu-CR] [Java] Fix concurrent problem while traversing the traces of KuduRpc.
Hello Alexey Serbin, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21041 to look at the new patch set (#9). Change subject: [Java] Fix concurrent problem while traversing the traces of KuduRpc. .. [Java] Fix concurrent problem while traversing the traces of KuduRpc. When the RPC request which is sent by java client times out, the RpcTimeoutTask is called by HashedWheelTimer. During the running of this task, KuduRpc#toString() is called. This method will traverse the traces(a synchronizedList) without any protection, which leads to exception in timeout task and then the callback will not be triggered. The exception message is as follow: An exception was thrown by TimerTask. java.util.ConcurrentModificationException: null at java.util.ArrayList$Itr.checkForComodification at java.util.ArrayList$Itr.next at org.apache.kudu.client.RpcTraceFrame .getHumanReadableSummaryStringForTraces at org.apache.kudu.client.KuduRpc.toString at java.lang.String.valueOf at java.lang.StringBuilder.append at org.apache.kudu.client.KuduRpc$RpcTimeoutTask.run at org.apache.kudu.shaded.io.netty.util .HashedWheelTimer$HashedWheelTimeout.expire So make a copy of the traces and then traverse the copy. This method is mostly called when some exceptions are raised, so the influence on performance should be acceptable. Change-Id: I1a642d93d01ebcfa05b01fe263023b5580d2542b --- M java/kudu-client/src/main/java/org/apache/kudu/client/KuduRpc.java 1 file changed, 10 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/41/21041/9 -- To view, visit http://gerrit.cloudera.org:8080/21041 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1a642d93d01ebcfa05b01fe263023b5580d2542b Gerrit-Change-Number: 21041 Gerrit-PatchSet: 9 Gerrit-Owner: Song Jiacheng Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [Java] Fix concurrent problem while traversing the traces of KuduRpc.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/21041 ) Change subject: [Java] Fix concurrent problem while traversing the traces of KuduRpc. .. Patch Set 8: (2 comments) Thank you for the fix! http://gerrit.cloudera.org:8080/#/c/21041/8//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21041/8//COMMIT_MSG@10 PS8, Line 10: HashedWheelTimeout HashedWheelTimer http://gerrit.cloudera.org:8080/#/c/21041/8//COMMIT_MSG@12 PS8, Line 12: which leads to : exception in timeout task Could you add an example of such a trace into the commit description? It could help people to identify the problem if they hit it. BTW, what is the other actors that might be modifying the collection of traces during that time? Are those only related to late responses that arrive past the timeout or that might be something else? -- To view, visit http://gerrit.cloudera.org:8080/21041 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1a642d93d01ebcfa05b01fe263023b5580d2542b Gerrit-Change-Number: 21041 Gerrit-PatchSet: 8 Gerrit-Owner: Song Jiacheng Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Comment-Date: Tue, 20 Feb 2024 17:13:58 + Gerrit-HasComments: Yes
[kudu-CR] [Java] Fix concurrent problem while traversing the traces of KuduRpc.
Hello Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21041 to look at the new patch set (#8). Change subject: [Java] Fix concurrent problem while traversing the traces of KuduRpc. .. [Java] Fix concurrent problem while traversing the traces of KuduRpc. When the RPC request which is sent by java client times out, the RpcTimeoutTask is called by HashedWheelTimeout. During the running of this task, KuduRpc#toString() is called. This method will traverse the traces(a synchronizedList) without any protection, which leads to exception in timeout task and then the callback will not be triggered. So make a copy of the traces and then traverse the copy. This method is mostly called when some exceptions are raised, so the influence on performance should be acceptable. Change-Id: I1a642d93d01ebcfa05b01fe263023b5580d2542b --- M java/kudu-client/src/main/java/org/apache/kudu/client/KuduRpc.java 1 file changed, 10 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/41/21041/8 -- To view, visit http://gerrit.cloudera.org:8080/21041 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1a642d93d01ebcfa05b01fe263023b5580d2542b Gerrit-Change-Number: 21041 Gerrit-PatchSet: 8 Gerrit-Owner: Song Jiacheng Gerrit-Reviewer: Kudu Jenkins (120)
