Todd Lipcon has uploaded a new change for review. http://gerrit.cloudera.org:8080/3456
Change subject: Fix stray memory writes due to tcmalloc profiling ...................................................................... Fix stray memory writes due to tcmalloc profiling This fixes an issue that has been causing frequent crashes in JD.com's production cluster as well as various Cloudera test clusters. The crashes would be in various different places, but the key signature was that offset 120 in some data structure or array (eg the 16th element of a vector) would be corrupted. After doing a git bisect using an integration testing cluster running an ITBLL workload, I found that this was a regression caused by the introduction of tcmalloc contention profiling[1]. The short explanation is that, if we experienced contention while freeing a Trace object, we could in some cases increment offset 120 of some other allocation which occurred soon after the deallocation of the Trace. The issue is described in more detail in a new comment in trace.h. With this patch, I was unable to reproduce the issue on the test cluster. No new test is added since this is quite timing-dependent and not amenable to unit testing or even stress testing. [1] commit f6691e744b9cb796e1bbc6e07953f21f387c9a88 Change-Id: I9afca83d9cc24585960f6bf68d8996c4736ce6cb Reviewed-on: http://gerrit.cloudera.org:8080/3445 Reviewed-by: David Ribeiro Alves <dral...@apache.org> Reviewed-by: Mike Percy <mpe...@apache.org> Tested-by: Kudu Jenkins (cherry picked from commit 0bb59836fb3e4b92e2d88674e8b88546faac5c8b) --- M src/kudu/util/trace.h 1 file changed, 23 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/56/3456/1 -- To view, visit http://gerrit.cloudera.org:8080/3456 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I9afca83d9cc24585960f6bf68d8996c4736ce6cb Gerrit-PatchSet: 1 Gerrit-Project: kudu Gerrit-Branch: branch-0.9.x Gerrit-Owner: Todd Lipcon <t...@apache.org>
