[kudu-CR] revert change to exactly once writes-itest
Andrew Wong has abandoned this change. ( http://gerrit.cloudera.org:8080/11836 ) Change subject: revert change to exactly_once_writes-itest .. Abandoned This "leak" was a false positive, see c5665046cb9f441e096650b8d10154598cf80fb8 for more details -- To view, visit http://gerrit.cloudera.org:8080/11836 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: If4091d3905d871acb48ec4d88c7b81ee48bf0eed Gerrit-Change-Number: 11836 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Wong Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy
[kudu-CR] revert change to exactly once writes-itest
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11836 ) Change subject: revert change to exactly_once_writes-itest .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/11836/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11836/2//COMMIT_MSG@20 PS2, Line 20: which we don't today > Leaks don't cause test failures? When did that change? I wonder if it's whe It seems in some cases, no. I'm still trying to get to the bottom of this, but what I think I'm seeing is that in the EMC process complains about a leak throughout the test, and at the very end when doing the leak check, the leak is somehow now caught. In many, if not most cases, the leak will be caught by the leak check that happens in the shutdown of ExternalDaemon. -- To view, visit http://gerrit.cloudera.org:8080/11836 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4091d3905d871acb48ec4d88c7b81ee48bf0eed Gerrit-Change-Number: 11836 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Wong Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Comment-Date: Thu, 01 Nov 2018 22:12:26 + Gerrit-HasComments: Yes
[kudu-CR] revert change to exactly once writes-itest
Mike Percy has posted comments on this change. ( http://gerrit.cloudera.org:8080/11836 ) Change subject: revert change to exactly_once_writes-itest .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/11836/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11836/2//COMMIT_MSG@20 PS2, Line 20: which we don't today Leaks don't cause test failures? When did that change? I wonder if it's when we switched from the tcmalloc leak checker to LSAN? Isn't that something we want? -- To view, visit http://gerrit.cloudera.org:8080/11836 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4091d3905d871acb48ec4d88c7b81ee48bf0eed Gerrit-Change-Number: 11836 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Wong Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Comment-Date: Wed, 31 Oct 2018 19:48:40 + Gerrit-HasComments: Yes
[kudu-CR] revert change to exactly once writes-itest
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11836 ) Change subject: revert change to exactly_once_writes-itest .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/11836/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11836/1//COMMIT_MSG@14 PS1, Line 14: So instead, I'm reverting the test change that led : to its leakiness. > You're right about the Jira; I've commented about it and added a log. Since Ah indeed, I'll look into getting to the bottom of this through those other leaks. -- To view, visit http://gerrit.cloudera.org:8080/11836 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4091d3905d871acb48ec4d88c7b81ee48bf0eed Gerrit-Change-Number: 11836 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Wong Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Comment-Date: Wed, 31 Oct 2018 05:01:20 + Gerrit-HasComments: Yes
[kudu-CR] revert change to exactly once writes-itest
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11836 ) Change subject: revert change to exactly_once_writes-itest .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/11836/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11836/1//COMMIT_MSG@9 PS1, Line 9: bc817a44867c586bf4e0539aa564b282c666a49d > Wrong hash; I think you meant bc817a44867c586bf4e0539aa564b282c666a49d here Done http://gerrit.cloudera.org:8080/#/c/11836/1//COMMIT_MSG@14 PS1, Line 14: So instead, I'm reverting the test change that led : to its leakiness. > But the changes you're reverting are to the wrong test; the leak was report You're right about the Jira; I've commented about it and added a log. Since filing the original ticket, I haven't seen that same flakiness. Yeah, I wonder if the spirit of that change implies that we should be ignoring these "transient" leaks, if they don't persistently show up in that final leak check. I'll bug Todd about it. I tried with that in ASAN_OPTIONS, I can try with LSAN_OPTIONS. It's notable though that given that the test aims to churn elections, slowing things down like this makes it pretty unlikely that the test will pass. -- To view, visit http://gerrit.cloudera.org:8080/11836 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4091d3905d871acb48ec4d88c7b81ee48bf0eed Gerrit-Change-Number: 11836 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Wong Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Comment-Date: Wed, 31 Oct 2018 05:00:15 + Gerrit-HasComments: Yes
[kudu-CR] revert change to exactly once writes-itest
Hello Mike Percy, Kudu Jenkins, Adar Dembo, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11836 to look at the new patch set (#2). Change subject: revert change to exactly_once_writes-itest .. revert change to exactly_once_writes-itest Since bc817a44867c586bf4e0539aa564b282c666a49d, in some environments, exactly_once_writes-itest will pass successfully but report a leak that would not fail the test itself. Try as I might to get to the bottom of the non-fatal leak, its root cause has evaded me. So instead, I'm reverting the test change that led to its leakiness. It seems it only fails some of the time when running a full ctest suite locally (i.e. not through dist-test). Some conclusions I've drawn about the leak: * Not easily reproducible via dist-test, even if we begin reporting LeakSanitizer errors on passing tests (which we don't today) * Not easily reproducible when running the test locally on its own * Not deterministically reproducible when running a full ctest suite locally Without the change, the leak was detected in 2/7 runs of the full test suite, and with the change, in 0/14. Change-Id: If4091d3905d871acb48ec4d88c7b81ee48bf0eed --- M src/kudu/integration-tests/exactly_once_writes-itest.cc 1 file changed, 2 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/36/11836/2 -- To view, visit http://gerrit.cloudera.org:8080/11836 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If4091d3905d871acb48ec4d88c7b81ee48bf0eed Gerrit-Change-Number: 11836 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Wong Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy
[kudu-CR] revert change to exactly once writes-itest
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/11836 ) Change subject: revert change to exactly_once_writes-itest .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/11836/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11836/1//COMMIT_MSG@14 PS1, Line 14: So instead, I'm reverting the test change that led : to its leakiness. > But the changes you're reverting are to the wrong test; the leak was report Also, a couple already filed JIRAs have leaks that seem eerily similar to this: KUDU-2608, KUDU-2607, KUDU-2594, and maybe KUDU-2583? If they're all due to the same underlying leak, you can try to repro with fast_unwind_on_malloc=0 in more than just exactly_once_writes-itest. -- To view, visit http://gerrit.cloudera.org:8080/11836 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4091d3905d871acb48ec4d88c7b81ee48bf0eed Gerrit-Change-Number: 11836 Gerrit-PatchSet: 1 Gerrit-Owner: Andrew Wong Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Comment-Date: Wed, 31 Oct 2018 04:42:49 + Gerrit-HasComments: Yes
[kudu-CR] revert change to exactly once writes-itest
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/11836 ) Change subject: revert change to exactly_once_writes-itest .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/11836/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11836/1//COMMIT_MSG@9 PS1, Line 9: 5894af6ff12291ffab2d3304b34f77fe6d112771 Wrong hash; I think you meant bc817a44867c586bf4e0539aa564b282c666a49d here. http://gerrit.cloudera.org:8080/#/c/11836/1//COMMIT_MSG@14 PS1, Line 14: So instead, I'm reverting the test change that led : to its leakiness. But the changes you're reverting are to the wrong test; the leak was reported in CrashyNodes, not ChurnyElections (at least, according to the log you attached in KUDU-2517). https://gerrit.cloudera.org/c/7052/ seems related. Maybe ask Todd what he remembers about that particular issue? BTW, were you able to repro the leak with LSAN_OPTIONS=fast_unwind_on_malloc=0? If you can repro the failure locally when running a full ctest suite, maybe leave your machine running overnight with the test suite running in a loop? -- To view, visit http://gerrit.cloudera.org:8080/11836 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4091d3905d871acb48ec4d88c7b81ee48bf0eed Gerrit-Change-Number: 11836 Gerrit-PatchSet: 1 Gerrit-Owner: Andrew Wong Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Comment-Date: Wed, 31 Oct 2018 04:34:54 + Gerrit-HasComments: Yes
[kudu-CR] revert change to exactly once writes-itest
Andrew Wong has removed a vote on this change. Change subject: revert change to exactly_once_writes-itest .. Removed Verified-1 by Kudu Jenkins (120) -- To view, visit http://gerrit.cloudera.org:8080/11836 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: If4091d3905d871acb48ec4d88c7b81ee48bf0eed Gerrit-Change-Number: 11836 Gerrit-PatchSet: 1 Gerrit-Owner: Andrew Wong Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] revert change to exactly once writes-itest
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11836 ) Change subject: revert change to exactly_once_writes-itest .. Patch Set 1: Verified+1 Unrelated failure of trace-test -- To view, visit http://gerrit.cloudera.org:8080/11836 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4091d3905d871acb48ec4d88c7b81ee48bf0eed Gerrit-Change-Number: 11836 Gerrit-PatchSet: 1 Gerrit-Owner: Andrew Wong Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Comment-Date: Wed, 31 Oct 2018 03:43:11 + Gerrit-HasComments: No
[kudu-CR] revert change to exactly once writes-itest
Andrew Wong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11836 Change subject: revert change to exactly_once_writes-itest .. revert change to exactly_once_writes-itest Since 5894af6ff12291ffab2d3304b34f77fe6d112771, in some environments, exactly_once_writes-itest will pass successfully but report a leak that would not fail the test itself. Try as I might to get to the bottom of the non-fatal leak, its root cause has evaded me. So instead, I'm reverting the test change that led to its leakiness. It seems it only fails some of the time when running a full ctest suite locally (i.e. not through dist-test). Some conclusions I've drawn about the leak: * Not easily reproducible via dist-test, even if we begin reporting LeakSanitizer errors on passing tests (which we don't today) * Not easily reproducible when running the test locally on its own * Not deterministically reproducible when running a full ctest suite locally Without the change, the leak was detected in 2/7 runs of the full test suite, and with the change, in 0/14. Change-Id: If4091d3905d871acb48ec4d88c7b81ee48bf0eed --- M src/kudu/integration-tests/exactly_once_writes-itest.cc 1 file changed, 2 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/36/11836/1 -- To view, visit http://gerrit.cloudera.org:8080/11836 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: If4091d3905d871acb48ec4d88c7b81ee48bf0eed Gerrit-Change-Number: 11836 Gerrit-PatchSet: 1 Gerrit-Owner: Andrew Wong
