[kudu-CR] KUDU-2652: deflake TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks
Andrew Wong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12166 ) Change subject: KUDU-2652: deflake TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks .. KUDU-2652: deflake TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks The test waits for a write workload to generate some orphaned blocks. With the default write pattern, this could sometimes take a while and cause the test to hit an error. This patch makes orphaned block generation faster by making the workload update a single row. Before this, I looped the test in debug mode with 32 stress threads and it failed 15/100 times. With the fix, this passed 1000/1000. Change-Id: I24a689702d29a744be8c113fedafec0307a90b1c Reviewed-on: http://gerrit.cloudera.org:8080/12166 Tested-by: Kudu Jenkins Reviewed-by: Adar Dembo Reviewed-by: Mike Percy --- M src/kudu/integration-tests/ts_recovery-itest.cc 1 file changed, 3 insertions(+), 0 deletions(-) Approvals: Kudu Jenkins: Verified Adar Dembo: Looks good to me, approved Mike Percy: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/12166 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I24a689702d29a744be8c113fedafec0307a90b1c Gerrit-Change-Number: 12166 Gerrit-PatchSet: 3 Gerrit-Owner: Andrew Wong Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy
[kudu-CR] KUDU-2652: deflake TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks
Mike Percy has posted comments on this change. ( http://gerrit.cloudera.org:8080/12166 ) Change subject: KUDU-2652: deflake TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12166 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I24a689702d29a744be8c113fedafec0307a90b1c Gerrit-Change-Number: 12166 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Wong Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Comment-Date: Wed, 09 Jan 2019 00:45:53 + Gerrit-HasComments: No
[kudu-CR] KUDU-2652: deflake TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/12166 ) Change subject: KUDU-2652: deflake TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12166 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I24a689702d29a744be8c113fedafec0307a90b1c Gerrit-Change-Number: 12166 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Wong Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Comment-Date: Wed, 09 Jan 2019 00:23:33 + Gerrit-HasComments: No
[kudu-CR] KUDU-2652: deflake TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12166 ) Change subject: KUDU-2652: deflake TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/12166/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/12166/1//COMMIT_MSG@11 PS1, Line 11: This patch makes orphaned block : generation faster by making the workload update a single row. > I would also suspect Will's compaction changes; I wonder if flushing outstr >Hmm, why is this obviously faster? The default is a random write workload, no a sequential one. It's a bit harder to reason about what the rowsets look like, but I'd expect either sequential writes or single-row-updates to definitely generate orphaned blocks. >I also wonder why this surfaced just now. I actually just saw this fail in a Cloudera-internal 1.8 build, so I don't think the flakiness was introduced by the recent compaction policy changes. http://gerrit.cloudera.org:8080/#/c/12166/1/src/kudu/integration-tests/ts_recovery-itest.cc File src/kudu/integration-tests/ts_recovery-itest.cc: http://gerrit.cloudera.org:8080/#/c/12166/1/src/kudu/integration-tests/ts_recovery-itest.cc@257 PS1, Line 257: // We want to generate orphaned blocks, so use a workload that we expect to > I think the explanation is that we end up generating lots of (major and/or Done, yes. -- To view, visit http://gerrit.cloudera.org:8080/12166 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I24a689702d29a744be8c113fedafec0307a90b1c Gerrit-Change-Number: 12166 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Wong Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Comment-Date: Tue, 08 Jan 2019 23:48:40 + Gerrit-HasComments: Yes
[kudu-CR] KUDU-2652: deflake TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks
Hello Mike Percy, Kudu Jenkins, Adar Dembo, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12166 to look at the new patch set (#2). Change subject: KUDU-2652: deflake TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks .. KUDU-2652: deflake TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks The test waits for a write workload to generate some orphaned blocks. With the default write pattern, this could sometimes take a while and cause the test to hit an error. This patch makes orphaned block generation faster by making the workload update a single row. Before this, I looped the test in debug mode with 32 stress threads and it failed 15/100 times. With the fix, this passed 1000/1000. Change-Id: I24a689702d29a744be8c113fedafec0307a90b1c --- M src/kudu/integration-tests/ts_recovery-itest.cc 1 file changed, 3 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/66/12166/2 -- To view, visit http://gerrit.cloudera.org:8080/12166 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I24a689702d29a744be8c113fedafec0307a90b1c Gerrit-Change-Number: 12166 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Wong Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy
[kudu-CR] KUDU-2652: deflake TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks
Mike Percy has posted comments on this change. ( http://gerrit.cloudera.org:8080/12166 ) Change subject: KUDU-2652: deflake TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/12166/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/12166/1//COMMIT_MSG@11 PS1, Line 11: This patch makes orphaned block : generation faster by making the workload update a single row. > Hmm, why is this obviously faster? Updating one row over and over means we' I would also suspect Will's compaction changes; I wonder if flushing outstrips the merge compactions in the old scenario now. -- To view, visit http://gerrit.cloudera.org:8080/12166 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I24a689702d29a744be8c113fedafec0307a90b1c Gerrit-Change-Number: 12166 Gerrit-PatchSet: 1 Gerrit-Owner: Andrew Wong Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Comment-Date: Tue, 08 Jan 2019 23:40:57 + Gerrit-HasComments: Yes
[kudu-CR] KUDU-2652: deflake TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks
Mike Percy has posted comments on this change. ( http://gerrit.cloudera.org:8080/12166 ) Change subject: KUDU-2652: deflake TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/12166/1/src/kudu/integration-tests/ts_recovery-itest.cc File src/kudu/integration-tests/ts_recovery-itest.cc: http://gerrit.cloudera.org:8080/#/c/12166/1/src/kudu/integration-tests/ts_recovery-itest.cc@257 PS1, Line 257: write_workload->set_write_pattern(TestWorkload::UPDATE_ONE_ROW); > Should add a comment here explaining why we're using this. I think the explanation is that we end up generating lots of (major and/or minor) delta compactions, right? -- To view, visit http://gerrit.cloudera.org:8080/12166 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I24a689702d29a744be8c113fedafec0307a90b1c Gerrit-Change-Number: 12166 Gerrit-PatchSet: 1 Gerrit-Owner: Andrew Wong Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Comment-Date: Tue, 08 Jan 2019 23:39:23 + Gerrit-HasComments: Yes
[kudu-CR] KUDU-2652: deflake TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/12166 ) Change subject: KUDU-2652: deflake TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/12166/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/12166/1//COMMIT_MSG@11 PS1, Line 11: This patch makes orphaned block : generation faster by making the workload update a single row. Hmm, why is this obviously faster? Updating one row over and over means we'll flush one MRS (INSERT and n UPDATEs), then flushing that DRS's DMS over and over. I guess there'll be some delta compaction which will delete blocks, but is that de facto more likely than the merge compaction you'd expect to see on the tiny DRSes produced by the default write pattern? I also wonder why this surfaced just now. I suspect Will's change to the merge compaction policy, but given the non-default values to flush_threshold_mb and flush_threshold_secs I would have expected the change to make merge compaction more likely not less. http://gerrit.cloudera.org:8080/#/c/12166/1/src/kudu/integration-tests/ts_recovery-itest.cc File src/kudu/integration-tests/ts_recovery-itest.cc: http://gerrit.cloudera.org:8080/#/c/12166/1/src/kudu/integration-tests/ts_recovery-itest.cc@257 PS1, Line 257: write_workload->set_write_pattern(TestWorkload::UPDATE_ONE_ROW); Should add a comment here explaining why we're using this. -- To view, visit http://gerrit.cloudera.org:8080/12166 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I24a689702d29a744be8c113fedafec0307a90b1c Gerrit-Change-Number: 12166 Gerrit-PatchSet: 1 Gerrit-Owner: Andrew Wong Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Comment-Date: Sat, 05 Jan 2019 02:01:52 + Gerrit-HasComments: Yes
[kudu-CR] KUDU-2652: deflake TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks
Andrew Wong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12166 Change subject: KUDU-2652: deflake TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks .. KUDU-2652: deflake TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks The test waits for a write workload to generate some orphaned blocks. With the default write pattern, this could sometimes take a while and cause the test to hit an error. This patch makes orphaned block generation faster by making the workload update a single row. Before this, I looped the test in debug mode with 32 stress threads and it failed 15/100 times. With the fix, this passed 1000/1000. Change-Id: I24a689702d29a744be8c113fedafec0307a90b1c --- M src/kudu/integration-tests/ts_recovery-itest.cc 1 file changed, 1 insertion(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/66/12166/1 -- To view, visit http://gerrit.cloudera.org:8080/12166 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I24a689702d29a744be8c113fedafec0307a90b1c Gerrit-Change-Number: 12166 Gerrit-PatchSet: 1 Gerrit-Owner: Andrew Wong
