[kudu-CR] exactly once writes-itest for faulty disks
David Ribeiro Alves has posted comments on this change. ( http://gerrit.cloudera.org:8080/8388 ) Change subject: exactly_once_writes-itest for faulty disks .. Patch Set 8: (7 comments) http://gerrit.cloudera.org:8080/#/c/8388/8/src/kudu/integration-tests/exactly_once_writes-itest.cc File src/kudu/integration-tests/exactly_once_writes-itest.cc: http://gerrit.cloudera.org:8080/#/c/8388/8/src/kudu/integration-tests/exactly_once_writes-itest.cc@77 PS8, Line 77: using cluster::ExternalTabletServer; nit: move before the two above http://gerrit.cloudera.org:8080/#/c/8388/8/src/kudu/integration-tests/exactly_once_writes-itest.cc@83 PS8, Line 83: int64_t xs nit: why is this the only int64_t ? doesn't seem like it would be feasible to have over 2B attempts http://gerrit.cloudera.org:8080/#/c/8388/8/src/kudu/integration-tests/exactly_once_writes-itest.cc@110 PS8, Line 110: If 'allow_crashes' is set, crashed servers are restarted. maybe add that if 'allow_crashes' is false the test is caused to fail (that is what happens, right?) http://gerrit.cloudera.org:8080/#/c/8388/8/src/kudu/integration-tests/exactly_once_writes-itest.cc@118 PS8, Line 118: matching commit indexes. not sure what this means... http://gerrit.cloudera.org:8080/#/c/8388/8/src/kudu/integration-tests/exactly_once_writes-itest.cc@234 PS8, Line 234: { : std::lock_guard l(lock); : if (num_complete == num_threads) break; : } : if (allow_crashes) { : RETURN_IF_FATALS(RestartAnyCrashedTabletServers(), : "Failed to restart crashed tablet servers"); : } else { : RETURN_IF_FATALS(AssertNoTabletServersCrashed(), : "Some tablet servers crashed"); : } : SleepFor(MonoDelta::FromMilliseconds(10)); nit: might be slightly cleaner to use a countdown latch (avoid the extra synchronization around the shared var and has a timeout function that replaces the sleep) http://gerrit.cloudera.org:8080/#/c/8388/8/src/kudu/integration-tests/exactly_once_writes-itest.cc@303 PS8, Line 303: LOG(INFO) debug? same below http://gerrit.cloudera.org:8080/#/c/8388/8/src/kudu/util/test_macros.h File src/kudu/util/test_macros.h: http://gerrit.cloudera.org:8080/#/c/8388/8/src/kudu/util/test_macros.h@27 PS8, Line 27: // Sometimes it's useful to return a status, so we're not forced to write void : // functions to check for fatals explain this better, can't figure it out just from reading this without much more context -- To view, visit http://gerrit.cloudera.org:8080/8388 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I966b41b71e39827f8f78fcca59a2208f13f71e53 Gerrit-Change-Number: 8388 Gerrit-PatchSet: 8 Gerrit-Owner: Andrew Wong Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot Gerrit-Comment-Date: Tue, 24 Jul 2018 21:14:31 + Gerrit-HasComments: Yes
[kudu-CR] exactly once writes-itest for faulty disks
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/8388 ) Change subject: exactly_once_writes-itest for faulty disks .. Patch Set 8: Verified+1 Unrelated flake -- To view, visit http://gerrit.cloudera.org:8080/8388 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I966b41b71e39827f8f78fcca59a2208f13f71e53 Gerrit-Change-Number: 8388 Gerrit-PatchSet: 8 Gerrit-Owner: Andrew Wong Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot Gerrit-Comment-Date: Tue, 24 Jul 2018 04:52:06 + Gerrit-HasComments: No
[kudu-CR] exactly once writes-itest for faulty disks
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/8388 ) Change subject: exactly_once_writes-itest for faulty disks .. Patch Set 8: It still runs and isn't too flaky (ran a thousand runs in release and TSAN)! Just changed some minor things -- To view, visit http://gerrit.cloudera.org:8080/8388 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I966b41b71e39827f8f78fcca59a2208f13f71e53 Gerrit-Change-Number: 8388 Gerrit-PatchSet: 8 Gerrit-Owner: Andrew Wong Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot Gerrit-Comment-Date: Tue, 24 Jul 2018 03:28:57 + Gerrit-HasComments: No
[kudu-CR] exactly once writes-itest for faulty disks
Hello Tidy Bot, David Ribeiro Alves, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8388 to look at the new patch set (#8). Change subject: exactly_once_writes-itest for faulty disks .. exactly_once_writes-itest for faulty disks This patch adds an ExactlyOnceWrites test that exercises intermittently crashy disks. This serves to not only test disk failures during writes, but also test disk failures during tablet copies, all while maintaining exactly-once semantics. exactly_once_writes-itest is also updated to use std::thread instead of kudu::Thread, and to favor returning Status instead of void. Change-Id: I966b41b71e39827f8f78fcca59a2208f13f71e53 --- M src/kudu/integration-tests/exactly_once_writes-itest.cc M src/kudu/util/test_macros.h 2 files changed, 241 insertions(+), 121 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/88/8388/8 -- To view, visit http://gerrit.cloudera.org:8080/8388 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I966b41b71e39827f8f78fcca59a2208f13f71e53 Gerrit-Change-Number: 8388 Gerrit-PatchSet: 8 Gerrit-Owner: Andrew Wong Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot
[kudu-CR] exactly once writes-itest for faulty disks
David Ribeiro Alves has posted comments on this change. ( http://gerrit.cloudera.org:8080/8388 ) Change subject: exactly_once_writes-itest for faulty disks .. Patch Set 7: this still worth merging? -- To view, visit http://gerrit.cloudera.org:8080/8388 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I966b41b71e39827f8f78fcca59a2208f13f71e53 Gerrit-Change-Number: 8388 Gerrit-PatchSet: 7 Gerrit-Owner: Andrew Wong Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot Gerrit-Comment-Date: Mon, 23 Jul 2018 22:46:18 + Gerrit-HasComments: No
[kudu-CR] exactly once writes-itest for faulty disks
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/8388 ) Change subject: exactly_once_writes-itest for faulty disks .. Patch Set 7: Retriggered. Failed due to clock sync issues. -- To view, visit http://gerrit.cloudera.org:8080/8388 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I966b41b71e39827f8f78fcca59a2208f13f71e53 Gerrit-Change-Number: 8388 Gerrit-PatchSet: 7 Gerrit-Owner: Andrew WongGerrit-Reviewer: Andrew Wong Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot Gerrit-Comment-Date: Thu, 30 Nov 2017 00:27:39 + Gerrit-HasComments: No
[kudu-CR] exactly once writes-itest for faulty disks
Hello Tidy Bot, David Ribeiro Alves, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8388 to look at the new patch set (#7). Change subject: exactly_once_writes-itest for faulty disks .. exactly_once_writes-itest for faulty disks This patch adds an ExactlyOnceWrites test that exercises intermittently crashy disks. This serves to not only test disk failures during writes, but also test disk failures during tablet copies, all while maintaining exactly-once semantics. exactly_once_writes-itest is also updated to use std::thread instead of kudu::Thread, and to favor returning Status instead of void. Change-Id: I966b41b71e39827f8f78fcca59a2208f13f71e53 --- M src/kudu/integration-tests/exactly_once_writes-itest.cc M src/kudu/util/test_macros.h 2 files changed, 229 insertions(+), 115 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/88/8388/7 -- To view, visit http://gerrit.cloudera.org:8080/8388 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I966b41b71e39827f8f78fcca59a2208f13f71e53 Gerrit-Change-Number: 8388 Gerrit-PatchSet: 7 Gerrit-Owner: Andrew WongGerrit-Reviewer: Andrew Wong Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot
[kudu-CR] exactly once writes-itest for faulty disks
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/8388 ) Change subject: exactly_once_writes-itest for faulty disks .. Patch Set 6: The build failed due to a timeout (it just took too long), so I adjusted the numbers a bit to speed it up. -- To view, visit http://gerrit.cloudera.org:8080/8388 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I966b41b71e39827f8f78fcca59a2208f13f71e53 Gerrit-Change-Number: 8388 Gerrit-PatchSet: 6 Gerrit-Owner: Andrew WongGerrit-Reviewer: Andrew Wong Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot Gerrit-Comment-Date: Wed, 29 Nov 2017 04:01:35 + Gerrit-HasComments: No
[kudu-CR] exactly once writes-itest for faulty disks
Hello Tidy Bot, David Ribeiro Alves, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8388 to look at the new patch set (#6). Change subject: exactly_once_writes-itest for faulty disks .. exactly_once_writes-itest for faulty disks This patch adds an ExactlyOnceWrites test that exercises intermittently crashy disks. This serves to not only test disk failures during writes, but also test disk failures during tablet copies, all while maintaining exactly-once semantics. exactly_once_writes-itest is also updated to use std::thread instead of kudu::Thread, and to favor returning Status instead of void. Change-Id: I966b41b71e39827f8f78fcca59a2208f13f71e53 --- M src/kudu/integration-tests/exactly_once_writes-itest.cc M src/kudu/util/test_macros.h 2 files changed, 228 insertions(+), 115 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/88/8388/6 -- To view, visit http://gerrit.cloudera.org:8080/8388 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I966b41b71e39827f8f78fcca59a2208f13f71e53 Gerrit-Change-Number: 8388 Gerrit-PatchSet: 6 Gerrit-Owner: Andrew WongGerrit-Reviewer: Andrew Wong Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot
[kudu-CR] exactly once writes-itest for faulty disks
Hello Tidy Bot, David Ribeiro Alves, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8388 to look at the new patch set (#5). Change subject: exactly_once_writes-itest for faulty disks .. exactly_once_writes-itest for faulty disks This patch adds an ExactlyOnceWrites test that exercises intermittently crashy disks. This serves to not only test disk failures during writes, but also test disk failures during tablet copies, all while maintaining exactly-once semantics. exactly_once_writes-itest is also updated to use std::thread instead of kudu::Thread, and to favor returning Status instead of void. Change-Id: I966b41b71e39827f8f78fcca59a2208f13f71e53 --- M src/kudu/integration-tests/exactly_once_writes-itest.cc M src/kudu/tablet/transactions/transaction_driver.cc M src/kudu/util/test_macros.h 3 files changed, 233 insertions(+), 117 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/88/8388/5 -- To view, visit http://gerrit.cloudera.org:8080/8388 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I966b41b71e39827f8f78fcca59a2208f13f71e53 Gerrit-Change-Number: 8388 Gerrit-PatchSet: 5 Gerrit-Owner: Andrew WongGerrit-Reviewer: Andrew Wong Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot
[kudu-CR] exactly once writes-itest for faulty disks
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/8388 ) Change subject: exactly_once_writes-itest for faulty disks .. Patch Set 4: (7 comments) http://gerrit.cloudera.org:8080/#/c/8388/3/src/kudu/integration-tests/exactly_once_writes-itest.cc File src/kudu/integration-tests/exactly_once_writes-itest.cc: http://gerrit.cloudera.org:8080/#/c/8388/3/src/kudu/integration-tests/exactly_once_writes-itest.cc@115 PS3, Line 115: Status DoTestWritesWithExactlyOnceSemantics(int num_batches, : bool allow_crashes); : : // Validates that the servers all have matchin > docs Done http://gerrit.cloudera.org:8080/#/c/8388/3/src/kudu/integration-tests/exactly_once_writes-itest.cc@128 PS3, Line 128: }; > warning: redundant get() call on smart pointer [google-readability-redundan Done http://gerrit.cloudera.org:8080/#/c/8388/3/src/kudu/integration-tests/exactly_once_writes-itest.cc@247 PS3, Line 247: for (auto& thread : threads) { > warning: redundant get() call on smart pointer [google-readability-redundan Done http://gerrit.cloudera.org:8080/#/c/8388/3/src/kudu/integration-tests/exactly_once_writes-itest.cc@264 PS3, Line 264: // Make sure we received the same responses for the same operations across all > warning: redundant get() call on smart pointer [google-readability-redundan Done http://gerrit.cloudera.org:8080/#/c/8388/3/src/kudu/integration-tests/exactly_once_writes-itest.cc@269 PS3, Line 269: string expected_ts = Substitute("T:$0 TSidx:$1 TSuuid:$2", j, j % FLAGS_num_replicas, > warning: redundant get() call on smart pointer [google-readability-redundan Done http://gerrit.cloudera.org:8080/#/c/8388/3/src/kudu/integration-tests/exactly_once_writes-itest.cc@342 PS3, Line 342: > nit: TestWritesWithExactlyOnceSemanticsWithCrashyNodes Done http://gerrit.cloudera.org:8080/#/c/8388/3/src/kudu/integration-tests/exactly_once_writes-itest.cc@382 PS3, Line 382: NO_FATALS(CreateClient(_)); : NO_FATALS(CreateTable()); : WaitForTSAndReplicas(); : ASSERT_FALSE(tablet_replicas_.empty()); : tablet_id_ = (*tablet_replicas_.begin()).first; : : // Fail under half of the servers so we always maintain a healthy majority. : for (int i = 0; i < cluster_->num_tablet_servers() / 2 - 1; i++) { : ExternalTabletServer* ts = cluster_->tablet_server(i); : // Fail blocks in all of the directories, avoiding the metadata directory. : vector dirs_to_fail(ts->data_dirs().begin() + 1, ts->data_dirs().end()); : ASSERT_OK(cluster_->SetFlag(ts, "env_inject_eio_globs", : JoinStrings(JoinPathSegmentsV(dirs_to_fail, "**"), ","))); : : // Fail only a fraction of the time to get better failure-path coverage. : ASSERT_OK(cluster_->SetF > not entirely your fault, but can we add an assertion that the failures did Done -- To view, visit http://gerrit.cloudera.org:8080/8388 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I966b41b71e39827f8f78fcca59a2208f13f71e53 Gerrit-Change-Number: 8388 Gerrit-PatchSet: 4 Gerrit-Owner: Andrew WongGerrit-Reviewer: Andrew Wong Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot Gerrit-Comment-Date: Tue, 28 Nov 2017 23:20:30 + Gerrit-HasComments: Yes
[kudu-CR] exactly once writes-itest for faulty disks
Hello Tidy Bot, David Ribeiro Alves, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8388 to look at the new patch set (#4). Change subject: exactly_once_writes-itest for faulty disks .. exactly_once_writes-itest for faulty disks This patch adds an ExactlyOnceWrites test that exercises intermittently crashy disks. This serves to not only test disk failures during writes, but also test disk failures during tablet copies, all while maintaining exactly-once semantics. exactly_once_writes-itest is also updated to use std::thread instead of kudu::Thread, and to favor returning Status instead of void. Change-Id: I966b41b71e39827f8f78fcca59a2208f13f71e53 --- M src/kudu/integration-tests/exactly_once_writes-itest.cc M src/kudu/util/test_macros.h 2 files changed, 228 insertions(+), 115 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/88/8388/4 -- To view, visit http://gerrit.cloudera.org:8080/8388 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I966b41b71e39827f8f78fcca59a2208f13f71e53 Gerrit-Change-Number: 8388 Gerrit-PatchSet: 4 Gerrit-Owner: Andrew WongGerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot
[kudu-CR] exactly once writes-itest for faulty disks
David Ribeiro Alves has posted comments on this change. ( http://gerrit.cloudera.org:8080/8388 ) Change subject: exactly_once_writes-itest for faulty disks .. Patch Set 3: (3 comments) http://gerrit.cloudera.org:8080/#/c/8388/3/src/kudu/integration-tests/exactly_once_writes-itest.cc File src/kudu/integration-tests/exactly_once_writes-itest.cc: http://gerrit.cloudera.org:8080/#/c/8388/3/src/kudu/integration-tests/exactly_once_writes-itest.cc@115 PS3, Line 115: Status ValidateLogs(); : : void ValidateWriteResponses(const WriteResponsePBsPerThread& responses, : bool* mismatched); docs http://gerrit.cloudera.org:8080/#/c/8388/3/src/kudu/integration-tests/exactly_once_writes-itest.cc@342 PS3, Line 342: TestWritesWithExactlyOnceSemanticsWithCrashNodes nit: TestWritesWithExactlyOnceSemanticsWithCrashyNodes http://gerrit.cloudera.org:8080/#/c/8388/3/src/kudu/integration-tests/exactly_once_writes-itest.cc@382 PS3, Line 382: // Fail under half of the servers so we always maintain a healthy majority. : for (int i = 0; i < cluster_->num_tablet_servers() / 2 - 1; i++) { : ExternalTabletServer* ts = cluster_->tablet_server(i); : // Fail blocks in all of the directories, avoiding the metadata directory : // so we don't crash. : vector dirs_to_fail(ts->data_dirs().begin() + 1, ts->data_dirs().end()); : ASSERT_OK(cluster_->SetFlag(ts, "env_inject_eio_globs", : JoinStrings(JoinPathSegmentsV(dirs_to_fail, "**"), ","))); : : // Fail only a fraction of the time to get better failure-path coverage. : ASSERT_OK(cluster_->SetFlag(ts, "env_inject_eio", "0.025")); : } : : // All servers should stay alive and complete their write workload. : ASSERT_OK(DoTestWritesWithExactlyOnceSemantics(num_batches, false /* Allow crashes */)); : ASSERT_OK(ValidateLogs()); not entirely your fault, but can we add an assertion that the failures did indeed happen somehow? -- To view, visit http://gerrit.cloudera.org:8080/8388 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I966b41b71e39827f8f78fcca59a2208f13f71e53 Gerrit-Change-Number: 8388 Gerrit-PatchSet: 3 Gerrit-Owner: Andrew WongGerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot Gerrit-Comment-Date: Tue, 28 Nov 2017 21:15:28 + Gerrit-HasComments: Yes
[kudu-CR] exactly once writes-itest for faulty disks
Hello Tidy Bot, David Ribeiro Alves, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8388 to look at the new patch set (#3). Change subject: exactly_once_writes-itest for faulty disks .. exactly_once_writes-itest for faulty disks This patch adds an ExactlyOnceWrites test that exercises intermittently crashy disks. This serves to not only test disk failures during writes, but also test disk failures during tablet copies, all while maintaining exactly-once semantics. exactly_once_writes-itest is also updated to use std::thread instead of kudu::Thread, and to favor returning Status instead of void. Change-Id: I966b41b71e39827f8f78fcca59a2208f13f71e53 --- M src/kudu/integration-tests/exactly_once_writes-itest.cc M src/kudu/util/test_macros.h 2 files changed, 205 insertions(+), 112 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/88/8388/3 -- To view, visit http://gerrit.cloudera.org:8080/8388 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I966b41b71e39827f8f78fcca59a2208f13f71e53 Gerrit-Change-Number: 8388 Gerrit-PatchSet: 3 Gerrit-Owner: Andrew WongGerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot