Re: Review Request 65759: Added inspect retries to the Docker executor.

[Gilbert Song]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65759/#review198438
---


Ship it!




Ship It!

- Gilbert Song


On Feb. 28, 2018, 10:05 a.m., Andrei Budnik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65759/
> ---
> 
> (Updated Feb. 28, 2018, 10:05 a.m.)
> 
> 
> Review request for mesos, Alexander Rukletsov, Gilbert Song, Greg Mann, and 
> Michael Park.
> 
> 
> Bugs: MESOS-8574
> https://issues.apache.org/jira/browse/MESOS-8574
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> This patch adds retries for `inspect` command to workaround docker
> daemon hangs. We assume that the docker daemon can be temporarily
> unresponsive. If it's unresponsive, then any started docker cli
> command hangs. To address the issue, we retry `inspect` in the loop.
> 
> 
> Diffs
> -
> 
>   src/docker/executor.cpp 93c3e1d1e86814e34cbe5b045f6e61911266c535 
> 
> 
> Diff: https://reviews.apache.org/r/65759/diff/6/
> 
> 
> Testing
> ---
> 
> internal CI
> 
> Manually, described in /r/65713
> 
> 
> Thanks,
> 
> Andrei Budnik
> 
>

Re: Review Request 65759: Added inspect retries to the Docker executor.

[Greg Mann]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65759/#review198414
---


Ship it!




Ship It!

- Greg Mann


On Feb. 28, 2018, 6:05 p.m., Andrei Budnik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65759/
> ---
> 
> (Updated Feb. 28, 2018, 6:05 p.m.)
> 
> 
> Review request for mesos, Alexander Rukletsov, Gilbert Song, Greg Mann, and 
> Michael Park.
> 
> 
> Bugs: MESOS-8574
> https://issues.apache.org/jira/browse/MESOS-8574
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> This patch adds retries for `inspect` command to workaround docker
> daemon hangs. We assume that the docker daemon can be temporarily
> unresponsive. If it's unresponsive, then any started docker cli
> command hangs. To address the issue, we retry `inspect` in the loop.
> 
> 
> Diffs
> -
> 
>   src/docker/executor.cpp 93c3e1d1e86814e34cbe5b045f6e61911266c535 
> 
> 
> Diff: https://reviews.apache.org/r/65759/diff/6/
> 
> 
> Testing
> ---
> 
> internal CI
> 
> Manually, described in /r/65713
> 
> 
> Thanks,
> 
> Andrei Budnik
> 
>

Re: Review Request 65759: Added inspect retries to the Docker executor.

[Andrei Budnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65759/
---

(Updated Feb. 28, 2018, 6:05 p.m.)


Review request for mesos, Alexander Rukletsov, Gilbert Song, Greg Mann, and 
Michael Park.


Bugs: MESOS-8574
https://issues.apache.org/jira/browse/MESOS-8574


Repository: mesos


Description
---

This patch adds retries for `inspect` command to workaround docker
daemon hangs. We assume that the docker daemon can be temporarily
unresponsive. If it's unresponsive, then any started docker cli
command hangs. To address the issue, we retry `inspect` in the loop.


Diffs (updated)
-

  src/docker/executor.cpp 93c3e1d1e86814e34cbe5b045f6e61911266c535 


Diff: https://reviews.apache.org/r/65759/diff/6/

Changes: https://reviews.apache.org/r/65759/diff/5-6/


Testing
---

internal CI

Manually, described in /r/65713


Thanks,

Andrei Budnik

Re: Review Request 65759: Added inspect retries to the Docker executor.

[Greg Mann]

> On Feb. 28, 2018, 10:04 a.m., Greg Mann wrote:
> > src/docker/executor.cpp
> > Lines 242-247 (patched)
> > 
> >
> > Doesn't this render the `onFailed` callback registered on L357 useless? 
> > i.e., the `inspect` future will never transition to the failed state?
> > 
> > If so, we should probably either remove the `onFailed` callback, or do 
> > this instead:
> > ```
> > if (!future.hasDiscard()) {
> >   return Break(future);
> > }
> > ```
> > 
> > I guess the question is: if the inspect call actually fails, rather 
> > than hanging, do we want to retry? It looks like there are several cases in 
> > `Docker::inspect` which will result in a failure (failed to create 
> > subprocess, failed to read stdout, etc..), and it looks to me like we could 
> > probably just retry in those cases. WDYT?
> 
> Andrei Budnik wrote:
> Good point! If a docker daemon returns non-zero, the docker libray will 
> retry `inspect`, then we'll get a message kile: 
> `I0228 17:28:13.275115  3248 docker.cpp:1369] Retrying inspect with 
> non-zero status code. cmd: 'docker -H unix:///var/run/docker.sock inspect 
> mesos-210b988c-c808-47e5-af65-75f40269755b', interval: 500ms`
> 
> But if the docker library returns a failure itself due to some severe bug 
> (failed to create subprocess, failed to read stdout, etc...), then IMO we 
> should stop retrying `inspect`:
> 
> ```
>[](const Future& future)
>   -> Future {
>   if (future.isReady()) {
> return Break(future.get());
>   }
>   if (future.isFailed()) {
> return Failure(future.failure());
>   }
>   return Continue();
> });
> ```

SGTM!


- Greg


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65759/#review198382
---


On Feb. 22, 2018, 8:32 p.m., Andrei Budnik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65759/
> ---
> 
> (Updated Feb. 22, 2018, 8:32 p.m.)
> 
> 
> Review request for mesos, Alexander Rukletsov, Gilbert Song, Greg Mann, and 
> Michael Park.
> 
> 
> Bugs: MESOS-8574
> https://issues.apache.org/jira/browse/MESOS-8574
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> This patch adds retries for `inspect` command to workaround docker
> daemon hangs. We assume that the docker daemon can be temporarily
> unresponsive. If it's unresponsive, then any started docker cli
> command hangs. To address the issue, we retry `inspect` in the loop.
> 
> 
> Diffs
> -
> 
>   src/docker/executor.cpp 93c3e1d1e86814e34cbe5b045f6e61911266c535 
> 
> 
> Diff: https://reviews.apache.org/r/65759/diff/4/
> 
> 
> Testing
> ---
> 
> internal CI
> 
> Manually, described in /r/65713
> 
> 
> Thanks,
> 
> Andrei Budnik
> 
>

Re: Review Request 65759: Added inspect retries to the Docker executor.

[Andrei Budnik]

> On Feb. 28, 2018, 10:04 a.m., Greg Mann wrote:
> > src/docker/executor.cpp
> > Lines 242-247 (patched)
> > 
> >
> > Doesn't this render the `onFailed` callback registered on L357 useless? 
> > i.e., the `inspect` future will never transition to the failed state?
> > 
> > If so, we should probably either remove the `onFailed` callback, or do 
> > this instead:
> > ```
> > if (!future.hasDiscard()) {
> >   return Break(future);
> > }
> > ```
> > 
> > I guess the question is: if the inspect call actually fails, rather 
> > than hanging, do we want to retry? It looks like there are several cases in 
> > `Docker::inspect` which will result in a failure (failed to create 
> > subprocess, failed to read stdout, etc..), and it looks to me like we could 
> > probably just retry in those cases. WDYT?

Good point! If a docker daemon returns non-zero, the docker libray will retry 
`inspect`, then we'll get a message kile: 
`I0228 17:28:13.275115  3248 docker.cpp:1369] Retrying inspect with non-zero 
status code. cmd: 'docker -H unix:///var/run/docker.sock inspect 
mesos-210b988c-c808-47e5-af65-75f40269755b', interval: 500ms`

But if the docker library returns a failure itself due to some severe bug 
(failed to create subprocess, failed to read stdout, etc...), then IMO we 
should stop retrying `inspect`:

```
   [](const Future& future)
  -> Future {
  if (future.isReady()) {
return Break(future.get());
  }
  if (future.isFailed()) {
return Failure(future.failure());
  }
  return Continue();
});
```


- Andrei


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65759/#review198382
---


On Feb. 22, 2018, 8:32 p.m., Andrei Budnik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65759/
> ---
> 
> (Updated Feb. 22, 2018, 8:32 p.m.)
> 
> 
> Review request for mesos, Alexander Rukletsov, Gilbert Song, Greg Mann, and 
> Michael Park.
> 
> 
> Bugs: MESOS-8574
> https://issues.apache.org/jira/browse/MESOS-8574
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> This patch adds retries for `inspect` command to workaround docker
> daemon hangs. We assume that the docker daemon can be temporarily
> unresponsive. If it's unresponsive, then any started docker cli
> command hangs. To address the issue, we retry `inspect` in the loop.
> 
> 
> Diffs
> -
> 
>   src/docker/executor.cpp 93c3e1d1e86814e34cbe5b045f6e61911266c535 
> 
> 
> Diff: https://reviews.apache.org/r/65759/diff/4/
> 
> 
> Testing
> ---
> 
> internal CI
> 
> Manually, described in /r/65713
> 
> 
> Thanks,
> 
> Andrei Budnik
> 
>

Re: Review Request 65759: Added inspect retries to the Docker executor.

[Greg Mann]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65759/#review198382
---




src/docker/executor.cpp
Lines 242-247 (patched)


Doesn't this render the `onFailed` callback registered on L357 useless? 
i.e., the `inspect` future will never transition to the failed state?

If so, we should probably either remove the `onFailed` callback, or do this 
instead:
```
if (!future.hasDiscard()) {
  return Break(future);
}
```

I guess the question is: if the inspect call actually fails, rather than 
hanging, do we want to retry? It looks like there are several cases in 
`Docker::inspect` which will result in a failure (failed to create subprocess, 
failed to read stdout, etc..), and it looks to me like we could probably just 
retry in those cases. WDYT?



src/docker/executor.cpp
Lines 243 (patched)


Nit: indent two more spaces, to avoid identical indentation to the 
following line.


- Greg Mann


On Feb. 22, 2018, 8:32 p.m., Andrei Budnik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65759/
> ---
> 
> (Updated Feb. 22, 2018, 8:32 p.m.)
> 
> 
> Review request for mesos, Alexander Rukletsov, Gilbert Song, Greg Mann, and 
> Michael Park.
> 
> 
> Bugs: MESOS-8574
> https://issues.apache.org/jira/browse/MESOS-8574
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> This patch adds retries for `inspect` command to workaround docker
> daemon hangs. We assume that the docker daemon can be temporarily
> unresponsive. If it's unresponsive, then any started docker cli
> command hangs. To address the issue, we retry `inspect` in the loop.
> 
> 
> Diffs
> -
> 
>   src/docker/executor.cpp 93c3e1d1e86814e34cbe5b045f6e61911266c535 
> 
> 
> Diff: https://reviews.apache.org/r/65759/diff/3/
> 
> 
> Testing
> ---
> 
> internal CI
> 
> Manually, described in /r/65713
> 
> 
> Thanks,
> 
> Andrei Budnik
> 
>

Re: Review Request 65759: Added inspect retries to the Docker executor.

[Mesos Reviewbot Windows]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65759/#review198328
---



PASS: Mesos patch 65759 was successfully built and tested.

Reviews applied: `['65683', '65743', '65713', '65759']`

All the build artifacts available at: 
http://dcos-win.westus.cloudapp.azure.com/mesos-build/review/65759

- Mesos Reviewbot Windows


On Feb. 22, 2018, 8:32 p.m., Andrei Budnik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65759/
> ---
> 
> (Updated Feb. 22, 2018, 8:32 p.m.)
> 
> 
> Review request for mesos, Alexander Rukletsov, Gilbert Song, Greg Mann, and 
> Michael Park.
> 
> 
> Bugs: MESOS-8574
> https://issues.apache.org/jira/browse/MESOS-8574
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> This patch adds retries for `inspect` command to workaround docker
> daemon hangs. We assume that the docker daemon can be temporarily
> unresponsive. If it's unresponsive, then any started docker cli
> command hangs. To address the issue, we retry `inspect` in the loop.
> 
> 
> Diffs
> -
> 
>   src/docker/executor.cpp 93c3e1d1e86814e34cbe5b045f6e61911266c535 
> 
> 
> Diff: https://reviews.apache.org/r/65759/diff/3/
> 
> 
> Testing
> ---
> 
> internal CI
> 
> Manually, described in /r/65713
> 
> 
> Thanks,
> 
> Andrei Budnik
> 
>

Re: Review Request 65759: Added inspect retries to the Docker executor.

[Mesos Reviewbot]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65759/#review198176
---



Bad patch!

Reviews applied: [65759, 65713, 65743, 65683]

Failed command: python support/apply-reviews.py -n -r 65683

Error:
2018-02-23 12:16:26 URL:https://reviews.apache.org/r/65683/diff/raw/ 
[7076/7076] -> "65683.patch" [1]
error: patch failed: src/docker/docker.cpp:1274
error: src/docker/docker.cpp: patch does not apply

Full log: https://builds.apache.org/job/Mesos-Reviewbot/21742/console

- Mesos Reviewbot


On Feb. 22, 2018, 8:32 p.m., Andrei Budnik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65759/
> ---
> 
> (Updated Feb. 22, 2018, 8:32 p.m.)
> 
> 
> Review request for mesos, Alexander Rukletsov, Gilbert Song, Greg Mann, and 
> Michael Park.
> 
> 
> Bugs: MESOS-8574
> https://issues.apache.org/jira/browse/MESOS-8574
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> This patch adds retries for `inspect` command to workaround docker
> daemon hangs. We assume that the docker daemon can be temporarily
> unresponsive. If it's unresponsive, then any started docker cli
> command hangs. To address the issue, we retry `inspect` in the loop.
> 
> 
> Diffs
> -
> 
>   src/docker/executor.cpp 80e2d81169f0d4303ca1ddbcef9fa87fe52601fc 
> 
> 
> Diff: https://reviews.apache.org/r/65759/diff/2/
> 
> 
> Testing
> ---
> 
> internal CI
> 
> Manually, described in /r/65713
> 
> 
> Thanks,
> 
> Andrei Budnik
> 
>

Re: Review Request 65759: Added inspect retries to the Docker executor.

[Mesos Reviewbot Windows]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65759/#review198175
---



FAIL: Failed to apply the dependent review: 65683.

Failed command: `python.exe .\support\apply-reviews.py -n -r 65683`

All the build artifacts available at: 
http://dcos-win.westus.cloudapp.azure.com/mesos-build/review/65759

Relevant logs:

- 
[apply-review-65683-stdout.log](http://dcos-win.westus.cloudapp.azure.com/mesos-build/review/65759/logs/apply-review-65683-stdout.log):

```
error: patch failed: src/docker/docker.cpp:1274
error: src/docker/docker.cpp: patch does not apply
```

- Mesos Reviewbot Windows


On Feb. 22, 2018, 8:32 p.m., Andrei Budnik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65759/
> ---
> 
> (Updated Feb. 22, 2018, 8:32 p.m.)
> 
> 
> Review request for mesos, Alexander Rukletsov, Gilbert Song, Greg Mann, and 
> Michael Park.
> 
> 
> Bugs: MESOS-8574
> https://issues.apache.org/jira/browse/MESOS-8574
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> This patch adds retries for `inspect` command to workaround docker
> daemon hangs. We assume that the docker daemon can be temporarily
> unresponsive. If it's unresponsive, then any started docker cli
> command hangs. To address the issue, we retry `inspect` in the loop.
> 
> 
> Diffs
> -
> 
>   src/docker/executor.cpp 80e2d81169f0d4303ca1ddbcef9fa87fe52601fc 
> 
> 
> Diff: https://reviews.apache.org/r/65759/diff/2/
> 
> 
> Testing
> ---
> 
> internal CI
> 
> Manually, described in /r/65713
> 
> 
> Thanks,
> 
> Andrei Budnik
> 
>

Re: Review Request 65759: Added inspect retries to the Docker executor.

[Andrei Budnik]

> On Feb. 23, 2018, 8:57 a.m., Greg Mann wrote:
> > src/docker/executor.cpp
> > Lines 25 (patched)
> > 
> >
> > Is this necessary?

Without this header, I'm getting:
```
../../src/docker/executor.cpp:230:32: error: ‘await’ was not declared in this 
scope
```


- Andrei


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65759/#review198168
---


On Feb. 22, 2018, 8:32 p.m., Andrei Budnik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65759/
> ---
> 
> (Updated Feb. 22, 2018, 8:32 p.m.)
> 
> 
> Review request for mesos, Alexander Rukletsov, Gilbert Song, Greg Mann, and 
> Michael Park.
> 
> 
> Bugs: MESOS-8574
> https://issues.apache.org/jira/browse/MESOS-8574
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> This patch adds retries for `inspect` command to workaround docker
> daemon hangs. We assume that the docker daemon can be temporarily
> unresponsive. If it's unresponsive, then any started docker cli
> command hangs. To address the issue, we retry `inspect` in the loop.
> 
> 
> Diffs
> -
> 
>   src/docker/executor.cpp 80e2d81169f0d4303ca1ddbcef9fa87fe52601fc 
> 
> 
> Diff: https://reviews.apache.org/r/65759/diff/2/
> 
> 
> Testing
> ---
> 
> internal CI
> 
> Manually, described in /r/65713
> 
> 
> Thanks,
> 
> Andrei Budnik
> 
>

Re: Review Request 65759: Added inspect retries to the Docker executor.

[Greg Mann]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65759/#review198168
---




src/docker/executor.cpp
Lines 25 (patched)


Is this necessary?



src/docker/executor.cpp
Lines 220 (patched)


s/docker/Docker/
s/retry/retry the/



src/docker/executor.cpp
Lines 221 (patched)


s/inspect_loop/inspectLoop/



src/docker/executor.cpp
Lines 233-236 (patched)


Suggestion:

"We need to clean up the hanging Docker CLI process. Discarding the 
`inspect` future triggers a callback in the Docker library that kills the 
subprocess and transitions the future."


- Greg Mann


On Feb. 22, 2018, 8:32 p.m., Andrei Budnik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65759/
> ---
> 
> (Updated Feb. 22, 2018, 8:32 p.m.)
> 
> 
> Review request for mesos, Alexander Rukletsov, Gilbert Song, Greg Mann, and 
> Michael Park.
> 
> 
> Bugs: MESOS-8574
> https://issues.apache.org/jira/browse/MESOS-8574
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> This patch adds retries for `inspect` command to workaround docker
> daemon hangs. We assume that the docker daemon can be temporarily
> unresponsive. If it's unresponsive, then any started docker cli
> command hangs. To address the issue, we retry `inspect` in the loop.
> 
> 
> Diffs
> -
> 
>   src/docker/executor.cpp 80e2d81169f0d4303ca1ddbcef9fa87fe52601fc 
> 
> 
> Diff: https://reviews.apache.org/r/65759/diff/1/
> 
> 
> Testing
> ---
> 
> internal CI
> 
> Manually, described in /r/65713
> 
> 
> Thanks,
> 
> Andrei Budnik
> 
>

Re: Review Request 65759: Added inspect retries to the Docker executor.

[Mesos Reviewbot]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65759/#review198145
---



Patch looks great!

Reviews applied: [65713, 65759]

Passed command: export OS='ubuntu:14.04' BUILDTOOL='autotools' COMPILER='gcc' 
CONFIGURATION='--verbose --disable-libtool-wrappers' ENVIRONMENT='GLOG_v=1 
MESOS_VERBOSE=1'; ./support/docker-build.sh

- Mesos Reviewbot


On Feb. 22, 2018, 12:32 p.m., Andrei Budnik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65759/
> ---
> 
> (Updated Feb. 22, 2018, 12:32 p.m.)
> 
> 
> Review request for mesos, Alexander Rukletsov, Gilbert Song, Greg Mann, and 
> Michael Park.
> 
> 
> Bugs: MESOS-8574
> https://issues.apache.org/jira/browse/MESOS-8574
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> This patch adds retries for `inspect` command to workaround docker
> daemon hangs. We assume that the docker daemon can be temporarily
> unresponsive. If it's unresponsive, then any started docker cli
> command hangs. To address the issue, we retry `inspect` in the loop.
> 
> 
> Diffs
> -
> 
>   src/docker/executor.cpp 80e2d81169f0d4303ca1ddbcef9fa87fe52601fc 
> 
> 
> Diff: https://reviews.apache.org/r/65759/diff/1/
> 
> 
> Testing
> ---
> 
> internal CI
> 
> Manually, described in /r/65713
> 
> 
> Thanks,
> 
> Andrei Budnik
> 
>

Re: Review Request 65759: Added inspect retries to the Docker executor.

[Mesos Reviewbot Windows]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65759/#review198139
---



PASS: Mesos patch 65759 was successfully built and tested.

Reviews applied: `['65713', '65759']`

All the build artifacts available at: 
http://dcos-win.westus.cloudapp.azure.com/mesos-build/review/65759

- Mesos Reviewbot Windows


On Feb. 22, 2018, 8:32 p.m., Andrei Budnik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65759/
> ---
> 
> (Updated Feb. 22, 2018, 8:32 p.m.)
> 
> 
> Review request for mesos, Alexander Rukletsov, Gilbert Song, Greg Mann, and 
> Michael Park.
> 
> 
> Bugs: MESOS-8574
> https://issues.apache.org/jira/browse/MESOS-8574
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> This patch adds retries for `inspect` command to workaround docker
> daemon hangs. We assume that the docker daemon can be temporarily
> unresponsive. If it's unresponsive, then any started docker cli
> command hangs. To address the issue, we retry `inspect` in the loop.
> 
> 
> Diffs
> -
> 
>   src/docker/executor.cpp 80e2d81169f0d4303ca1ddbcef9fa87fe52601fc 
> 
> 
> Diff: https://reviews.apache.org/r/65759/diff/1/
> 
> 
> Testing
> ---
> 
> internal CI
> 
> Manually, described in /r/65713
> 
> 
> Thanks,
> 
> Andrei Budnik
> 
>