attilapiros commented on pull request #28331:
URL: https://github.com/apache/spark/pull/28331#issuecomment-635492967
So it is not recalculation but the timing: the decommissioned executor is
still giving back those shuffle data files:
You can check it by adding a temporary log:
```diff
diff --git
a/core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala
b/core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala
index b959e83599..ca45dc0905 100644
---
a/core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala
+++
b/core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala
@@ -85,9 +85,11 @@ private[spark] class IndexShuffleBlockResolver(
*/
def getDataFile(shuffleId: Int, mapId: Long, dirs:
Option[Array[String]]): File = {
val blockId = ShuffleDataBlockId(shuffleId, mapId, NOOP_REDUCE_ID)
- dirs
+ val file = dirs
.map(ExecutorDiskUtils.getFile(_, blockManager.subDirsPerLocalDir,
blockId.name))
.getOrElse(blockManager.diskBlockManager.getFile(blockId))
+ logInfo(s"Attila: shuffle file: ${file.getAbsolutePath()}", new
Exception())
+ file
}
/**
```
Then please remove the `work` directory content and execute this test only
i.e. by sbt:
```
> testOnly *.BlockManagerDecommissionSuite -- -z "verify that shuffle blocks
are migrated"
[warn] Multiple main classes detected. Run 'show discoveredMainClasses' to
see the list
[info] Packaging
/Users/attilazsoltpiros/git/attilapiros/reviewing/core/target/scala-2.12/spark-core_2.12-3.1.0-SNAPSHOT.jar
...
[info] Done packaging.
[info] BlockManagerDecommissionSuite:
[info] - verify that shuffle blocks are migrated. (12 seconds, 968
milliseconds)
[info] ScalaTest
[info] Run completed in 15 seconds, 145 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[info] Passed: Total 1, Failed 0, Errors 0, Passed 1
[success] Total time: 19 s, completed May 28, 2020 7:17:21 PM
```
You can check the logs in the `work` directory:
```
╭─attilazsoltpiros@apiros-MBP16 ~/git/attilapiros/reviewing/work ‹pr/28331*›
╰─$ ag Attila
app-20200528101709-0000/0/target/unit-tests.log
69:20/05/28 19:17:14.044 Executor task launch worker for task 10 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/33/shuffle_0_10_0.data
86:20/05/28 19:17:14.363 Executor task launch worker for task 10 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/33/shuffle_0_10_0.data
106:20/05/28 19:17:14.886 Executor task launch worker for task 12 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/31/shuffle_0_12_0.data
123:20/05/28 19:17:14.894 Executor task launch worker for task 12 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/31/shuffle_0_12_0.data
143:20/05/28 19:17:15.409 Executor task launch worker for task 13 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/30/shuffle_0_13_0.data
160:20/05/28 19:17:15.416 Executor task launch worker for task 13 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/30/shuffle_0_13_0.data
180:20/05/28 19:17:15.933 Executor task launch worker for task 14 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/2f/shuffle_0_14_0.data
197:20/05/28 19:17:15.940 Executor task launch worker for task 14 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/2f/shuffle_0_14_0.data
217:20/05/28 19:17:16.457 Executor task launch worker for task 15 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/0e/shuffle_0_15_0.data
234:20/05/28 19:17:16.463 Executor task launch worker for task 15 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/0e/shuffle_0_15_0.data
254:20/05/28 19:17:16.975 Executor task launch worker for task 16 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/13/shuffle_0_16_0.data
271:20/05/28 19:17:16.981 Executor task launch worker for task 16 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/13/shuffle_0_16_0.data
291:20/05/28 19:17:17.497 Executor task launch worker for task 17 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/0c/shuffle_0_17_0.data
308:20/05/28 19:17:17.502 Executor task launch worker for task 17 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/0c/shuffle_0_17_0.data
328:20/05/28 19:17:18.017 Executor task launch worker for task 18 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/15/shuffle_0_18_0.data
345:20/05/28 19:17:18.022 Executor task launch worker for task 18 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/15/shuffle_0_18_0.data
365:20/05/28 19:17:18.538 Executor task launch worker for task 19 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/0a/shuffle_0_19_0.data
382:20/05/28 19:17:18.543 Executor task launch worker for task 19 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/0a/shuffle_0_19_0.data
412:20/05/28 19:17:19.177 Executor task launch worker for task 20 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/0a/shuffle_0_19_0.data
437:20/05/28 19:17:19.204 Executor task launch worker for task 21 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/33/shuffle_0_10_0.data
467:20/05/28 19:17:19.248 Executor task launch worker for task 23 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/31/shuffle_0_12_0.data
492:20/05/28 19:17:19.259 Executor task launch worker for task 24 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/30/shuffle_0_13_0.data
517:20/05/28 19:17:19.271 Executor task launch worker for task 25 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/2f/shuffle_0_14_0.data
542:20/05/28 19:17:19.282 Executor task launch worker for task 26 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/0e/shuffle_0_15_0.data
567:20/05/28 19:17:19.295 Executor task launch worker for task 27 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/13/shuffle_0_16_0.data
592:20/05/28 19:17:19.305 Executor task launch worker for task 28 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/0c/shuffle_0_17_0.data
617:20/05/28 19:17:19.316 Executor task launch worker for task 29 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/15/shuffle_0_18_0.data
646:20/05/28 19:17:20.375 Executor task launch worker for task 30 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/0a/shuffle_0_19_0.data
671:20/05/28 19:17:20.386 Executor task launch worker for task 31 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/33/shuffle_0_10_0.data
701:20/05/28 19:17:20.411 Executor task launch worker for task 33 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/31/shuffle_0_12_0.data
726:20/05/28 19:17:20.422 Executor task launch worker for task 34 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/30/shuffle_0_13_0.data
751:20/05/28 19:17:20.433 Executor task launch worker for task 35 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/2f/shuffle_0_14_0.data
776:20/05/28 19:17:20.443 Executor task launch worker for task 36 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/0e/shuffle_0_15_0.data
801:20/05/28 19:17:20.454 Executor task launch worker for task 37 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/13/shuffle_0_16_0.data
826:20/05/28 19:17:20.466 Executor task launch worker for task 38 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/0c/shuffle_0_17_0.data
851:20/05/28 19:17:20.476 Executor task launch worker for task 39 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/15/shuffle_0_18_0.data
app-20200528101709-0000/1/target/unit-tests.log
97:20/05/28 19:17:14.044 Executor task launch worker for task 11 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-f1e44135-db75-4b91-88c1-4f8819398e51/blockmgr-bb40b0c5-fd6e-45fd-8a2f-d3f3d38fd441/2e/shuffle_0_11_0.data
114:20/05/28 19:17:14.363 Executor task launch worker for task 11 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-f1e44135-db75-4b91-88c1-4f8819398e51/blockmgr-bb40b0c5-fd6e-45fd-8a2f-d3f3d38fd441/2e/shuffle_0_11_0.data
132:20/05/28 19:17:19.231 shuffle-server-6-1 INFO IndexShuffleBlockResolver:
Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-f1e44135-db75-4b91-88c1-4f8819398e51/blockmgr-bb40b0c5-fd6e-45fd-8a2f-d3f3d38fd441/2e/shuffle_0_11_0.data
189:20/05/28 19:17:20.397 shuffle-server-6-1 INFO IndexShuffleBlockResolver:
Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-f1e44135-db75-4b91-88c1-4f8819398e51/blockmgr-bb40b0c5-fd6e-45fd-8a2f-d3f3d38fd441/2e/shuffle_0_11_0.data
```
So from this you can see `shuffle_0_11_0.data` is only accessed in one of
the executor. You can see this more easily by this:
```
╭─attilazsoltpiros@apiros-MBP16 ~/git/attilapiros/reviewing/work ‹pr/28331*›
╰─$ ag shuffle_0_11_0.data
app-20200528101709-0000/1/target/unit-tests.log
97:20/05/28 19:17:14.044 Executor task launch worker for task 11 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-f1e44135-db75-4b91-88c1-4f8819398e51/blockmgr-bb40b0c5-fd6e-45fd-8a2f-d3f3d38fd441/2e/shuffle_0_11_0.data
114:20/05/28 19:17:14.363 Executor task launch worker for task 11 INFO
IndexShuffleBlockResolver: Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-f1e44135-db75-4b91-88c1-4f8819398e51/blockmgr-bb40b0c5-fd6e-45fd-8a2f-d3f3d38fd441/2e/shuffle_0_11_0.data
132:20/05/28 19:17:19.231 shuffle-server-6-1 INFO IndexShuffleBlockResolver:
Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-f1e44135-db75-4b91-88c1-4f8819398e51/blockmgr-bb40b0c5-fd6e-45fd-8a2f-d3f3d38fd441/2e/shuffle_0_11_0.data
189:20/05/28 19:17:20.397 shuffle-server-6-1 INFO IndexShuffleBlockResolver:
Attila: shuffle file:
/Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-f1e44135-db75-4b91-88c1-4f8819398e51/blockmgr-bb40b0c5-fd6e-45fd-8a2f-d3f3d38fd441/2e/shuffle_0_11_0.data
╭─attilazsoltpiros@apiros-MBP16 ~/git/attilapiros/reviewing/work ‹pr/28331*›
```
And you can check the call stack too for these accesses.
We should harden this test to produce failure when this happens. The test
should fail even when the decommission is not triggered.
And I believe we need some tests at deeper level (this one might be too
high: way above unit testing).
And It came to my mind how good it would be to have some metrics for the
number of migrated blocks too (but is not a priority but something to think
about).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
