[GitHub] jeongyooneo commented on a change in pull request #99: [NEMO-64] Fix map stage hang under DataSkewPolicy
jeongyooneo commented on a change in pull request #99: [NEMO-64] Fix map stage
hang under DataSkewPolicy
URL: https://github.com/apache/incubator-nemo/pull/99#discussion_r209884464
##
File path: runtime/common/src/main/proto/ControlMessage.proto
##
@@ -117,8 +117,6 @@ message BlockStateChangedMsg {
message DataSizeMetricMsg {
// TODO #96: Modularize DataSkewPolicy to use MetricVertex and
BarrierVertex.
repeated PartitionSizeEntry partitionSize = 1;
-required string blockId = 2;
Review comment:
Now metric vertex only acts as a barrier, and all run-time optimization is
done via ExecutionProperties of IREdges. So id of metric vertex isn't needed
anymore.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] jeongyooneo commented on a change in pull request #99: [NEMO-64] Fix map stage hang under DataSkewPolicy
jeongyooneo commented on a change in pull request #99: [NEMO-64] Fix map stage hang under DataSkewPolicy URL: https://github.com/apache/incubator-nemo/pull/99#discussion_r209194883 ## File path: runtime/master/src/main/java/edu/snu/nemo/runtime/master/scheduler/BatchScheduler.java ## @@ -93,6 +93,7 @@ private BatchScheduler(final TaskDispatcher taskDispatcher, .subscribe(updatePhysicalPlanEventHandler.getEventClass(), updatePhysicalPlanEventHandler); } this.executorRegistry = executorRegistry; +this.dataSkewDynOptDataHandler = new DataSkewDynOptDataHandler(); Review comment: For that case, I'll add `DynOptDataHandlerMap` to keep a list of data handlers and add comments on it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jeongyooneo commented on a change in pull request #99: [NEMO-64] Fix map stage hang under DataSkewPolicy
jeongyooneo commented on a change in pull request #99: [NEMO-64] Fix map stage hang under DataSkewPolicy URL: https://github.com/apache/incubator-nemo/pull/99#discussion_r209194883 ## File path: runtime/master/src/main/java/edu/snu/nemo/runtime/master/scheduler/BatchScheduler.java ## @@ -93,6 +93,7 @@ private BatchScheduler(final TaskDispatcher taskDispatcher, .subscribe(updatePhysicalPlanEventHandler.getEventClass(), updatePhysicalPlanEventHandler); } this.executorRegistry = executorRegistry; +this.dataSkewDynOptDataHandler = new DataSkewDynOptDataHandler(); Review comment: For that case, I'll refactor `DynOptDataHandler` to keep a list of data handlers and add comments on it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jeongyooneo commented on a change in pull request #99: [NEMO-64] Fix map stage hang under DataSkewPolicy
jeongyooneo commented on a change in pull request #99: [NEMO-64] Fix map stage hang under DataSkewPolicy URL: https://github.com/apache/incubator-nemo/pull/99#discussion_r209187268 ## File path: runtime/executor/src/test/java/edu/snu/nemo/runtime/executor/datatransfer/DataTransferTest.java ## @@ -300,6 +303,15 @@ private void writeAndRead(final BlockManagerWorker sender, dummyIREdge.setProperty(DataPersistenceProperty.of(DataPersistenceProperty.Value.Keep)); dummyIREdge.setProperty(EncoderProperty.of(ENCODER_FACTORY)); dummyIREdge.setProperty(DecoderProperty.of(DECODER_FACTORY)); +if (dummyIREdge.getPropertyValue(CommunicationPatternProperty.class).get() Review comment: It is actually used in `InputReader#read`(L355 and others). As now input data for each task is set as Execution Properties(`DefaultMetricPass`) instead of the constructor in `StageEdge`, we need to initialize it here for shuffle read. So this is not for testing, but simply a required process to enable `InputReader#read`. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jeongyooneo commented on a change in pull request #99: [NEMO-64] Fix map stage hang under DataSkewPolicy
jeongyooneo commented on a change in pull request #99: [NEMO-64] Fix map stage hang under DataSkewPolicy URL: https://github.com/apache/incubator-nemo/pull/99#discussion_r209187268 ## File path: runtime/executor/src/test/java/edu/snu/nemo/runtime/executor/datatransfer/DataTransferTest.java ## @@ -300,6 +303,15 @@ private void writeAndRead(final BlockManagerWorker sender, dummyIREdge.setProperty(DataPersistenceProperty.of(DataPersistenceProperty.Value.Keep)); dummyIREdge.setProperty(EncoderProperty.of(ENCODER_FACTORY)); dummyIREdge.setProperty(DecoderProperty.of(DECODER_FACTORY)); +if (dummyIREdge.getPropertyValue(CommunicationPatternProperty.class).get() Review comment: It is actually used in `InputReader#read`(L355 and others). As now input data for each task is set as Execution Properties(`DefaultMetricPass`) instead of the constructor in `StageEdge`, we need to initialize it here for shuffle read. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jeongyooneo commented on a change in pull request #99: [NEMO-64] Fix map stage hang under DataSkewPolicy
jeongyooneo commented on a change in pull request #99: [NEMO-64] Fix map stage hang under DataSkewPolicy URL: https://github.com/apache/incubator-nemo/pull/99#discussion_r209187268 ## File path: runtime/executor/src/test/java/edu/snu/nemo/runtime/executor/datatransfer/DataTransferTest.java ## @@ -300,6 +303,15 @@ private void writeAndRead(final BlockManagerWorker sender, dummyIREdge.setProperty(DataPersistenceProperty.of(DataPersistenceProperty.Value.Keep)); dummyIREdge.setProperty(EncoderProperty.of(ENCODER_FACTORY)); dummyIREdge.setProperty(DecoderProperty.of(DECODER_FACTORY)); +if (dummyIREdge.getPropertyValue(CommunicationPatternProperty.class).get() Review comment: It is actually used in `InputReader#read`(L355 and others). As now input data for each task is set as Execution Properties(`DefaultMetricPass`) instead of the constructor in `StageEdge`, we need to initialize it here for shuffle read. So this is not for testing, but simply required process to enable `InputReader#read`. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
