johnyangk commented on a change in pull request #2: [NEMO-7] Intra-TaskGroup 
pipelining
URL: https://github.com/apache/incubator-nemo/pull/2#discussion_r172710971
 
 

 ##########
 File path: 
runtime/executor/src/main/java/edu/snu/nemo/runtime/executor/TaskGroupExecutor.java
 ##########
 @@ -60,13 +55,36 @@
   private final DataTransferFactory channelFactory;
   private final MetricCollector metricCollector;
 
-  /**
-   * Map of task IDs in this task group to their readers/writers.
-   */
-  private final Map<String, List<InputReader>> physicalTaskIdToInputReaderMap;
-  private final Map<String, List<OutputWriter>> 
physicalTaskIdToOutputWriterMap;
-
-  private boolean isExecutionRequested;
+  // Map of task ID to its intra-TaskGroup data pipe.
+  private final Map<Task, List<LocalPipe>> taskToInputPipesMap;
+  private final Map<Task, LocalPipe> taskToOutputPipeMap;  // one and only one 
Pipe per task
+  // Readers/writers that deals with inter-TaskGroup data.
+  private final List<InputReader> inputReaders;
+  private final Map<Task, List<InputReader>> taskToInputReadersMap;
+  private final Map<Task, List<InputReader>> taskToSideInputReadersMap;
+  private final Map<Task, List<OutputWriter>> taskToOutputWritersMap;
+  private final Map<InputReader, List<Task>> inputReaderToTasksMap;
+  private final Map<String, Iterator> idToSrcIteratorMap;
+  private final Map<String, List<Task>> srcIteratorIdToTasksMap;
+  private final Map<String, List<Task>> iteratorIdToTasksMap;
+  private final LinkedBlockingQueue<Pair<String, 
DataUtil.IteratorWithNumBytes>> iteratorQueue;
+  private volatile Map<String, List<Task>> pipeIdToDstTasksMap;
+  private final Set<Transform> preparedTransforms;
+  private final Set<String> finishedTaskIds;
+  private final AtomicInteger completedFutures;
+  private int numBoundedSources;
+  private int numIterators;
+
+  // For metrics
+  private long serBlockSize;
+  private long encodedBlockSize;
+  private long accumulatedBlockedReadTime;
+
+  private volatile boolean isExecutionRequested;
+  private volatile String logicalTaskIdPutOnHold;
+
+  private static final String ITERATORID_PREFIX = "ITERATOR_";
 
 Review comment:
   Can you do without iterator ids?
   (e.g., composing iterators, using pointers in a DAG)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Reply via email to