abstractdog commented on code in PR #263:
URL: https://github.com/apache/tez/pull/263#discussion_r1104337334
##########
tez-mapreduce/src/main/java/org/apache/hadoop/mapred/split/TezGroupedSplitsInputFormat.java:
##########
@@ -129,14 +138,69 @@ public class TezGroupedSplitsRecordReader implements
RecordReader<K, V> {
int idx = 0;
long progress;
RecordReader<K, V> curReader;
-
+ private final AtomicInteger initIndex;
+ private final int numReaders;
+ private ExecutorService initReaderExecService;
+ private BlockingDeque<Future<RecordReader<K, V>>> initedReaders;
+ private AtomicBoolean failureOccurred = new AtomicBoolean(false);
+
public TezGroupedSplitsRecordReader(TezGroupedSplit split, JobConf job,
Reporter reporter) throws IOException {
this.groupedSplit = split;
this.job = job;
this.reporter = reporter;
+ this.initIndex = new AtomicInteger(0);
+ int numThreads =
conf.getInt(TezSplitGrouper.TEZ_GROUPING_SPLIT_INIT_THREADS,
+ TezSplitGrouper.TEZ_GROUPING_SPLIT_INIT_THREADS_DEFAULT);
+ this.numReaders = Math.min(groupedSplit.wrappedSplits.size(),
+
conf.getInt(TezSplitGrouper.TEZ_GROUPING_SPLIT_INIT_NUM_RECORDREADERS,
+
TezSplitGrouper.TEZ_GROUPING_SPLIT_INIT_NUM_RECORDREADERS_DEFAULT));
+ // skip multi-threaded split opening when number of readers is less than
1
Review Comment:
if you skip ONLY when numReaders is less than, then you actually do:
```
if (numReaders >= 1){
}
...skip...
```
but the intention is:
```
if (numReaders > 1){
}
...skip...
```
so skip if numReaders is less than or equal to 1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
