Github user echeipesh commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/25#discussion_r27981389
--- Diff:
core/src/main/java/org/apache/accumulo/core/client/mapred/AbstractInputFormat.java
---
@@ -629,32 +676,37 @@ public float getProgress() throws IOException {
}
for (Map.Entry<KeyExtent,List<Range>> extentRanges :
tserverBin.getValue().entrySet()) {
Range ke = extentRanges.getKey().toDataRange();
- for (Range r : extentRanges.getValue()) {
- if (autoAdjust) {
- // divide ranges into smaller ranges, based on the tablets
- RangeInputSplit split = new RangeInputSplit(tableName,
tableId, ke.clip(r), new String[] {location});
-
- split.setOffline(tableConfig.isOfflineScan());
-
split.setIsolatedScan(tableConfig.shouldUseIsolatedScanners());
-
split.setUsesLocalIterators(tableConfig.shouldUseLocalIterators());
- split.setMockInstance(mockInstance);
- split.setFetchedColumns(tableConfig.getFetchedColumns());
- split.setPrincipal(principal);
- split.setToken(token);
- split.setInstanceName(instance.getInstanceName());
- split.setZooKeepers(instance.getZooKeepers());
- split.setAuths(auths);
- split.setIterators(tableConfig.getIterators());
- split.setLogLevel(logLevel);
-
- splits.add(split);
- } else {
- // don't divide ranges
- ArrayList<String> locations = splitsToAdd.get(r);
- if (locations == null)
- locations = new ArrayList<String>(1);
- locations.add(location);
- splitsToAdd.put(r, locations);
+ if (batchScan) {
+ // group ranges by tablet to be read by a BatchScanner
+ ArrayList<Range> clippedRanges = new ArrayList<Range>();
+ for(Range r: extentRanges.getValue())
--- End diff --
The intent of the change is to keep splits local and batch access to them
to reduce overhead. This implies clipping the ranges. The alternative could be
a split with two locations that has 90%+ of ranges on vs the other location. As
I understand the scheduler at that point has no weight for locality preference.
On other hand having option to not clip would make the options more
logically consistent and justify having number of threads per batch reader as
configuration.
I personally think the first option breaks the least eggs, what do you
think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
