[
https://issues.apache.org/jira/browse/ACCUMULO-3602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14484080#comment-14484080
]
ASF GitHub Bot commented on ACCUMULO-3602:
------------------------------------------
Github user keith-turner commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/25#discussion_r27921845
--- Diff:
core/src/main/java/org/apache/accumulo/core/client/mapreduce/lib/impl/InputConfigurator.java
---
@@ -517,6 +517,74 @@ public static Boolean isOfflineScan(Class<?>
implementingClass, Configuration co
}
/**
+ * Controls the use of the {@link BatchScanner} in this job.
+ * Using this feature will group ranges by their source tablet per
InputSplit and use BatchScanner to read them.
+ *
+ * <p>
+ * By default, this feature is <b>disabled</b>.
+ *
+ * @param implementingClass
+ * the class whose name will be used as a prefix for the
property configuration key
+ * @param conf
+ * the Hadoop configuration object to configure
+ * @param enableFeature
+ * the feature is enabled if true, disabled otherwise
+ * @since 1.7.0
+ */
+ public static void setBatchScan(Class<?> implementingClass,
Configuration conf, boolean enableFeature) {
+ conf.setBoolean(enumToConfKey(implementingClass,
Features.BATCH_SCANNER), enableFeature);
+ }
+
+ /**
+ * Determines whether a configuration has the BatchScanner feature
enabled.
+ *
+ * @param implementingClass
+ * the class whose name will be used as a prefix for the
property configuration key
+ * @param conf
+ * the Hadoop configuration object to configure
+ * @return true if the feature is enabled, false otherwise
+ * @since 1.7.0
+ * @see #setBatchScan(Class, Configuration, boolean)
+ */
+ public static Boolean isBatchScan(Class<?> implementingClass,
Configuration conf) {
+ return conf.getBoolean(enumToConfKey(implementingClass,
Features.BATCH_SCANNER), false);
+ }
+
+ /**
+ * Controls the use of the {@link BatchScanner} in this job.
+ * Using this feature will group ranges by their source tablet per
InputSplit and use BatchScanner to read them.
+ *
+ * <p>
+ * By default, this feature is <b>disabled</b>.
+ *
+ * @param implementingClass
+ * the class whose name will be used as a prefix for the
property configuration key
+ * @param conf
+ * the Hadoop configuration object to configure
+ * @param count
+ * number of threads to use with BatchScanner
+ * @since 1.7.0
+ */
+ public static void setBatchScanThreads(Class<?> implementingClass,
Configuration conf, int count) {
--- End diff --
The batch scanner will not use more than one thread per tablet. If each
mapper is using a batch scanner to read multiple ranges from a single tablet,
then the batch scanner will only use a single thread to do this even if it has
more threads it could use.
> BatchScanner optimization for AccumuloInputFormat
> -------------------------------------------------
>
> Key: ACCUMULO-3602
> URL: https://issues.apache.org/jira/browse/ACCUMULO-3602
> Project: Accumulo
> Issue Type: Improvement
> Components: client
> Affects Versions: 1.6.1, 1.6.2
> Reporter: Eugene Cheipesh
> Assignee: Eugene Cheipesh
> Labels: performance
> Fix For: 1.7.0
>
>
> Currently {{AccumuloInputFormat}} produces a split for reach {{Range}}
> specified in the configuration. Some table indexing schemes, for instance
> z-order geospacial index, produce large number of small ranges resulting in
> large number of splits. This is specifically a concern when using
> {{AccumuloInputFormat}} as a source for Spark RDD where each Split is mapped
> to an RDD partition.
> Large number of small RDD partitions leads to poor parallism on read and high
> overhead on processing. A desirable alternative is to group ranges by tablet
> into a single split and use {{BatchScanner}} to produce the records. Grouping
> by tablets is useful because it represents Accumulos attempt to distributed
> stored records and can be influance by the user through table splits.
> The grouping functionality already exists in the internal {{TabletLocator}}
> class.
> Current proposal is to modify {{AbstractInputFormat}} such that it generates
> either {{RangeInputSplit}} or {{MultiRangeInputSplit}} based on a new setting
> in {{InputConfigurator}}. {{AccumuloInputFormat}} would then be able to
> inspect the type of the split and instantiate an appropriate reader.
> The functinality of {{TabletLocator}} should be exposed as a public API in
> 1.7 as it is useful for optimizations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
