[
https://issues.apache.org/jira/browse/ACCUMULO-3602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14486343#comment-14486343
]
ASF GitHub Bot commented on ACCUMULO-3602:
------------------------------------------
Github user keith-turner commented on the pull request:
https://github.com/apache/accumulo/pull/25#issuecomment-91069467
I was discussing the use case I mentioned offline w/ @ctubbsii. This use
case was a large number of ranges that can not be generated by a function. We
determined that function could handle this case well by storing the ranges
somewhere else beside the job conf. For example could do the following.
* Store 10,000,000 sorted ranges in file in distributed cache (assume
thousands of tablets)
* Using the provided function, each mapper opens the file and reads the
ranges for the tablet its working on.
* The ranges returned by the function are used to initialize the batch
scanner for each mapper.
It seems like all of the use cases that the current implementation
satisfies could also be satisfied with the functor implementation.
If this PR does not follow the functor approach discussed, thats ok w/ me.
I can open up a follow on issue to record the discussion if its not pursued
here.
> BatchScanner optimization for AccumuloInputFormat
> -------------------------------------------------
>
> Key: ACCUMULO-3602
> URL: https://issues.apache.org/jira/browse/ACCUMULO-3602
> Project: Accumulo
> Issue Type: Improvement
> Components: client
> Affects Versions: 1.6.1, 1.6.2
> Reporter: Eugene Cheipesh
> Assignee: Eugene Cheipesh
> Labels: performance
> Fix For: 1.7.0
>
>
> Currently {{AccumuloInputFormat}} produces a split for reach {{Range}}
> specified in the configuration. Some table indexing schemes, for instance
> z-order geospacial index, produce large number of small ranges resulting in
> large number of splits. This is specifically a concern when using
> {{AccumuloInputFormat}} as a source for Spark RDD where each Split is mapped
> to an RDD partition.
> Large number of small RDD partitions leads to poor parallism on read and high
> overhead on processing. A desirable alternative is to group ranges by tablet
> into a single split and use {{BatchScanner}} to produce the records. Grouping
> by tablets is useful because it represents Accumulos attempt to distributed
> stored records and can be influance by the user through table splits.
> The grouping functionality already exists in the internal {{TabletLocator}}
> class.
> Current proposal is to modify {{AbstractInputFormat}} such that it generates
> either {{RangeInputSplit}} or {{MultiRangeInputSplit}} based on a new setting
> in {{InputConfigurator}}. {{AccumuloInputFormat}} would then be able to
> inspect the type of the split and instantiate an appropriate reader.
> The functinality of {{TabletLocator}} should be exposed as a public API in
> 1.7 as it is useful for optimizations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
