[
https://issues.apache.org/jira/browse/ACCUMULO-3602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14486048#comment-14486048
]
ASF GitHub Bot commented on ACCUMULO-3602:
------------------------------------------
Github user ctubbsii commented on the pull request:
https://github.com/apache/accumulo/pull/25#issuecomment-91037784
In JIRA, I [mentioned][1] "sometimes it's better to query a larger range
and let an iterator filter out non-matching results".
I think the createRanges method @keith-turner describes could work if the
function is executed in the RecordReader (it also simplifies this issue
significantly, because you wouldn't need to create a new InputSplit type, but
simply add an option to the AccumuloInputFormat). There's still some risk of
memory exhaustion with a large number of ranges within a tablet (especially if
the ranges were an exhaustive set of row-records to retrieve).
However, I still think that for many things, it's probably better to simply
use an iterator with some filter criteria. It could be a SkippingIterator that
seeks to ranges which are pre-configured on that iterator, or it could be a
Filter which has some filter criteria.
[1]:
https://issues.apache.org/jira/browse/ACCUMULO-3602?focusedCommentId=14327767&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14327767
> BatchScanner optimization for AccumuloInputFormat
> -------------------------------------------------
>
> Key: ACCUMULO-3602
> URL: https://issues.apache.org/jira/browse/ACCUMULO-3602
> Project: Accumulo
> Issue Type: Improvement
> Components: client
> Affects Versions: 1.6.1, 1.6.2
> Reporter: Eugene Cheipesh
> Assignee: Eugene Cheipesh
> Labels: performance
> Fix For: 1.7.0
>
>
> Currently {{AccumuloInputFormat}} produces a split for reach {{Range}}
> specified in the configuration. Some table indexing schemes, for instance
> z-order geospacial index, produce large number of small ranges resulting in
> large number of splits. This is specifically a concern when using
> {{AccumuloInputFormat}} as a source for Spark RDD where each Split is mapped
> to an RDD partition.
> Large number of small RDD partitions leads to poor parallism on read and high
> overhead on processing. A desirable alternative is to group ranges by tablet
> into a single split and use {{BatchScanner}} to produce the records. Grouping
> by tablets is useful because it represents Accumulos attempt to distributed
> stored records and can be influance by the user through table splits.
> The grouping functionality already exists in the internal {{TabletLocator}}
> class.
> Current proposal is to modify {{AbstractInputFormat}} such that it generates
> either {{RangeInputSplit}} or {{MultiRangeInputSplit}} based on a new setting
> in {{InputConfigurator}}. {{AccumuloInputFormat}} would then be able to
> inspect the type of the split and instantiate an appropriate reader.
> The functinality of {{TabletLocator}} should be exposed as a public API in
> 1.7 as it is useful for optimizations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
