[
https://issues.apache.org/jira/browse/DRILL-5106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15722892#comment-15722892
]
Chunhui Shi commented on DRILL-5106:
------------------------------------
And one more improvement can be made: bufAdd keep pushing and popping if
footerCount is 0, this is unnecessary cost.
The same when headerCount is zero, we don't have to do doSkipHeader all the
time. Even when headerCount is not zero, Ideally we should not need to do
doSkipHeader all the time if there is a way we could skip enough records then
we go to another stage/function there we don't need to check skip or not again
and again.
> Refactor SkipRecordsInspector to exclude check for predefined file formats
> --------------------------------------------------------------------------
>
> Key: DRILL-5106
> URL: https://issues.apache.org/jira/browse/DRILL-5106
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - Hive
> Affects Versions: 1.9.0
> Reporter: Arina Ielchiieva
> Priority: Minor
>
> After changes introduced in DRILL-4982, SkipRecordInspector is used only for
> predefined formats (using hasHeaderFooter: false / true). But
> SkipRecordInspector has its own check for formats where skip strategy can be
> applied. Acceptable file formats are stored in private final Set<Object>
> fileFormats and initialized in constructor, currently it contains only one
> format - TextInputFormat. Now this check is redundant and may lead to
> ignoring hasHeaderFooter setting to true for any other format except of Text.
> To do:
> 1. remove private final Set<Object> fileFormats
> 2. remove if block from SkipRecordsInspector.retrievePositiveIntProperty:
> {code}
> if
> (!fileFormats.contains(tableProperties.get(hive_metastoreConstants.FILE_INPUT_FORMAT)))
> {
> return propertyIntValue;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
