[
https://issues.apache.org/jira/browse/HIVE-21924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880913#comment-16880913
]
Sankar Hariappan commented on HIVE-21924:
-----------------------------------------
Currently, header and footer skipping logic is done in FetchOperator which
works fine if we perform "select * from table" but it returns incorrect result
if predicates or aggregates are specified in the query.
https://github.com/apache/hive/blob/f923b7490a6085292e2d05425dcdc83b994d42e4/ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java#L572-L582
{code}
headerCount = Utilities.getHeaderCount(currDesc.getTableDesc());
footerCount = Utilities.getFooterCount(currDesc.getTableDesc(), job);
// Skip header lines.
opNotEOF = Utilities.skipHeader(currRecReader, headerCount, key,
value);
// Initialize footer buffer.
if (opNotEOF && footerCount > 0) {
footerBuffer = new FooterBuffer();
opNotEOF = footerBuffer.initializeBuffer(job, currRecReader,
footerCount, key, value);
}
{code}
Changing the split generation logic as mentioned in the description would
resolve this issue.
> Split text files even if header/footer exists
> ---------------------------------------------
>
> Key: HIVE-21924
> URL: https://issues.apache.org/jira/browse/HIVE-21924
> Project: Hive
> Issue Type: Improvement
> Components: File Formats
> Affects Versions: 2.4.0, 4.0.0, 3.2.0
> Reporter: Prasanth Jayachandran
> Priority: Major
>
> https://github.com/apache/hive/blob/967a1cc98beede8e6568ce750ebeb6e0d048b8ea/ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java#L494-L503
>
> {code}
> int headerCount = 0;
> int footerCount = 0;
> if (table != null) {
> headerCount = Utilities.getHeaderCount(table);
> footerCount = Utilities.getFooterCount(table, conf);
> if (headerCount != 0 || footerCount != 0) {
> // Input file has header or footer, cannot be splitted.
> HiveConf.setLongVar(conf, ConfVars.MAPREDMINSPLITSIZE,
> Long.MAX_VALUE);
> }
> }
> {code}
> this piece of code makes the CSV (or any text files with header/footer) files
> not splittable if header or footer is present.
> If only header is present, we can find the offset after first line break and
> use that to split. Similarly for footer, may be read few KB's of data at the
> end and find the last line break offset. Use that to determine the data range
> which can be used for splitting. Few reads during split generation are
> cheaper than not splitting the file at all.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
