[
https://issues.apache.org/jira/browse/IMPALA-10186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17697848#comment-17697848
]
Gabor Kaszab commented on IMPALA-10186:
---------------------------------------
Additionally, Hive is not able to read this parquet file and reports error:
{code:java}
Caused by: java.io.EOFException: Cannot seek to negative offset
at org.apache.hadoop.hdfs.DFSInputStream.seek(DFSInputStream.java:1463)
at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:66)
at
org.apache.parquet.hadoop.util.H2SeekableInputStream.seek(H2SeekableInputStream.java:60)
at
org.apache.parquet.hadoop.ParquetFileReader$ConsecutivePartList.readAll(ParquetFileReader.java:1410)
at
org.apache.parquet.hadoop.ParquetFileReader.readNextFilteredRowGroup(ParquetFileReader.java:926){code}
> Write invalid parquet PageLocations which table sort by some columns
> --------------------------------------------------------------------
>
> Key: IMPALA-10186
> URL: https://issues.apache.org/jira/browse/IMPALA-10186
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Reporter: guojingfeng
> Assignee: Zoltán Borók-Nagy
> Priority: Major
> Labels: parquet
>
> Current parquet writer write -1 of PageLocation.offset and
> PageLocation.first_row_index when meet a empty page.
> hdfs-parquet-file-writer.cc Line: 808 ~ 819
> {code:java}
> // Write data pages
> for (const DataPage& page : pages_) {
> if (page.header.data_page_header.num_values == 0) {
> // Skip empty pages
> location.offset = -1;
> location.compressed_page_size = 0;
> location.first_row_index = -1;
> AddLocationToOffsetIndex(location);
> continue;
> }
> {code}
> But -1 values may cause ComputeCandidatePages function run into unexpected
> status.
> {code:java}
> bool ComputeCandidatePages(
> const vector<parquet::PageLocation>& page_locations,
> const vector<RowRange>& candidate_ranges,
> const int64_t num_rows, vector<int>* candidate_pages) {
> if (!ValidatePageLocations(page_locations, num_rows)) return false
> {code}
> and then cause IMPALA-9952
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
