Gabor Szadovszky created PARQUET-1977:
-----------------------------------------
Summary: Invalid data_page_offset
Key: PARQUET-1977
URL: https://issues.apache.org/jira/browse/PARQUET-1977
Project: Parquet
Issue Type: Bug
Components: parquet-mr
Affects Versions: 1.12.0
Reporter: Gabor Szadovszky
Assignee: Gabor Szadovszky
The thrift field
[{{data_page_offset}}|https://github.com/gszadovszky/parquet-format/blob/master/src/main/thrift/parquet.thrift#L733]
is filled with incorrect value. Currently, it always points to the beginning
of the column chunk which is not correct according to the spec in case there is
a dictionary page. This is not a regression as it was written incorrectly since
the beginning of parquet-mr.
Meanwhile PARQUET-1850 fixed that we never wrote the field
[{{dictionary_page_offset}}|https://github.com/gszadovszky/parquet-format/blob/master/src/main/thrift/parquet.thrift#L739].
After the fix we correctly write this field if there is a dictionary filter.
The problem is we are using the same value to fill both fields. So there are
two possibilities:
* There is no dictionary page in the column chunk so {{data_page_offset}} is
filled with the correct value while {{dictionary_page_offset}} is not filled
which is still correct. We are good.
* There is a dictionary page at the beginning of the column chunk so
{{data_page_offset}} and {{dictionary_page_offset}} are both contains the same
value. This is not only a regression but it causes issues in other
implementations (e.g. Impala) where footer validation is more strict than in
parquet-mr because {{dictionary_page_offset}} shall be less than
{{data_page_offset}} at all time if it is filled.
So, we need to fill {{data_page_offset}} correctly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
