[
https://issues.apache.org/jira/browse/PARQUET-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16861280#comment-16861280
]
ASF GitHub Bot commented on PARQUET-1580:
-----------------------------------------
bbraams commented on pull request #647: PARQUET-1580: Page-level CRC checksum
verfication for DataPageV1
URL: https://github.com/apache/parquet-mr/pull/647
This PR implements page-level CRC checksums for DataPageV1. It is the
implementation follow up to the clarification of the checksums in
`parquet-format` ([Jira](https://jira.apache.org/jira/browse/PARQUET-1539),
[PR](https://github.com/apache/parquet-format/pull/126)). Key points:
- Checksums are calculated using the CRC32 implementation in `java.util.zip`
- Disabled by default, writing out checksums on the write path and
verification on the read path are feature flagged individually (see
Documentation below)
- Backwards compatibility with Parquet files that do not have the CRC field
set, even if checksum verification is turned on
Please excuse the size of the PR, the bulk of it is config propagation
boilerplate.
### Jira
- [x] My PR addresses the following [Parquet
Jira](https://jira.apache.org/jira/browse/PARQUET-1580) issue:
- https://jira.apache.org/jira/browse/PARQUET-1580
### Tests
This PR adds the following tests in a new test suite
`TestDataPageV1Checksums`:
- Test writing out checksums without verification on read, verify calculated
checksums are correct
- Test not writing out checksums without verification on read, tests feature
can be turned off entirely
- Test not writing out checksums with verification on read, tests backward
compatibility of read path with files without checksums
- Test writing out checksums with verification on read, tests that the
checksum is correctly set and that we can read what we wrote
- Test corruption detection by corrupting bytes in data section of pages,
tests that these corruptions go undetected with the feature disabled and are
detected with the feature enabled
- Test checksums with compression enabled, tests that the checksum is
calculated on the compressed version of the data (as per the specification)
- Test checksums with compression and nested schema, tests that the checksum
is calculated on the compressed concatenation of the repetition levels,
definition levels and the actual data (as per the specification)
### Documentation
The feature is feature flagged and is disabled by default. Both writing out
checksums and verifying them on the read path can be turned on individually,
via the following two new config flags:
* `parquet.page.write-checksum.enabled` (default: false)
* `parquet.page.verify-checksum.enabled` (default: false)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Page-level CRC checksum verification for DataPageV1
> ---------------------------------------------------
>
> Key: PARQUET-1580
> URL: https://issues.apache.org/jira/browse/PARQUET-1580
> Project: Parquet
> Issue Type: Improvement
> Components: parquet-mr
> Reporter: Boudewijn Braams
> Assignee: Boudewijn Braams
> Priority: Major
> Labels: pull-request-available
>
> Now that the specification with respect to page level checksums has been
> clarified (see https://jira.apache.org/jira/browse/PARQUET-1539), we can
> actually implement it. What needs to be done:
> * Implement calculating and writing checksums on write path
> * Implement checksum validation on read path
> * Make writing out checksums and validation on reads optional (default off)
> * Implement tests
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
