[
https://issues.apache.org/jira/browse/PARQUET-2227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17677039#comment-17677039
]
ASF GitHub Bot commented on PARQUET-2227:
-----------------------------------------
wgtmac commented on PR #1014:
URL: https://github.com/apache/parquet-mr/pull/1014#issuecomment-1383160117
> I agree that merging the key-value metadata is not an easy question. Let's
discuss it separately as it is not related to this PR.
>
> I also agree to store the current writer (parquet-mr) in `created_by` in
case of rewriting. It is not easy to decide what would be the proper solution
anyway. `created_by` is usually used for handling potential erroneous writes.
Let's say there was an issue in parquet-mr at the version 1.2.3 that written a
specific encoding of integers wrongly (not according to spec). What if we
rewrite the file but do not re-encode the pages? Can we still handle the
original issue? What if the rewriter re-encodes the related pages? Let's store
the original writer in `original.created.by` for now. Let's discuss this topic
separately however, I am not sure if we can find a proper solution.
I agree. Now I have updated this PR to preserve the old writer version into
`original.created.by` and added a test to make sure it is preserved. Please
take a look. Thanks!
> Refactor different file rewriters to use single implementation
> --------------------------------------------------------------
>
> Key: PARQUET-2227
> URL: https://issues.apache.org/jira/browse/PARQUET-2227
> Project: Parquet
> Issue Type: Sub-task
> Components: parquet-mr
> Reporter: Gang Wu
> Assignee: Gang Wu
> Priority: Major
>
> A new ParquetRewriter is implemented to support all logics in the
> ColumnPruner, CompressionConverter, ColumnMasker, and ColumnEncrypter. And
> refactor all the old rewriters to use ParquetRewriter under the hood.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
