[
https://issues.apache.org/jira/browse/PARQUET-2042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17501903#comment-17501903
]
ASF GitHub Bot commented on PARQUET-2042:
-----------------------------------------
mwong38 commented on pull request #900:
URL: https://github.com/apache/parquet-mr/pull/900#issuecomment-1059918680
After proto3 made everything optional, there is no way to know whether a
primitive has been set or not. That is, you could no longer represent a
"nullable" primitive. (They later brought `optional` keyword back, but the
damage is done). The solution was to wrap a primitive in a message. For
example, to represent a nullable `double`, there is the
[DoubleValue](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#doublevalue).
It's analogous to Java's boxed `Double`.
Parquet supports nullable primitives natively. If we were to represent these
well-known Protobuf types directly, it will be nested inside a deeper data
structure; very inconvenient and a waste of space. What I did here is "unwrap"
these primitives and make use of Parquet's nullaibility.
Additionally, I convert other well-known types such as Timestamp and Date to
Parquet's. Again, rather than represent the data structure in its raw nested
form (seconds/nanos or year, month, day, etc), they are converted to Parquet's
native or logical representation of Timestamp and Date. You can turn on or off
this features in the `ProtoSchemaConverter` constructor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
> Unwrap common Protobuf wrappers and logical Timestamps, Date, TimeOfDay
> -----------------------------------------------------------------------
>
> Key: PARQUET-2042
> URL: https://issues.apache.org/jira/browse/PARQUET-2042
> Project: Parquet
> Issue Type: Improvement
> Components: parquet-protobuf
> Reporter: Michael Wong
> Priority: Major
>
> Related to https://issues.apache.org/jira/browse/PARQUET-1595
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
