[
https://issues.apache.org/jira/browse/IMPALA-5050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16773066#comment-16773066
]
Csaba Ringhofer commented on IMPALA-5050:
-----------------------------------------
[~arodoni_cloudera]
I found a few points that should be extended:
https://github.com/apache/impala/blob/b8a8edddcb727a28c2d15bdb3533a32454364ade/docs/topics/impala_parquet.xml#L1120
INT64 + OriginalType TIMESTAMP_MILLIS -> TIMESTAMP
INT64 + OriginalType TIMESTAMP_MICROS -> TIMESTAMP
INT64 + LogicalType TIMESTAMP -> TIMESTAMP
Note that these columns can be still read as BIGINT too, so existing queries
will work the same way as they used to.
https://github.com/apache/impala/blob/b8a8edddcb727a28c2d15bdb3533a32454364ade/docs/shared/impala_common.xml#L2149
I think that these columns written by Sqoop can be read by Impala after this
change, but I didn't verify this.
https://github.com/apache/impala/blob/b8a8edddcb727a28c2d15bdb3533a32454364ade/docs/topics/impala_timestamp.xml#L197
It could be mentioned that Hive cannot write INT64 timestamps at the moment,
but the implementation is in progress: HIVE-21216
https://github.com/apache/impala/blob/b8a8edddcb727a28c2d15bdb3533a32454364ade/docs/topics/impala_timestamp.xml#L218
It should be mentioned that convert_legacy_hive_parquet_utc_timestamps only
affects INT96 timestamps. INT64 timestamp with only OriginalType are assumed to
be always UTC normalized, so the UTC->local conversion will be always done.
INT64 timestamps with LogicalType specify whether UTC->local conversion is
necessary depending in the Parquet metadata.
> Add support to read TIMESTAMP_MILLIS and TIMESTAMP_MICROS to the parquet
> scanner
> --------------------------------------------------------------------------------
>
> Key: IMPALA-5050
> URL: https://issues.apache.org/jira/browse/IMPALA-5050
> Project: IMPALA
> Issue Type: New Feature
> Components: Backend
> Affects Versions: Impala 2.9.0
> Reporter: Lars Volker
> Assignee: Csaba Ringhofer
> Priority: Major
> Fix For: Impala 3.2.0
>
>
> This requires updating {{parquet.thrift}} to a version that includes the
> {{TIMESTAMP_MICROS}} logical type.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
