[
https://issues.apache.org/jira/browse/NIFI-16535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sebastian Kannengießer updated NIFI-16535:
------------------------------------------
Description:
During our migration tasks from NiFi 1.28.1 to NiFi 2.12.0 we found the
following problem/bug with NiFi's approach to handling Parquet file timestamps
with Type: INT64 physical type and Timestamp logical type with them being
attibuted not being not adjusted to UTC.
{{isAdjustedToUTC=false}}
as described here:
[https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#local-semantics-timestamps-not-normalized-to-utc]
NiFi's ParquetReader reads them as LocalDateTime types and fails casting them
to Long:
!image-2026-11-02-14-15-56-766.png|width=584,height=205!
A test parquet file is attached to this ticket.
We consider this a blocker, at least for us having Parquet-specification
conforming data, with currently no way circumventing this easily within NiFi.
I found the related issue [https://issues.apache.org/jira/browse/NIFI-15548]
which was fixed in 2.8.0. That one seems to have been focusing on the different
units (micros, millis, nanos) rather than the timestamps being local tz ones,
though.
A fix for this would be very much appreciated.
was:
During our migration tasks from NiFi 1.28.1 to NiFi 2.12.0 we found the
following problem/bug with NiFi's approach to handling Parquet file timestamps
with Type: INT64 physical type and Timestamp logical type with them being
attibuted not being not adjusted to UTC.
{{isAdjustedToUTC=false}}
as described here:
[https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#local-semantics-timestamps-not-normalized-to-utc]
NiFi's ParquetReader reads them as LocalDateTime types and fails casting them
to Long:
!image-2026-11-02-14-15-56-766.png|width=584,height=205!
A test parquet file is attached to this ticket.
We consider this a blocker, at least for us having Parquet-specification
conforming data, with currently no way circumventing this easily within NiFi.
I found the relating issue [https://issues.apache.org/jira/browse/NIFI-15548]
which was fixed in 2.8.0. That one seems to have been focusing on the different
units (micros, millis, nanos) rather than the timestamps being local tz ones,
though.
A fix for this would be very much appreciated.
> ParquetReader cannot handle timestamp fields with isAdjustedToUTC=false
> -----------------------------------------------------------------------
>
> Key: NIFI-16535
> URL: https://issues.apache.org/jira/browse/NIFI-16535
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: nifi-api-2.12.0
> Environment: Apache NiFi Docker Image with Hadoop and Parquet
> Nar-Bundles version 2.12.0 added
> Reporter: Sebastian Kannengießer
> Priority: Blocker
> Attachments: image-2026-11-02-14-15-56-766.png,
> nifi_local_timestamp_test.parquet
>
>
> During our migration tasks from NiFi 1.28.1 to NiFi 2.12.0 we found the
> following problem/bug with NiFi's approach to handling Parquet file
> timestamps with Type: INT64 physical type and Timestamp logical type with
> them being attibuted not being not adjusted to UTC.
> {{isAdjustedToUTC=false}}
> as described here:
> [https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#local-semantics-timestamps-not-normalized-to-utc]
>
> NiFi's ParquetReader reads them as LocalDateTime types and fails casting them
> to Long:
> !image-2026-11-02-14-15-56-766.png|width=584,height=205!
> A test parquet file is attached to this ticket.
> We consider this a blocker, at least for us having Parquet-specification
> conforming data, with currently no way circumventing this easily within NiFi.
> I found the related issue [https://issues.apache.org/jira/browse/NIFI-15548]
> which was fixed in 2.8.0. That one seems to have been focusing on the
> different units (micros, millis, nanos) rather than the timestamps being
> local tz ones, though.
>
> A fix for this would be very much appreciated.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
