[
https://issues.apache.org/jira/browse/NIFI-12426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17790792#comment-17790792
]
Matt Burgess commented on NIFI-12426:
-------------------------------------
Ugh I forgot it doesn't parse correctly. Are we going to backport the
DateTimeFormatter stuff to 1.x?
As a workaround the user will have to manually truncate the microseconds in the
meantime. If you can configure a RecordReader to ensure it will ensure it will
infer timestamps as strings (such as JsonTreeReader with no Timestamp Format
set), you can use JoltTransform if you know the timestamp fields and length of
the values using a spec such as:
{{[
{
"operation": "modify-overwrite-beta",
"spec": {
"ts_evt": "=substring(@(1,ts_evt),0,23)"
}
}
]}}
or a ScriptedTransformRecord using a Regex to programmatically find all the
timestamp fields with microseconds and truncate them. Then downstream you can
use a Reader that infers using a Timestamp Format of milliseconds and it will
infer the values as timestamps correctly.
> Support microseconds in RegexDateTimeMatcher
> --------------------------------------------
>
> Key: NIFI-12426
> URL: https://issues.apache.org/jira/browse/NIFI-12426
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Reporter: Matt Burgess
> Assignee: Matt Burgess
> Priority: Major
> Fix For: 1.25.0, 2.0.0
>
>
> If a timestamp in the input has microseconds and a RecordReader is using
> Infer Schema, the data type will be inferred as a string rather than a
> timestamp regardless of the Timestamp Format property in the reader. Although
> SimpleDateFormat doesn't support microseconds, it is forgiving in the parsing
> of a timestamp string and accepts ".SSSSSS" as a milliseconds format even
> though the microseconds will not be honored.
> However when inferring the schema, the input must also pass the
> RegexDateTimeMatcher which checks that it "looks like" a timestamp and within
> the legitimate length boundaries. This matcher enforces a 3-digit length of
> milliseconds and will fail to match input with microseconds. This matcher
> should accept 6 digits of fractional seconds and allow the other matchers to
> proceed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
