[jira] [Commented] (NIFI-12885) MapRecord.getAsDate timestamp breaking bug
[
https://issues.apache.org/jira/browse/NIFI-12885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840646#comment-17840646
]
ASF subversion and git services commented on NIFI-12885:
Commit b6d044853cfcf568c246362bb5a337ea3bf2d951 in nifi's branch
refs/heads/main from knguyen1
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=b6d044853c ]
NIFI-12885 Added Record Methods for Local and Offset Dates (#8502)
Signed-off-by: David Handermann
> MapRecord.getAsDate timestamp breaking bug
> --
>
> Key: NIFI-12885
> URL: https://issues.apache.org/jira/browse/NIFI-12885
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 2.0.0-M2
>Reporter: crissaegrim
>Assignee: crissaegrim
>Priority: Major
> Attachments: .png
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> I think I found a breaking bug from this commit
> [https://github.com/apache/nifi/commit/250fe90b348fac515ea597c1985ca432ac7c3ac3#diff-ce496d3f0fc5a7e8a3c0431972f7069b4cf1af2e94f3a199f595ef195eb5ebfa]
> The below passes in 1.20.0 but fails in 2.0
> {code:java}
> @Test
> void testBasic() throws Exception {
> // setup
> final String schemaText = "{" +
> "\"type\" : \"record\"," +
> "\"name\" : \"TestRecord\"," +
> "\"namespace\" : \"org.apache.nifi\"," +
> "\"fields\" : [ {" +
> "\"name\" : \"my_datestamp_field\"," +
> "\"type\" : {" +
> "\"type\" : \"long\"," +
> "\"logicalType\" : \"timestamp-millis\"" +
> "}" +
> "} ]" +
> "}";
> final RecordSchema schemaParsed = AvroTypeUtil.createSchema(new
> Schema.Parser().parse(schemaText));
> final HashMap item = new HashMap<>();
> item.put("my_datestamp_field", "2022-01-01 10:00:00.000");
> // act
> final MapRecord record = new MapRecord(schemaParsed, item);
> final Date myDateStampField = record.getAsDate("my_datestamp_field",
> "-MM-dd HH:mm:ss.SSS");
> // assert
> // fails in 2.0; actual in 2.0.0-M2 is `164099520`
> assertEquals(164103120L, myDateStampField.getTime());
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12885) MapRecord.getAsDate timestamp breaking bug
[
https://issues.apache.org/jira/browse/NIFI-12885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17826049#comment-17826049
]
David Handermann commented on NIFI-12885:
-
I agree that defining the schema type and then parsing it again is not ideal.
Thanks for offering to put together a pull request!
> MapRecord.getAsDate timestamp breaking bug
> --
>
> Key: NIFI-12885
> URL: https://issues.apache.org/jira/browse/NIFI-12885
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 2.0.0-M2
>Reporter: crissaegrim
>Assignee: crissaegrim
>Priority: Major
> Attachments: .png
>
>
> I think I found a breaking bug from this commit
> [https://github.com/apache/nifi/commit/250fe90b348fac515ea597c1985ca432ac7c3ac3#diff-ce496d3f0fc5a7e8a3c0431972f7069b4cf1af2e94f3a199f595ef195eb5ebfa]
> The below passes in 1.20.0 but fails in 2.0
> {code:java}
> @Test
> void testBasic() throws Exception {
> // setup
> final String schemaText = "{" +
> "\"type\" : \"record\"," +
> "\"name\" : \"TestRecord\"," +
> "\"namespace\" : \"org.apache.nifi\"," +
> "\"fields\" : [ {" +
> "\"name\" : \"my_datestamp_field\"," +
> "\"type\" : {" +
> "\"type\" : \"long\"," +
> "\"logicalType\" : \"timestamp-millis\"" +
> "}" +
> "} ]" +
> "}";
> final RecordSchema schemaParsed = AvroTypeUtil.createSchema(new
> Schema.Parser().parse(schemaText));
> final HashMap item = new HashMap<>();
> item.put("my_datestamp_field", "2022-01-01 10:00:00.000");
> // act
> final MapRecord record = new MapRecord(schemaParsed, item);
> final Date myDateStampField = record.getAsDate("my_datestamp_field",
> "-MM-dd HH:mm:ss.SSS");
> // assert
> // fails in 2.0; actual in 2.0.0-M2 is `164099520`
> assertEquals(164103120L, myDateStampField.getTime());
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12885) MapRecord.getAsDate timestamp breaking bug
[
https://issues.apache.org/jira/browse/NIFI-12885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17826045#comment-17826045
]
crissaegrim commented on NIFI-12885:
Think it's anti-pattern to define a schema that expects a `DateTime`, only to
parse it yourself downstream. To unblock my project, yes, I will parse with
`java.time` as suggested.
I will assign this ticket to myself and add a new method and rename the
convenience method, as you have suggested above.
> MapRecord.getAsDate timestamp breaking bug
> --
>
> Key: NIFI-12885
> URL: https://issues.apache.org/jira/browse/NIFI-12885
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 2.0.0-M2
>Reporter: crissaegrim
>Priority: Major
> Attachments: .png
>
>
> I think I found a breaking bug from this commit
> [https://github.com/apache/nifi/commit/250fe90b348fac515ea597c1985ca432ac7c3ac3#diff-ce496d3f0fc5a7e8a3c0431972f7069b4cf1af2e94f3a199f595ef195eb5ebfa]
> The below passes in 1.20.0 but fails in 2.0
> {code:java}
> @Test
> void testBasic() throws Exception {
> // setup
> final String schemaText = "{" +
> "\"type\" : \"record\"," +
> "\"name\" : \"TestRecord\"," +
> "\"namespace\" : \"org.apache.nifi\"," +
> "\"fields\" : [ {" +
> "\"name\" : \"my_datestamp_field\"," +
> "\"type\" : {" +
> "\"type\" : \"long\"," +
> "\"logicalType\" : \"timestamp-millis\"" +
> "}" +
> "} ]" +
> "}";
> final RecordSchema schemaParsed = AvroTypeUtil.createSchema(new
> Schema.Parser().parse(schemaText));
> final HashMap item = new HashMap<>();
> item.put("my_datestamp_field", "2022-01-01 10:00:00.000");
> // act
> final MapRecord record = new MapRecord(schemaParsed, item);
> final Date myDateStampField = record.getAsDate("my_datestamp_field",
> "-MM-dd HH:mm:ss.SSS");
> // assert
> // fails in 2.0; actual in 2.0.0-M2 is `164099520`
> assertEquals(164103120L, myDateStampField.getTime());
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12885) MapRecord.getAsDate timestamp breaking bug
[
https://issues.apache.org/jira/browse/NIFI-12885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17825852#comment-17825852
]
David Handermann commented on NIFI-12885:
-
Thanks for providing the example [~crissaegrim], that is helpful.
Based on the example, if the {{history_modified_date}} is a string or a number,
then another get method could be used, and the isRecordClean() method could
perform the conversion to a timestamp using the applicable java.time class.
That would avoid having to wait for changes to MapRecord, and also provide
compatibility with existing records.
> MapRecord.getAsDate timestamp breaking bug
> --
>
> Key: NIFI-12885
> URL: https://issues.apache.org/jira/browse/NIFI-12885
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 2.0.0-M2
>Reporter: crissaegrim
>Priority: Major
> Attachments: .png
>
>
> I think I found a breaking bug from this commit
> [https://github.com/apache/nifi/commit/250fe90b348fac515ea597c1985ca432ac7c3ac3#diff-ce496d3f0fc5a7e8a3c0431972f7069b4cf1af2e94f3a199f595ef195eb5ebfa]
> The below passes in 1.20.0 but fails in 2.0
> {code:java}
> @Test
> void testBasic() throws Exception {
> // setup
> final String schemaText = "{" +
> "\"type\" : \"record\"," +
> "\"name\" : \"TestRecord\"," +
> "\"namespace\" : \"org.apache.nifi\"," +
> "\"fields\" : [ {" +
> "\"name\" : \"my_datestamp_field\"," +
> "\"type\" : {" +
> "\"type\" : \"long\"," +
> "\"logicalType\" : \"timestamp-millis\"" +
> "}" +
> "} ]" +
> "}";
> final RecordSchema schemaParsed = AvroTypeUtil.createSchema(new
> Schema.Parser().parse(schemaText));
> final HashMap item = new HashMap<>();
> item.put("my_datestamp_field", "2022-01-01 10:00:00.000");
> // act
> final MapRecord record = new MapRecord(schemaParsed, item);
> final Date myDateStampField = record.getAsDate("my_datestamp_field",
> "-MM-dd HH:mm:ss.SSS");
> // assert
> // fails in 2.0; actual in 2.0.0-M2 is `164099520`
> assertEquals(164103120L, myDateStampField.getTime());
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12885) MapRecord.getAsDate timestamp breaking bug
[
https://issues.apache.org/jira/browse/NIFI-12885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17825680#comment-17825680
]
crissaegrim commented on NIFI-12885:
> Can you provide any additional details on the source of the data for the
> MapRecord?
The custom processor I built expects a an input that can be read by
`RecordReader`. `RecordReader` is schema-aware (does not infer) and depends on
some `DateTime` fields (e.g. -MM-dd HH:mm:ss.SSS`). It reads some records
and compares some datetime fields, where time is important. Here's a snippet.
The `Record` objects simply come from the `RecordReader.nextRecord()` calls.
Then I call `Record.getAsDate("field_name", "format")`.
!.png!
> MapRecord.getAsDate timestamp breaking bug
> --
>
> Key: NIFI-12885
> URL: https://issues.apache.org/jira/browse/NIFI-12885
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 2.0.0-M2
>Reporter: crissaegrim
>Priority: Major
> Attachments: .png
>
>
> I think I found a breaking bug from this commit
> [https://github.com/apache/nifi/commit/250fe90b348fac515ea597c1985ca432ac7c3ac3#diff-ce496d3f0fc5a7e8a3c0431972f7069b4cf1af2e94f3a199f595ef195eb5ebfa]
> The below passes in 1.20.0 but fails in 2.0
> {code:java}
> @Test
> void testBasic() throws Exception {
> // setup
> final String schemaText = "{" +
> "\"type\" : \"record\"," +
> "\"name\" : \"TestRecord\"," +
> "\"namespace\" : \"org.apache.nifi\"," +
> "\"fields\" : [ {" +
> "\"name\" : \"my_datestamp_field\"," +
> "\"type\" : {" +
> "\"type\" : \"long\"," +
> "\"logicalType\" : \"timestamp-millis\"" +
> "}" +
> "} ]" +
> "}";
> final RecordSchema schemaParsed = AvroTypeUtil.createSchema(new
> Schema.Parser().parse(schemaText));
> final HashMap item = new HashMap<>();
> item.put("my_datestamp_field", "2022-01-01 10:00:00.000");
> // act
> final MapRecord record = new MapRecord(schemaParsed, item);
> final Date myDateStampField = record.getAsDate("my_datestamp_field",
> "-MM-dd HH:mm:ss.SSS");
> // assert
> // fails in 2.0; actual in 2.0.0-M2 is `164099520`
> assertEquals(164103120L, myDateStampField.getTime());
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12885) MapRecord.getAsDate timestamp breaking bug
[
https://issues.apache.org/jira/browse/NIFI-12885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17825673#comment-17825673
]
David Handermann commented on NIFI-12885:
-
Thanks for the additional background.
The {{getAsDate()}} method is more of a convenience function that performs
optional conversion from a string using the provided format. If the source of
the values is a JDBC ResultSet, then {{getValue()}} can be used to return a
java.sql.Timestamp as shown in one of the current test classes. If the source
value is a string, then the conversion could be performed after retrieving the
value from the MapRecord.
Can you provide any additional details on the source of the data for the
MapRecord?
Going forward, it sounds like it could be useful to add a new method, and
change {{getAsDate()}} to {{getAsLocalDate()}} to clarify the behavior. On the
other hand, removing the convenience function would then require calling code
to perform the conversion as needed. This might be better, as date and time
conversion can vary significantly from one use case to another.
> MapRecord.getAsDate timestamp breaking bug
> --
>
> Key: NIFI-12885
> URL: https://issues.apache.org/jira/browse/NIFI-12885
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 2.0.0-M2
>Reporter: crissaegrim
>Priority: Major
>
> I think I found a breaking bug from this commit
> [https://github.com/apache/nifi/commit/250fe90b348fac515ea597c1985ca432ac7c3ac3#diff-ce496d3f0fc5a7e8a3c0431972f7069b4cf1af2e94f3a199f595ef195eb5ebfa]
> The below passes in 1.20.0 but fails in 2.0
> {code:java}
> @Test
> void testBasic() throws Exception {
> // setup
> final String schemaText = "{" +
> "\"type\" : \"record\"," +
> "\"name\" : \"TestRecord\"," +
> "\"namespace\" : \"org.apache.nifi\"," +
> "\"fields\" : [ {" +
> "\"name\" : \"my_datestamp_field\"," +
> "\"type\" : {" +
> "\"type\" : \"long\"," +
> "\"logicalType\" : \"timestamp-millis\"" +
> "}" +
> "} ]" +
> "}";
> final RecordSchema schemaParsed = AvroTypeUtil.createSchema(new
> Schema.Parser().parse(schemaText));
> final HashMap item = new HashMap<>();
> item.put("my_datestamp_field", "2022-01-01 10:00:00.000");
> // act
> final MapRecord record = new MapRecord(schemaParsed, item);
> final Date myDateStampField = record.getAsDate("my_datestamp_field",
> "-MM-dd HH:mm:ss.SSS");
> // assert
> // fails in 2.0; actual in 2.0.0-M2 is `164099520`
> assertEquals(164103120L, myDateStampField.getTime());
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12885) MapRecord.getAsDate timestamp breaking bug
[
https://issues.apache.org/jira/browse/NIFI-12885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17825455#comment-17825455
]
crissaegrim commented on NIFI-12885:
Hi. Thanks for the reply.
> Outside of this test class, did you observe runtime behavior changes in any
> Processors?
Only in a custom processor I'm writing. `getAsDate` used to return the time
component too, so my tests there were breaking when we switch versions.
And agreed, on adding a method. I looked through nifi code today. I don't
think any tests are testing for the time component. I think that's the
shortcoming here–`MapRecord` stopped providing a time component, which I think
could break a lot of folks' data when materialized as `MapRecord` and
`getAsDate` is called and downstream expects a timestamp.
> MapRecord.getAsDate timestamp breaking bug
> --
>
> Key: NIFI-12885
> URL: https://issues.apache.org/jira/browse/NIFI-12885
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 2.0.0-M2
>Reporter: crissaegrim
>Priority: Major
>
> I think I found a breaking bug from this commit
> [https://github.com/apache/nifi/commit/250fe90b348fac515ea597c1985ca432ac7c3ac3#diff-ce496d3f0fc5a7e8a3c0431972f7069b4cf1af2e94f3a199f595ef195eb5ebfa]
> The below passes in 1.20.0 but fails in 2.0
> {code:java}
> @Test
> void testBasic() throws Exception {
> // setup
> final String schemaText = "{" +
> "\"type\" : \"record\"," +
> "\"name\" : \"TestRecord\"," +
> "\"namespace\" : \"org.apache.nifi\"," +
> "\"fields\" : [ {" +
> "\"name\" : \"my_datestamp_field\"," +
> "\"type\" : {" +
> "\"type\" : \"long\"," +
> "\"logicalType\" : \"timestamp-millis\"" +
> "}" +
> "} ]" +
> "}";
> final RecordSchema schemaParsed = AvroTypeUtil.createSchema(new
> Schema.Parser().parse(schemaText));
> final HashMap item = new HashMap<>();
> item.put("my_datestamp_field", "2022-01-01 10:00:00.000");
> // act
> final MapRecord record = new MapRecord(schemaParsed, item);
> final Date myDateStampField = record.getAsDate("my_datestamp_field",
> "-MM-dd HH:mm:ss.SSS");
> // assert
> // fails in 2.0; actual in 2.0.0-M2 is `164099520`
> assertEquals(164103120L, myDateStampField.getTime());
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12885) MapRecord.getAsDate timestamp breaking bug
[
https://issues.apache.org/jira/browse/NIFI-12885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17825451#comment-17825451
]
David Handermann commented on NIFI-12885:
-
Thanks for highlighting this change [~crissaegrim].
Part of the challenge is that {{getAsDate()}} is somewhat ambigous based on
previous usage. Test classes including TestQueryRecord and
ResultSetRecordSetTest expect the results of getAsDate() to return what is
effectively a LocalDate, containing only year, month, and day.
The changes referenced in NIFI-9458 make that behavior explicit by return a
java.sql.Date object containing only the year, month, and day information. This
different than java.sql.Timestamp, which contains resolution down to the
millisecond.
These existing unit tests pass based on this expected behavior, and the
references are otherwise limited. Outside of this test class, did you observe
runtime behavior changes in any Processors?
There are several possible ways forward, one of which may be to change the
method signature to return a java.time.LocalDate, making the behavior clear.
Another option could include adding a method such as getAsOffsetDateTime(),
which could return an instance of java.time.OffsetDateTime containing
additional precision.
> MapRecord.getAsDate timestamp breaking bug
> --
>
> Key: NIFI-12885
> URL: https://issues.apache.org/jira/browse/NIFI-12885
> Project: Apache NiFi
> Issue Type: New Feature
>Affects Versions: 2.0.0-M2
>Reporter: crissaegrim
>Priority: Critical
>
> I think I found a breaking bug from this commit
> [https://github.com/apache/nifi/commit/250fe90b348fac515ea597c1985ca432ac7c3ac3#diff-ce496d3f0fc5a7e8a3c0431972f7069b4cf1af2e94f3a199f595ef195eb5ebfa]
> The below passes in 1.20.0 but fails in 2.0
> {code:java}
> @Test
> void testBasic() throws Exception {
> // setup
> final String schemaText = "{" +
> "\"type\" : \"record\"," +
> "\"name\" : \"TestRecord\"," +
> "\"namespace\" : \"org.apache.nifi\"," +
> "\"fields\" : [ {" +
> "\"name\" : \"my_datestamp_field\"," +
> "\"type\" : {" +
> "\"type\" : \"long\"," +
> "\"logicalType\" : \"timestamp-millis\"" +
> "}" +
> "} ]" +
> "}";
> final RecordSchema schemaParsed = AvroTypeUtil.createSchema(new
> Schema.Parser().parse(schemaText));
> final HashMap item = new HashMap<>();
> item.put("my_datestamp_field", "2022-01-01 10:00:00.000");
> // act
> final MapRecord record = new MapRecord(schemaParsed, item);
> final Date myDateStampField = record.getAsDate("my_datestamp_field",
> "-MM-dd HH:mm:ss.SSS");
> // assert
> // fails in 2.0; actual in 2.0.0-M2 is `164099520`
> assertEquals(164103120L, myDateStampField.getTime());
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
