[jira] [Assigned] (NIFI-4930) Nar-Dependency-Version - timestamped snapshot version problem
[ https://issues.apache.org/jira/browse/NIFI-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Derek Straka reassigned NIFI-4930: -- Assignee: Bryan Bende (was: Derek Straka) > Nar-Dependency-Version - timestamped snapshot version problem > - > > Key: NIFI-4930 > URL: https://issues.apache.org/jira/browse/NIFI-4930 > Project: Apache NiFi > Issue Type: Bug > Components: Tools and Build >Affects Versions: 1.2.0 >Reporter: Arne Degenring >Assignee: Bryan Bende >Priority: Major > Labels: Nar-Maven-Plugin > Original Estimate: 5m > Remaining Estimate: 5m > > The NAR Maven Plugin 1.2.0 is using artifact.getVersion() to determine the > Nar-Dependency-Version, however this version number can be a timestamped > snapshot version number if the dependency was downloaded from a remote repo. > The Nar-Dependency-Version can then differ from the declared Nar-Version in > the other JAR. > Suggested Fix in NarMojo.java: use artifact.getBaseVersion() instead of > artifact.getVersion(): > Replace line 705: > narDependency = new NarDependency(artifact.getGroupId(), > artifact.getArtifactId(), artifact.getVersion()); > with: > narDependency = new NarDependency(artifact.getGroupId(), > artifact.getArtifactId(), artifact.getBaseVersion()); > > -- > Long explanation, from > [https://lists.apache.org/thread.html/ce44faa7b1a54aac417fca06200712de629c3bc2ed692813c0a10a73@%3Cusers.nifi.apache.org%3E] > We have seen a strange problem on NiFi 1.4.0 where custom processors could > suddenly not be started, because of incompatibility with custom services: > 2018-03-02 13:40:35,490 ERROR [main] o.apache.nifi.controller.FlowController > Unable to start ...[id=5d57d39a-015c-1000--d654d90b] due to > java.lang.IllegalStateException: Processor ... is not in a valid state due to > ... is invalid because ... - 1.4-SNAPSHOT from ... is not compatible with > ... - 1.4-SNAPSHOT ...] > It seems that the root cause was related to: > 2018-03-02 13:39:55,086 WARN [main] org.apache.nifi.nar.NarClassLoaders While > loading ...:...:1.5-SNAPSHOT' unable to locate exact NAR dependency > '...:...:1.4-20180302.33-16'. Only found one possible match > ...::1.4-SNAPSHOT'. Continuing... > It turned out that our various custom NARs were built on different built > agents. > Maven has the (ugly) behaviour that when a snapshot version is present in the > local repository after local build, the version number will be e.g. > 1.4-SNAPSHOT. However if it is downloaded from a remote repository, the > version number includes a timestamp such as 1.4-20180302.33-16. > So the manifest of the depending NARs contained: > Nar-Dependency-Version: 1.4-20180302.33-16 > while the other NAR file declared: > Nar-Version: 1.4-SNAPSHOT > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-4930) Nar-Dependency-Version - timestamped snapshot version problem
[ https://issues.apache.org/jira/browse/NIFI-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Derek Straka updated NIFI-4930: --- Status: Patch Available (was: Open) > Nar-Dependency-Version - timestamped snapshot version problem > - > > Key: NIFI-4930 > URL: https://issues.apache.org/jira/browse/NIFI-4930 > Project: Apache NiFi > Issue Type: Bug > Components: Tools and Build >Affects Versions: 1.2.0 >Reporter: Arne Degenring >Assignee: Derek Straka >Priority: Major > Labels: Nar-Maven-Plugin > Original Estimate: 5m > Remaining Estimate: 5m > > The NAR Maven Plugin 1.2.0 is using artifact.getVersion() to determine the > Nar-Dependency-Version, however this version number can be a timestamped > snapshot version number if the dependency was downloaded from a remote repo. > The Nar-Dependency-Version can then differ from the declared Nar-Version in > the other JAR. > Suggested Fix in NarMojo.java: use artifact.getBaseVersion() instead of > artifact.getVersion(): > Replace line 705: > narDependency = new NarDependency(artifact.getGroupId(), > artifact.getArtifactId(), artifact.getVersion()); > with: > narDependency = new NarDependency(artifact.getGroupId(), > artifact.getArtifactId(), artifact.getBaseVersion()); > > -- > Long explanation, from > [https://lists.apache.org/thread.html/ce44faa7b1a54aac417fca06200712de629c3bc2ed692813c0a10a73@%3Cusers.nifi.apache.org%3E] > We have seen a strange problem on NiFi 1.4.0 where custom processors could > suddenly not be started, because of incompatibility with custom services: > 2018-03-02 13:40:35,490 ERROR [main] o.apache.nifi.controller.FlowController > Unable to start ...[id=5d57d39a-015c-1000--d654d90b] due to > java.lang.IllegalStateException: Processor ... is not in a valid state due to > ... is invalid because ... - 1.4-SNAPSHOT from ... is not compatible with > ... - 1.4-SNAPSHOT ...] > It seems that the root cause was related to: > 2018-03-02 13:39:55,086 WARN [main] org.apache.nifi.nar.NarClassLoaders While > loading ...:...:1.5-SNAPSHOT' unable to locate exact NAR dependency > '...:...:1.4-20180302.33-16'. Only found one possible match > ...::1.4-SNAPSHOT'. Continuing... > It turned out that our various custom NARs were built on different built > agents. > Maven has the (ugly) behaviour that when a snapshot version is present in the > local repository after local build, the version number will be e.g. > 1.4-SNAPSHOT. However if it is downloaded from a remote repository, the > version number includes a timestamp such as 1.4-20180302.33-16. > So the manifest of the depending NARs contained: > Nar-Dependency-Version: 1.4-20180302.33-16 > while the other NAR file declared: > Nar-Version: 1.4-SNAPSHOT > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4930) Nar-Dependency-Version - timestamped snapshot version problem
[ https://issues.apache.org/jira/browse/NIFI-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16484064#comment-16484064 ] Derek Straka commented on NIFI-4930: [~bende] - Yes. I'm going to submit a PR some time today. > Nar-Dependency-Version - timestamped snapshot version problem > - > > Key: NIFI-4930 > URL: https://issues.apache.org/jira/browse/NIFI-4930 > Project: Apache NiFi > Issue Type: Bug > Components: Tools and Build >Affects Versions: 1.2.0 >Reporter: Arne Degenring >Assignee: Derek Straka >Priority: Major > Labels: Nar-Maven-Plugin > Original Estimate: 5m > Remaining Estimate: 5m > > The NAR Maven Plugin 1.2.0 is using artifact.getVersion() to determine the > Nar-Dependency-Version, however this version number can be a timestamped > snapshot version number if the dependency was downloaded from a remote repo. > The Nar-Dependency-Version can then differ from the declared Nar-Version in > the other JAR. > Suggested Fix in NarMojo.java: use artifact.getBaseVersion() instead of > artifact.getVersion(): > Replace line 705: > narDependency = new NarDependency(artifact.getGroupId(), > artifact.getArtifactId(), artifact.getVersion()); > with: > narDependency = new NarDependency(artifact.getGroupId(), > artifact.getArtifactId(), artifact.getBaseVersion()); > > -- > Long explanation, from > [https://lists.apache.org/thread.html/ce44faa7b1a54aac417fca06200712de629c3bc2ed692813c0a10a73@%3Cusers.nifi.apache.org%3E] > We have seen a strange problem on NiFi 1.4.0 where custom processors could > suddenly not be started, because of incompatibility with custom services: > 2018-03-02 13:40:35,490 ERROR [main] o.apache.nifi.controller.FlowController > Unable to start ...[id=5d57d39a-015c-1000--d654d90b] due to > java.lang.IllegalStateException: Processor ... is not in a valid state due to > ... is invalid because ... - 1.4-SNAPSHOT from ... is not compatible with > ... - 1.4-SNAPSHOT ...] > It seems that the root cause was related to: > 2018-03-02 13:39:55,086 WARN [main] org.apache.nifi.nar.NarClassLoaders While > loading ...:...:1.5-SNAPSHOT' unable to locate exact NAR dependency > '...:...:1.4-20180302.33-16'. Only found one possible match > ...::1.4-SNAPSHOT'. Continuing... > It turned out that our various custom NARs were built on different built > agents. > Maven has the (ugly) behaviour that when a snapshot version is present in the > local repository after local build, the version number will be e.g. > 1.4-SNAPSHOT. However if it is downloaded from a remote repository, the > version number includes a timestamp such as 1.4-20180302.33-16. > So the manifest of the depending NARs contained: > Nar-Dependency-Version: 1.4-20180302.33-16 > while the other NAR file declared: > Nar-Version: 1.4-SNAPSHOT > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (NIFI-4930) Nar-Dependency-Version - timestamped snapshot version problem
[ https://issues.apache.org/jira/browse/NIFI-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Derek Straka reassigned NIFI-4930: -- Assignee: Derek Straka > Nar-Dependency-Version - timestamped snapshot version problem > - > > Key: NIFI-4930 > URL: https://issues.apache.org/jira/browse/NIFI-4930 > Project: Apache NiFi > Issue Type: Bug > Components: Tools and Build >Affects Versions: 1.2.0 >Reporter: Arne Degenring >Assignee: Derek Straka >Priority: Major > Labels: Nar-Maven-Plugin > Original Estimate: 5m > Remaining Estimate: 5m > > The NAR Maven Plugin 1.2.0 is using artifact.getVersion() to determine the > Nar-Dependency-Version, however this version number can be a timestamped > snapshot version number if the dependency was downloaded from a remote repo. > The Nar-Dependency-Version can then differ from the declared Nar-Version in > the other JAR. > Suggested Fix in NarMojo.java: use artifact.getBaseVersion() instead of > artifact.getVersion(): > Replace line 705: > narDependency = new NarDependency(artifact.getGroupId(), > artifact.getArtifactId(), artifact.getVersion()); > with: > narDependency = new NarDependency(artifact.getGroupId(), > artifact.getArtifactId(), artifact.getBaseVersion()); > > -- > Long explanation, from > [https://lists.apache.org/thread.html/ce44faa7b1a54aac417fca06200712de629c3bc2ed692813c0a10a73@%3Cusers.nifi.apache.org%3E] > We have seen a strange problem on NiFi 1.4.0 where custom processors could > suddenly not be started, because of incompatibility with custom services: > 2018-03-02 13:40:35,490 ERROR [main] o.apache.nifi.controller.FlowController > Unable to start ...[id=5d57d39a-015c-1000--d654d90b] due to > java.lang.IllegalStateException: Processor ... is not in a valid state due to > ... is invalid because ... - 1.4-SNAPSHOT from ... is not compatible with > ... - 1.4-SNAPSHOT ...] > It seems that the root cause was related to: > 2018-03-02 13:39:55,086 WARN [main] org.apache.nifi.nar.NarClassLoaders While > loading ...:...:1.5-SNAPSHOT' unable to locate exact NAR dependency > '...:...:1.4-20180302.33-16'. Only found one possible match > ...::1.4-SNAPSHOT'. Continuing... > It turned out that our various custom NARs were built on different built > agents. > Maven has the (ugly) behaviour that when a snapshot version is present in the > local repository after local build, the version number will be e.g. > 1.4-SNAPSHOT. However if it is downloaded from a remote repository, the > version number includes a timestamp such as 1.4-20180302.33-16. > So the manifest of the depending NARs contained: > Nar-Dependency-Version: 1.4-20180302.33-16 > while the other NAR file declared: > Nar-Version: 1.4-SNAPSHOT > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (NIFI-4983) JsonTreeReader and JsonPathReader should utilize specified date/time/timestamp format when used from ValidateRecord
[ https://issues.apache.org/jira/browse/NIFI-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Derek Straka reassigned NIFI-4983: -- Assignee: Derek Straka > JsonTreeReader and JsonPathReader should utilize specified > date/time/timestamp format when used from ValidateRecord > --- > > Key: NIFI-4983 > URL: https://issues.apache.org/jira/browse/NIFI-4983 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Koji Kawamura >Assignee: Derek Straka >Priority: Major > Attachments: NIFI-4983.xml > > > Similar to CSVRecordReader, JSON readers should utilize specified date format > for Record validation. > JsonPathRowRecordReader.convert(final Object value, final DataType dataType) > is called when coerceTypes is false. But the method currently does not > convert String values for Date/Time/Timestamp field types at all. The method > should try convert like CSVRecordReader does. Without doing so, > ValidateRecord routes any records validated against a record schema having > Date/Time/Timestamp fields to 'invalid' relationship. > https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/json/JsonPathRowRecordReader.java#L138 > A NiFi template file to illustrate the issue is attached. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-4951) convertToAvroObject does not utilize AvroTypeUtil when trying to convert Date, Time, or Timestamps
[ https://issues.apache.org/jira/browse/NIFI-4951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Derek Straka updated NIFI-4951: --- Assignee: (was: Derek Straka) Status: Patch Available (was: In Progress) > convertToAvroObject does not utilize AvroTypeUtil when trying to convert > Date, Time, or Timestamps > -- > > Key: NIFI-4951 > URL: https://issues.apache.org/jira/browse/NIFI-4951 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Derek Straka >Priority: Major > > Currently, AvroTypeUtil::convertToAvroObject assumes that the logical types > have already been converted into their internal representation. While this > may sometimes be the case, it would be useful to allow String->Logical type > conversion. The feature can be added by using the appropriate > DataTypeUtils::to method rather than blindly attempting to use > the underlying type. The DataTypeUtils already performs the integral type > parsing, so the existing functionality remains intact. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-4951) convertToAvroObject does not utilize AvroTypeUtil when trying to convert Date, Time, or Timestamps
Derek Straka created NIFI-4951: -- Summary: convertToAvroObject does not utilize AvroTypeUtil when trying to convert Date, Time, or Timestamps Key: NIFI-4951 URL: https://issues.apache.org/jira/browse/NIFI-4951 Project: Apache NiFi Issue Type: Improvement Reporter: Derek Straka Assignee: Derek Straka Currently, AvroTypeUtil::convertToAvroObject assumes that the logical types have already been converted into their internal representation. While this may sometimes be the case, it would be useful to allow String->Logical type conversion. The feature can be added by using the appropriate DataTypeUtils::to method rather than blindly attempting to use the underlying type. The DataTypeUtils already performs the integral type parsing, so the existing functionality remains intact. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-4912) Update jackson version to latest stable
[ https://issues.apache.org/jira/browse/NIFI-4912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Derek Straka updated NIFI-4912: --- Status: Patch Available (was: Open) > Update jackson version to latest stable > --- > > Key: NIFI-4912 > URL: https://issues.apache.org/jira/browse/NIFI-4912 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.5.0 >Reporter: Derek Straka >Assignee: Derek Straka >Priority: Major > Fix For: 1.6.0 > > > The current jackson version is out of date and contains several CVEs as well > as outstanding bugs. Update to the latest stable version which at the time > of writing is 2.9.4 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-4912) Update jackson version to latest stable
Derek Straka created NIFI-4912: -- Summary: Update jackson version to latest stable Key: NIFI-4912 URL: https://issues.apache.org/jira/browse/NIFI-4912 Project: Apache NiFi Issue Type: Improvement Components: Core Framework Affects Versions: 1.5.0 Reporter: Derek Straka Assignee: Derek Straka Fix For: 1.6.0 The current jackson version is out of date and contains several CVEs as well as outstanding bugs. Update to the latest stable version which at the time of writing is 2.9.4 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (NIFI-4880) Add the ability to utilize aliases in Avro to Avro record conversion
[ https://issues.apache.org/jira/browse/NIFI-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Derek Straka reassigned NIFI-4880: -- Assignee: Derek Straka > Add the ability to utilize aliases in Avro to Avro record conversion > > > Key: NIFI-4880 > URL: https://issues.apache.org/jira/browse/NIFI-4880 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.5.0 >Reporter: Derek Straka >Assignee: Derek Straka >Priority: Minor > > Currently, the Avro to Avro conversions will ignore fields that are not > mapped verbatim. In avro schemas, it is possible for fields to be aliased to > one another. It would be useful if a 1:1 mapping was not available, the > aliases for the field were searched to locate a prospective value and then > add the default. > > The functionality can be accomplished by adding some logic to > AvroTypeUtil::createAvroRecord. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4882) CSVRecordReader should utilize specified date/time/timestamp format at its convertSimpleIfPossible method
[
https://issues.apache.org/jira/browse/NIFI-4882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365791#comment-16365791
]
Derek Straka commented on NIFI-4882:
I'll make a patchset for this with the proposed refactor.
> CSVRecordReader should utilize specified date/time/timestamp format at its
> convertSimpleIfPossible method
> -
>
> Key: NIFI-4882
> URL: https://issues.apache.org/jira/browse/NIFI-4882
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Koji Kawamura
>Priority: Major
>
> CSVRecordReader.convertSimpleIfPossible method is used by ValidateRecord. The
> method does not coerce values to the target schema field type if the raw
> string representation in the input CSV file is not compatible.
> The type compatibility check is implemented as follows. But it does not use
> user specified date/time/timestamp format:
> {code}
> // This will return 'false' for input '01/01/1900' when user
> specified custom format 'MM/dd/'
> if (DataTypeUtils.isCompatibleDataType(trimmed, dataType)) {
> // The LAZY_DATE_FORMAT should be used to check
> compatibility, too.
> return DataTypeUtils.convertType(trimmed, dataType,
> LAZY_DATE_FORMAT, LAZY_TIME_FORMAT, LAZY_TIMESTAMP_FORMAT, fieldName);
> } else {
> return value;
> }
> {code}
> If input date strings have different format than the default format
> '-MM-dd', then ValidateRecord processor can not validate input records.
> JacksonCSVRecordReader has the identical methods with CSVRecordReader. Those
> classes should have an abstract class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-4880) Add the ability to utilize aliases in Avro to Avro record conversion
[ https://issues.apache.org/jira/browse/NIFI-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364807#comment-16364807 ] Derek Straka commented on NIFI-4880: I'll be creating a changeset for this functionality and adding some unit tests. > Add the ability to utilize aliases in Avro to Avro record conversion > > > Key: NIFI-4880 > URL: https://issues.apache.org/jira/browse/NIFI-4880 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.5.0 >Reporter: Derek Straka >Priority: Minor > > Currently, the Avro to Avro conversions will ignore fields that are not > mapped verbatim. In avro schemas, it is possible for fields to be aliased to > one another. It would be useful if a 1:1 mapping was not available, the > aliases for the field were searched to locate a prospective value and then > add the default. > > The functionality can be accomplished by adding some logic to > AvroTypeUtil::createAvroRecord. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-4880) Add the ability to utilize aliases in Avro to Avro record conversion
Derek Straka created NIFI-4880: -- Summary: Add the ability to utilize aliases in Avro to Avro record conversion Key: NIFI-4880 URL: https://issues.apache.org/jira/browse/NIFI-4880 Project: Apache NiFi Issue Type: Improvement Affects Versions: 1.5.0 Reporter: Derek Straka Currently, the Avro to Avro conversions will ignore fields that are not mapped verbatim. In avro schemas, it is possible for fields to be aliased to one another. It would be useful if a 1:1 mapping was not available, the aliases for the field were searched to locate a prospective value and then add the default. The functionality can be accomplished by adding some logic to AvroTypeUtil::createAvroRecord. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
