[jira] [Commented] (NIFI-13191) Deprecate InvokeAWSGatewayApi Processor
[ https://issues.apache.org/jira/browse/NIFI-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845126#comment-17845126 ] Otto Fowler commented on NIFI-13191: I don't see or haven't seen any user issues with this since introduction, so I'm not sure anyone is using it beyond the initial users I wrote it for, and I doubt they are still using it as I think that product is no longer in production. It was still working in the 1x last I tried it, so there are options for anyone silently using it. Deprecation is the correct move. > Deprecate InvokeAWSGatewayApi Processor > --- > > Key: NIFI-13191 > URL: https://issues.apache.org/jira/browse/NIFI-13191 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joe Gresock >Assignee: Joe Gresock >Priority: Minor > Fix For: 1.27.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Similar to the other AWS client library upgrade issues, this targets the > InvokeAWSGatewayApi processor. > Note: After discussing with [~exceptionfactory] and [~otto], we agree that > this processor can be deprecated, and removed in NiFi 2.0. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12178) Add integration-tests and docker-tests GithHub Action Workflows
[
https://issues.apache.org/jira/browse/NIFI-12178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773068#comment-17773068
]
Otto Fowler commented on NIFI-12178:
I think this stuff is great. The only thing I will add for consideration is
that Apache's GitHub resources are not infinite, ( join the build list for the
repetitive discussions on these limits as some projects exhaust them for all
the others ) and we should be cognizant of that.
> Add integration-tests and docker-tests GithHub Action Workflows
> ---
>
> Key: NIFI-12178
> URL: https://issues.apache.org/jira/browse/NIFI-12178
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Major
> Fix For: 2.latest
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> There are a lot of {{integration-tests}} and {{docker}} builds in the NiFi
> repository that aren't being regularly executed and verified.
> GitHub Action Workflows should be added (similar to the existing
> {{system-tests}} Workflow) that regularly exercise these tests, allowing the
> community to better monitor the health of NiFi components.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12055) ListenSyslog: Parse Messages didn't recognize some of the syslogevents
[
https://issues.apache.org/jira/browse/NIFI-12055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17766193#comment-17766193
]
Otto Fowler commented on NIFI-12055:
I opened an issue of my upstream library ( that nifi uses an old version of for
5424 processors ): https://github.com/palindromicity/simple-syslog/issues/29
> ListenSyslog: Parse Messages didn't recognize some of the syslogevents
> --
>
> Key: NIFI-12055
> URL: https://issues.apache.org/jira/browse/NIFI-12055
> Project: Apache NiFi
> Issue Type: Task
> Components: Extensions
>Affects Versions: 1.23.2
> Environment: Debian VM
>Reporter: Dirk Mader
>Priority: Major
> Fix For: 1.23.2
>
> Attachments: dump_syslog.tcpd
>
>
> I tested with an OpenBSD Current to send syslog to ListenSyslog.
> But most of the Events are running into "invalid".
> In the attached tcpdump are 4 Events: 3 of them will marked as *invalid* by
> "Parsing Messages" 1 of them is marked as as *success*
> The success message is {{"the last message repeated 2 times"}}
> The only change in properties was the UDP Port to 5140
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12055) ListenSyslog: Parse Messages didn't recognize some of the syslogevents
[
https://issues.apache.org/jira/browse/NIFI-12055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17766007#comment-17766007
]
Otto Fowler commented on NIFI-12055:
I am not sure what you mean. I see the <38> in all the messages. I wrote a
test using the bytes from Wireshark and saw the one succeed and rest fail as
you did.
I don't think the failures have anything to do with the facility etc
Sometimes things don't include the hostname in the syslog by default and you
have enable it by configuration. If you can figure out how to do this on your
source side it will be yummy.
You can write a custom GROK to parse these messages without the hostname part
for sure though
> ListenSyslog: Parse Messages didn't recognize some of the syslogevents
> --
>
> Key: NIFI-12055
> URL: https://issues.apache.org/jira/browse/NIFI-12055
> Project: Apache NiFi
> Issue Type: Task
> Components: Extensions
>Affects Versions: 1.23.2
> Environment: Debian VM
>Reporter: Dirk Mader
>Priority: Major
> Fix For: 1.23.2
>
> Attachments: dump_syslog.tcpd
>
>
> I tested with an OpenBSD Current to send syslog to ListenSyslog.
> But most of the Events are running into "invalid".
> In the attached tcpdump are 4 Events: 3 of them will marked as *invalid* by
> "Parsing Messages" 1 of them is marked as as *success*
> The success message is {{"the last message repeated 2 times"}}
> The only change in properties was the UDP Port to 5140
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12055) ListenSyslog: Parse Messages didn't recognize some of the syslogevents
[
https://issues.apache.org/jira/browse/NIFI-12055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17765140#comment-17765140
]
Otto Fowler commented on NIFI-12055:
Our regex for 3164 syslog expects the HOSTNAME field per the spec. None of
these provide that, but in the case that works, 'last' parses as a host name
> ListenSyslog: Parse Messages didn't recognize some of the syslogevents
> --
>
> Key: NIFI-12055
> URL: https://issues.apache.org/jira/browse/NIFI-12055
> Project: Apache NiFi
> Issue Type: Task
> Components: Extensions
>Affects Versions: 1.23.2
> Environment: Debian VM
>Reporter: Dirk Mader
>Priority: Major
> Fix For: 1.23.2
>
> Attachments: dump_syslog.tcpd
>
>
> I tested with an OpenBSD Current to send syslog to ListenSyslog.
> But most of the Events are running into "invalid".
> In the attached tcpdump are 4 Events: 3 of them will marked as *invalid* by
> "Parsing Messages" 1 of them is marked as as *success*
> The success message is {{"the last message repeated 2 times"}}
> The only change in properties was the UDP Port to 5140
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-11833) Remove deprecated classes in nifi-commons
[ https://issues.apache.org/jira/browse/NIFI-11833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otto Fowler updated NIFI-11833: --- Resolution: Done Status: Resolved (was: Patch Available) > Remove deprecated classes in nifi-commons > - > > Key: NIFI-11833 > URL: https://issues.apache.org/jira/browse/NIFI-11833 > Project: Apache NiFi > Issue Type: Task > Components: Core Framework >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Fix For: 2.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Some classes are deprecated in 1.x and can be removed in 2.x: > * > ./nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/DataOutputStream.java > * > ./nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/BufferedInputStream.java > * > ./nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/BufferedOutputStream.java > * > ./nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/ByteArrayInputStream.java > * > ./nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/ByteArrayOutputStream.java > * > ./nifi-commons/nifi-write-ahead-log/src/main/java/org/wali/MinimalLockingWriteAheadLog.java -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11240) Introduce Python API for building Processors
[ https://issues.apache.org/jira/browse/NIFI-11240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17695866#comment-17695866 ] Otto Fowler commented on NIFI-11240: We should include a cookie cutter template creating projects. > Introduce Python API for building Processors > > > Key: NIFI-11240 > URL: https://issues.apache.org/jira/browse/NIFI-11240 > Project: Apache NiFi > Issue Type: Epic > Components: Core Framework, Documentation Website, Extensions >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 2.0.0 > > > The scripting processors are very common for data transformation in NiFi. In > particular, the Jython based scripts are quite heavily used. However, Jython > is run on the JVM and does not support CPython libraries. As a result, it's > syntax compatible but doesn't make use of the wealth of Python libraries. And > the wealth of Python libraries are what make Python popular to begin with. > Additionally, use of many script-based processors hurts the UX. They are > cumbersome to configure, with script files and/or script bodies. They result > in a dataflow that's difficult to understand because instead of nicely named > processors like CompressContent the type and default name are > "ExecuteScript." They're also difficult to share. > I have been playing with Py4J for introduce a true Python-based API for > developing Processors. This will introduce new APIs, new framework changes, > and documentation. And this will likely take a while to stabilize. However, > the sooner that we are able to land it into the hands of users, the better. > Therefore, I pose that we introduce it in multiple milestones. We can create > sub-tickets for different milestones, but in general it should follow: > Milestone 1: Initial implementation. Provides the capability and an API for > building processors. Includes sample code and some documentation. Includes > tests to ensure proper operation. Should not be used in production. API will > not be stable and may change frequently. Performance may be subpar. Get into > the hands of developers to begin exploring and providing feedback / > submitting PRs. > Milestone 2: Bug fixes. API refinement. Improve performance. > Milestone 3: Additional bug fixes and API refinement. API should become more > stable. > Milestone 4: Additional bug fixes. API becomes stable. Documentation is clear > and sufficient. Recommend production use. > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-9015) Ability to derive custom Rest API processes from InvokeHTTP
[ https://issues.apache.org/jira/browse/NIFI-9015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otto Fowler resolved NIFI-9015. --- Resolution: Duplicate This is accomplished in NIFI-10244 > Ability to derive custom Rest API processes from InvokeHTTP > --- > > Key: NIFI-9015 > URL: https://issues.apache.org/jira/browse/NIFI-9015 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > In setups with custom development, there is a lot of boilerplate > customization to InvokeHTTP around some known set of rest apis. As such flow > developers / users have to 'know' and understand these things in order to > setup possibly multiple InvokeHTTP instances with these details. > Some users may instead want to create a top level derived processor with > custom setup and parameters for a specific rest api for easy configuration > and deployment. > The > [InvokeAWSGatewayApi|https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-aws-nar/1.14.0/org.apache.nifi.processors.aws.wag.InvokeAWSGatewayApi/index.html] > processor ( which itself had to be copied from InvokeHTTP because of this > limitation ) offers this, such that you can create a derived processor for a > custom AWS Web Gateway service. > With a coming processor registry, the ability for people to contribute > specialized processors for certain APIs would be great. These may be a > subset or a super set of the existing properties. We may also move the > 'base' properties to a new UI tab in the configuration, such that you can add > custom extra configuration in the first focus tab. We could also allow for > removing properties as well from the base tab, etc etc > This task would involve refactoring the invokeHTTP processor such that there > is a reusable base, with the InvokeHTTP processor being a pass-through > default implementation ( IE> a new derived from base would have all the same > features as InvokeHTTP ). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (NIFI-9015) Ability to derive custom Rest API processes from InvokeHTTP
[ https://issues.apache.org/jira/browse/NIFI-9015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otto Fowler closed NIFI-9015. - > Ability to derive custom Rest API processes from InvokeHTTP > --- > > Key: NIFI-9015 > URL: https://issues.apache.org/jira/browse/NIFI-9015 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > In setups with custom development, there is a lot of boilerplate > customization to InvokeHTTP around some known set of rest apis. As such flow > developers / users have to 'know' and understand these things in order to > setup possibly multiple InvokeHTTP instances with these details. > Some users may instead want to create a top level derived processor with > custom setup and parameters for a specific rest api for easy configuration > and deployment. > The > [InvokeAWSGatewayApi|https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-aws-nar/1.14.0/org.apache.nifi.processors.aws.wag.InvokeAWSGatewayApi/index.html] > processor ( which itself had to be copied from InvokeHTTP because of this > limitation ) offers this, such that you can create a derived processor for a > custom AWS Web Gateway service. > With a coming processor registry, the ability for people to contribute > specialized processors for certain APIs would be great. These may be a > subset or a super set of the existing properties. We may also move the > 'base' properties to a new UI tab in the configuration, such that you can add > custom extra configuration in the first focus tab. We could also allow for > removing properties as well from the base tab, etc etc > This task would involve refactoring the invokeHTTP processor such that there > is a reusable base, with the InvokeHTTP processor being a pass-through > default implementation ( IE> a new derived from base would have all the same > features as InvokeHTTP ). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-9015) Ability to derive custom Rest API processes from InvokeHTTP
[ https://issues.apache.org/jira/browse/NIFI-9015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otto Fowler reassigned NIFI-9015: - Assignee: Otto Fowler > Ability to derive custom Rest API processes from InvokeHTTP > --- > > Key: NIFI-9015 > URL: https://issues.apache.org/jira/browse/NIFI-9015 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > In setups with custom development, there is a lot of boilerplate > customization to InvokeHTTP around some known set of rest apis. As such flow > developers / users have to 'know' and understand these things in order to > setup possibly multiple InvokeHTTP instances with these details. > Some users may instead want to create a top level derived processor with > custom setup and parameters for a specific rest api for easy configuration > and deployment. > The > [InvokeAWSGatewayApi|https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-aws-nar/1.14.0/org.apache.nifi.processors.aws.wag.InvokeAWSGatewayApi/index.html] > processor ( which itself had to be copied from InvokeHTTP because of this > limitation ) offers this, such that you can create a derived processor for a > custom AWS Web Gateway service. > With a coming processor registry, the ability for people to contribute > specialized processors for certain APIs would be great. These may be a > subset or a super set of the existing properties. We may also move the > 'base' properties to a new UI tab in the configuration, such that you can add > custom extra configuration in the first focus tab. We could also allow for > removing properties as well from the base tab, etc etc > This task would involve refactoring the invokeHTTP processor such that there > is a reusable base, with the InvokeHTTP processor being a pass-through > default implementation ( IE> a new derived from base would have all the same > features as InvokeHTTP ). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-9015) Ability to derive custom Rest API processes from InvokeHTTP
[ https://issues.apache.org/jira/browse/NIFI-9015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17580890#comment-17580890 ] Otto Fowler commented on NIFI-9015: --- [~exceptionfactory]: I have been following the PR review and I think that great work. I'll close this ticket. > Ability to derive custom Rest API processes from InvokeHTTP > --- > > Key: NIFI-9015 > URL: https://issues.apache.org/jira/browse/NIFI-9015 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Otto Fowler >Priority: Major > > In setups with custom development, there is a lot of boilerplate > customization to InvokeHTTP around some known set of rest apis. As such flow > developers / users have to 'know' and understand these things in order to > setup possibly multiple InvokeHTTP instances with these details. > Some users may instead want to create a top level derived processor with > custom setup and parameters for a specific rest api for easy configuration > and deployment. > The > [InvokeAWSGatewayApi|https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-aws-nar/1.14.0/org.apache.nifi.processors.aws.wag.InvokeAWSGatewayApi/index.html] > processor ( which itself had to be copied from InvokeHTTP because of this > limitation ) offers this, such that you can create a derived processor for a > custom AWS Web Gateway service. > With a coming processor registry, the ability for people to contribute > specialized processors for certain APIs would be great. These may be a > subset or a super set of the existing properties. We may also move the > 'base' properties to a new UI tab in the configuration, such that you can add > custom extra configuration in the first focus tab. We could also allow for > removing properties as well from the base tab, etc etc > This task would involve refactoring the invokeHTTP processor such that there > is a reusable base, with the InvokeHTTP processor being a pass-through > default implementation ( IE> a new derived from base would have all the same > features as InvokeHTTP ). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-10207) ProcessorRunStatusEntity does not declare RUN_ONCE in ApiModelValue
[ https://issues.apache.org/jira/browse/NIFI-10207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otto Fowler resolved NIFI-10207. Fix Version/s: 1.17.0 Resolution: Fixed > ProcessorRunStatusEntity does not declare RUN_ONCE in ApiModelValue > --- > > Key: NIFI-10207 > URL: https://issues.apache.org/jira/browse/NIFI-10207 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.14.0 >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > Fix For: 1.17.0 > > Time Spent: 1h > Remaining Estimate: 0h > > A new run status of RUN_ONCE was added to the system in NIFI-8188. > The ApiModelValue was not set however, so OpenApi/Swagger generations do not > have support for RUN_ONCE as an allowable value, and get errors. > see: https://github.com/Chaffelson/nipyapi/issues/309 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10207) ProcessorRunStatusEntity does not declare RUN_ONCE in ApiModelValue
[ https://issues.apache.org/jira/browse/NIFI-10207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otto Fowler updated NIFI-10207: --- Description: A new run status of RUN_ONCE was added to the system in NIFI-8188. The ApiModelValue was not set however, so OpenApi/Swagger generations do not have support for RUN_ONCE as an allowable value, and get errors. see: https://github.com/Chaffelson/nipyapi/issues/309 > ProcessorRunStatusEntity does not declare RUN_ONCE in ApiModelValue > --- > > Key: NIFI-10207 > URL: https://issues.apache.org/jira/browse/NIFI-10207 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.14.0 >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > A new run status of RUN_ONCE was added to the system in NIFI-8188. > The ApiModelValue was not set however, so OpenApi/Swagger generations do not > have support for RUN_ONCE as an allowable value, and get errors. > see: https://github.com/Chaffelson/nipyapi/issues/309 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-10207) ProcessorRunStatusEntity does not declare RUN_ONCE in ApiModelValue
[ https://issues.apache.org/jira/browse/NIFI-10207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otto Fowler reassigned NIFI-10207: -- Assignee: Otto Fowler > ProcessorRunStatusEntity does not declare RUN_ONCE in ApiModelValue > --- > > Key: NIFI-10207 > URL: https://issues.apache.org/jira/browse/NIFI-10207 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.14.0 >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-10207) ProcessorRunStatusEntity does not declare RUN_ONCE in ApiModelValue
Otto Fowler created NIFI-10207: -- Summary: ProcessorRunStatusEntity does not declare RUN_ONCE in ApiModelValue Key: NIFI-10207 URL: https://issues.apache.org/jira/browse/NIFI-10207 Project: Apache NiFi Issue Type: Bug Components: Core Framework Affects Versions: 1.14.0 Reporter: Otto Fowler -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-9863) Controller Service for managing custom Grok patterns
[ https://issues.apache.org/jira/browse/NIFI-9863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521937#comment-17521937 ] Otto Fowler commented on NIFI-9863: --- Yeah, this is fine in the backlog > Controller Service for managing custom Grok patterns > > > Key: NIFI-9863 > URL: https://issues.apache.org/jira/browse/NIFI-9863 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Otto Fowler >Priority: Major > > Managing custom Grok expressions in properties for the Grok processors or > Record readers is cumbersome and not ideal. > Having a service that managed these expressions in a centralized and reusable > way would be a benefit to those using Grok patterns. > This service would allow the configuration of some number custom Grok > patterns as the service configuration. The MVP would be manual entry, but > loading patterns from File ( upload to configuration? ) or from some external > location could be allowed as well down the line. > In use, it could be argued that the patterns should be loaded from something > like the schema registry. > consumers of the service should then be able select the specific service > instance and then using dependent properties select which patterns provided > by the service to consume. > To this end, it may be nice to have the service support pattern 'groups', > such that you can select all patterns for a group at once. This would be the > easy button version of the linked multiple expressions to grok reader issue. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (NIFI-9863) Controller Service for managing custom Grok patterns
[ https://issues.apache.org/jira/browse/NIFI-9863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otto Fowler updated NIFI-9863: -- Priority: Minor (was: Major) > Controller Service for managing custom Grok patterns > > > Key: NIFI-9863 > URL: https://issues.apache.org/jira/browse/NIFI-9863 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Otto Fowler >Priority: Minor > > Managing custom Grok expressions in properties for the Grok processors or > Record readers is cumbersome and not ideal. > Having a service that managed these expressions in a centralized and reusable > way would be a benefit to those using Grok patterns. > This service would allow the configuration of some number custom Grok > patterns as the service configuration. The MVP would be manual entry, but > loading patterns from File ( upload to configuration? ) or from some external > location could be allowed as well down the line. > In use, it could be argued that the patterns should be loaded from something > like the schema registry. > consumers of the service should then be able select the specific service > instance and then using dependent properties select which patterns provided > by the service to consume. > To this end, it may be nice to have the service support pattern 'groups', > such that you can select all patterns for a group at once. This would be the > easy button version of the linked multiple expressions to grok reader issue. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-9863) Controller Service for managing custom Grok patterns
[ https://issues.apache.org/jira/browse/NIFI-9863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17519127#comment-17519127 ] Otto Fowler commented on NIFI-9863: --- A new reader may indeed be a better idea. For the reader: I'm not sure how you would do the selection in the UI, I would think a custom page/tab/view that lets you select the items. The items should have a display name ( editable in the configuration ) and an actual id that doesn't change etc etc. > Controller Service for managing custom Grok patterns > > > Key: NIFI-9863 > URL: https://issues.apache.org/jira/browse/NIFI-9863 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Otto Fowler >Priority: Major > > Managing custom Grok expressions in properties for the Grok processors or > Record readers is cumbersome and not ideal. > Having a service that managed these expressions in a centralized and reusable > way would be a benefit to those using Grok patterns. > This service would allow the configuration of some number custom Grok > patterns as the service configuration. The MVP would be manual entry, but > loading patterns from File ( upload to configuration? ) or from some external > location could be allowed as well down the line. > In use, it could be argued that the patterns should be loaded from something > like the schema registry. > consumers of the service should then be able select the specific service > instance and then using dependent properties select which patterns provided > by the service to consume. > To this end, it may be nice to have the service support pattern 'groups', > such that you can select all patterns for a group at once. This would be the > easy button version of the linked multiple expressions to grok reader issue. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Comment Edited] (NIFI-9863) Controller Service for managing custom Grok patterns
[ https://issues.apache.org/jira/browse/NIFI-9863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17517143#comment-17517143 ] Otto Fowler edited comment on NIFI-9863 at 4/4/22 10:46 PM: [I'm going to keep this Grok specific, but there is a generic class of operation here I *think*] On the controller and its configuration side: * Allow entering in of grok patterns with test and validation * Allow entering of grok expressions with test and validation * Allow grouping of expressions into logical sets * Allow schema discovery from expressions and association with the expression * the schema of a set needs to be the same for each expression in the set * possibly schema for a pattern as well So, let me manage my grok patterns in a reusable way, and help me to make sure that they do what I think they do, and that the schema is known so I don't mess that up. from the consumer? List getPatterns() List getAvailablePatterns() PatternObjectMayHaveSchema getPattern(String name) List getExpressions() List getAvailableExpressions() ExpressionObjectMayHaveSchema getExpression(String name) List getExpressionSets() List getAvailableExpressionSets() ExpressionSet getExpressionSet(String name) We can also have a cache for compiled expressions ( that times out ) associated was (Author: ottobackwards): [I'm going to keep this Grok specific, but there is a generic class of operation here I *think*] On the controller and its configuration side: * Allow entering in of grok patterns with test and validation * Allow entering of grok expressions with test and validation * Allow grouping of expressions into logical sets * Allow schema discovery from expressions and association with the expression * the schema of a set needs to be the same for each expression in the set * possibly schema for a pattern as well So, let me manage my grok patterns in a reusable way, and help me to make sure that they do what I think they do, and that the schema is know so I don't mess that up. from the consumer? List getPatterns() List getAvailablePatterns() PatternObjectMayHaveSchema getPattern(String name) List getExpressions() List getAvailableExpressions() ExpressionObjectMayHaveSchema getExpression(String name) List getExpressionSets() List getAvailableExpressionSets() ExpressionSet getExpressionSet(String name) We can also have a cache for compiled expressions ( that times out ) associated > Controller Service for managing custom Grok patterns > > > Key: NIFI-9863 > URL: https://issues.apache.org/jira/browse/NIFI-9863 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Otto Fowler >Priority: Major > > Managing custom Grok expressions in properties for the Grok processors or > Record readers is cumbersome and not ideal. > Having a service that managed these expressions in a centralized and reusable > way would be a benefit to those using Grok patterns. > This service would allow the configuration of some number custom Grok > patterns as the service configuration. The MVP would be manual entry, but > loading patterns from File ( upload to configuration? ) or from some external > location could be allowed as well down the line. > In use, it could be argued that the patterns should be loaded from something > like the schema registry. > consumers of the service should then be able select the specific service > instance and then using dependent properties select which patterns provided > by the service to consume. > To this end, it may be nice to have the service support pattern 'groups', > such that you can select all patterns for a group at once. This would be the > easy button version of the linked multiple expressions to grok reader issue. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-9863) Controller Service for managing custom Grok patterns
[ https://issues.apache.org/jira/browse/NIFI-9863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17517143#comment-17517143 ] Otto Fowler commented on NIFI-9863: --- [I'm going to keep this Grok specific, but there is a generic class of operation here I *think*] On the controller and it's configuration side: * Allow entering in of grok patterns with test and validation * Allow entering of grok expressions with test and validation * Allow grouping of expressions into logical sets * Allow schema discovery from expressions and association with the expression * the schema of a set needs to be the same for each expression in the set * possibly schema for a pattern as well So, let me manage my grok patterns in a reusable way, and help me to make sure that they do what I think they do, and that the schema is know so I don't mess that up. from the consumer? List getPatterns() List getAvailablePatterns() PatternObjectMayHaveSchema getPattern(String name) List getExpressions() List getAvailableExpressions() ExpressionObjectMayHaveSchema getExpression(String name) List getExpressionSets() List getAvailableExpressionSets() ExpressionSet getExpressionSet(String name) We can also have a cache for compiled expressions ( that times out ) associated > Controller Service for managing custom Grok patterns > > > Key: NIFI-9863 > URL: https://issues.apache.org/jira/browse/NIFI-9863 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Otto Fowler >Priority: Major > > Managing custom Grok expressions in properties for the Grok processors or > Record readers is cumbersome and not ideal. > Having a service that managed these expressions in a centralized and reusable > way would be a benefit to those using Grok patterns. > This service would allow the configuration of some number custom Grok > patterns as the service configuration. The MVP would be manual entry, but > loading patterns from File ( upload to configuration? ) or from some external > location could be allowed as well down the line. > In use, it could be argued that the patterns should be loaded from something > like the schema registry. > consumers of the service should then be able select the specific service > instance and then using dependent properties select which patterns provided > by the service to consume. > To this end, it may be nice to have the service support pattern 'groups', > such that you can select all patterns for a group at once. This would be the > easy button version of the linked multiple expressions to grok reader issue. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Comment Edited] (NIFI-9863) Controller Service for managing custom Grok patterns
[ https://issues.apache.org/jira/browse/NIFI-9863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17517143#comment-17517143 ] Otto Fowler edited comment on NIFI-9863 at 4/4/22 10:45 PM: [I'm going to keep this Grok specific, but there is a generic class of operation here I *think*] On the controller and its configuration side: * Allow entering in of grok patterns with test and validation * Allow entering of grok expressions with test and validation * Allow grouping of expressions into logical sets * Allow schema discovery from expressions and association with the expression * the schema of a set needs to be the same for each expression in the set * possibly schema for a pattern as well So, let me manage my grok patterns in a reusable way, and help me to make sure that they do what I think they do, and that the schema is know so I don't mess that up. from the consumer? List getPatterns() List getAvailablePatterns() PatternObjectMayHaveSchema getPattern(String name) List getExpressions() List getAvailableExpressions() ExpressionObjectMayHaveSchema getExpression(String name) List getExpressionSets() List getAvailableExpressionSets() ExpressionSet getExpressionSet(String name) We can also have a cache for compiled expressions ( that times out ) associated was (Author: ottobackwards): [I'm going to keep this Grok specific, but there is a generic class of operation here I *think*] On the controller and it's configuration side: * Allow entering in of grok patterns with test and validation * Allow entering of grok expressions with test and validation * Allow grouping of expressions into logical sets * Allow schema discovery from expressions and association with the expression * the schema of a set needs to be the same for each expression in the set * possibly schema for a pattern as well So, let me manage my grok patterns in a reusable way, and help me to make sure that they do what I think they do, and that the schema is know so I don't mess that up. from the consumer? List getPatterns() List getAvailablePatterns() PatternObjectMayHaveSchema getPattern(String name) List getExpressions() List getAvailableExpressions() ExpressionObjectMayHaveSchema getExpression(String name) List getExpressionSets() List getAvailableExpressionSets() ExpressionSet getExpressionSet(String name) We can also have a cache for compiled expressions ( that times out ) associated > Controller Service for managing custom Grok patterns > > > Key: NIFI-9863 > URL: https://issues.apache.org/jira/browse/NIFI-9863 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Otto Fowler >Priority: Major > > Managing custom Grok expressions in properties for the Grok processors or > Record readers is cumbersome and not ideal. > Having a service that managed these expressions in a centralized and reusable > way would be a benefit to those using Grok patterns. > This service would allow the configuration of some number custom Grok > patterns as the service configuration. The MVP would be manual entry, but > loading patterns from File ( upload to configuration? ) or from some external > location could be allowed as well down the line. > In use, it could be argued that the patterns should be loaded from something > like the schema registry. > consumers of the service should then be able select the specific service > instance and then using dependent properties select which patterns provided > by the service to consume. > To this end, it may be nice to have the service support pattern 'groups', > such that you can select all patterns for a group at once. This would be the > easy button version of the linked multiple expressions to grok reader issue. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Comment Edited] (NIFI-9863) Controller Service for managing custom Grok patterns
[ https://issues.apache.org/jira/browse/NIFI-9863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17516424#comment-17516424 ] Otto Fowler edited comment on NIFI-9863 at 4/3/22 2:58 AM: --- Sorry [~exceptionfactory] Even our documentation is confusing on that matter: Grok Pattern File Path to a file that contains Grok Patterns to use for parsing logs. If not specified, a built-in default Pattern file will be used. If specified, all patterns in the given pattern file will override the default patterns. See the Controller Service's Additional Details for a list of pre-defined patterns. This property requires exactly one file to be provided.. - pattern here too Grok Expression Specifies the format of a log line in Grok format. This allows the Record Reader to understand how to parse each log line. If a line in the log file does not match this pattern, the line will be assumed to belong to the previous log message.If other Grok expressions are referenced by this expression, they need to be supplied in the Grok Pattern File So what I had in mind was a controller that supplied patterns, or more specifically a controller INTERFACE for a controller that supplied patterns, some default implementation But, there would be no reason for it not to provide expressions as well as patterns. Maybe some kind of lookup would be more generic actually, since it is just looking up string by name? Sorry for the confusion. I usually just use pattern for Grok * from back when we used it in metron and a contributed to it a small bit. And as you see the docs slightly confusing ( at least to me ). The idea in my head, maybe invalid ( or probably ) is that expression like this, that may be reused between different processors in the flow, should be shared and updatable from a single source as opposed to have to type them in properties over and over again as much as possible. This is probably something that I wouldn't implement until there is an ask though was (Author: ottobackwards): Sorry [~exceptionfactory] Even our documentation is confusing on that matter: Grok Pattern File Path to a file that contains Grok Patterns to use for parsing logs. If not specified, a built-in default Pattern file will be used. If specified, all patterns in the given pattern file will override the default patterns. See the Controller Service's Additional Details for a list of pre-defined patterns. This property requires exactly one file to be provided.. - pattern here too Grok Expression Specifies the format of a log line in Grok format. This allows the Record Reader to understand how to parse each log line. If a line in the log file does not match this pattern, the line will be assumed to belong to the previous log message.If other Grok expressions are referenced by this expression, they need to be supplied in the Grok Pattern File So what I had in mind was a controller that supplied patterns, or more specifically a controller INTERFACE for a controller that supplied patterns, some default implementation But, there would be no reason for it to provide expressions as well as patterns. Maybe some kind of lookup would be more generic actually. Sorry for the confusion. I usually just use pattern for Grok * from back when we used it in metron and a contributed to it a small bit. And as you see the docs slightly confusing ( at least to me ). The idea in my head, maybe invalid ( or probably ) is that expression like this, that may be reused between different processors in the flow, should be shared and updatable from a single source as opposed to have to type them in properties over and over again as much as possible. This is probably something that I wouldn't implement until there is an ask though > Controller Service for managing custom Grok patterns > > > Key: NIFI-9863 > URL: https://issues.apache.org/jira/browse/NIFI-9863 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Otto Fowler >Priority: Major > > Managing custom Grok expressions in properties for the Grok processors or > Record readers is cumbersome and not ideal. > Having a service that managed these expressions in a centralized and reusable > way would be a benefit to those using Grok patterns. > This service would allow the configuration of some number custom Grok > patterns as the service configuration. The MVP would be manual entry, but > loading patterns from File ( upload to configuration? ) or from some external > location could be allowed as well down the line. > In use, it could be argued that the patterns should be loaded from something > like the schema registry. > consumers of the service should then be able
[jira] [Commented] (NIFI-9863) Controller Service for managing custom Grok patterns
[ https://issues.apache.org/jira/browse/NIFI-9863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17516424#comment-17516424 ] Otto Fowler commented on NIFI-9863: --- Sorry [~exceptionfactory] Even our documentation is confusing on that matter: Grok Pattern File Path to a file that contains Grok Patterns to use for parsing logs. If not specified, a built-in default Pattern file will be used. If specified, all patterns in the given pattern file will override the default patterns. See the Controller Service's Additional Details for a list of pre-defined patterns. This property requires exactly one file to be provided.. - pattern here too Grok Expression Specifies the format of a log line in Grok format. This allows the Record Reader to understand how to parse each log line. If a line in the log file does not match this pattern, the line will be assumed to belong to the previous log message.If other Grok expressions are referenced by this expression, they need to be supplied in the Grok Pattern File So what I had in mind was a controller that supplied patterns, or more specifically a controller INTERFACE for a controller that supplied patterns, some default implementation But, there would be no reason for it to provide expressions as well as patterns. Maybe some kind of lookup would be more generic actually. Sorry for the confusion. I usually just use pattern for Grok * from back when we used it in metron and a contributed to it a small bit. And as you see the docs slightly confusing ( at least to me ). The idea in my head, maybe invalid ( or probably ) is that expression like this, that may be reused between different processors in the flow, should be shared and updatable from a single source as opposed to have to type them in properties over and over again as much as possible. This is probably something that I wouldn't implement until there is an ask though > Controller Service for managing custom Grok patterns > > > Key: NIFI-9863 > URL: https://issues.apache.org/jira/browse/NIFI-9863 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Otto Fowler >Priority: Major > > Managing custom Grok expressions in properties for the Grok processors or > Record readers is cumbersome and not ideal. > Having a service that managed these expressions in a centralized and reusable > way would be a benefit to those using Grok patterns. > This service would allow the configuration of some number custom Grok > patterns as the service configuration. The MVP would be manual entry, but > loading patterns from File ( upload to configuration? ) or from some external > location could be allowed as well down the line. > In use, it could be argued that the patterns should be loaded from something > like the schema registry. > consumers of the service should then be able select the specific service > instance and then using dependent properties select which patterns provided > by the service to consume. > To this end, it may be nice to have the service support pattern 'groups', > such that you can select all patterns for a group at once. This would be the > easy button version of the linked multiple expressions to grok reader issue. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (NIFI-9863) Controller Service for managing custom Grok patterns
[ https://issues.apache.org/jira/browse/NIFI-9863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otto Fowler updated NIFI-9863: -- Description: Managing custom Grok expressions in properties for the Grok processors or Record readers is cumbersome and not ideal. Having a service that managed these expressions in a centralized and reusable way would be a benefit to those using Grok patterns. This service would allow the configuration of some number custom Grok patterns as the service configuration. The MVP would be manual entry, but loading patterns from File ( upload to configuration? ) or from some external location could be allowed as well down the line. In use, it could be argued that the patterns should be loaded from something like the schema registry. consumers of the service should then be able select the specific service instance and then using dependent properties select which patterns provided by the service to consume. To this end, it may be nice to have the service support pattern 'groups', such that you can select all patterns for a group at once. This would be the easy button version of the linked multiple expressions to grok reader issue. > Controller Service for managing custom Grok patterns > > > Key: NIFI-9863 > URL: https://issues.apache.org/jira/browse/NIFI-9863 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Otto Fowler >Priority: Major > > Managing custom Grok expressions in properties for the Grok processors or > Record readers is cumbersome and not ideal. > Having a service that managed these expressions in a centralized and reusable > way would be a benefit to those using Grok patterns. > This service would allow the configuration of some number custom Grok > patterns as the service configuration. The MVP would be manual entry, but > loading patterns from File ( upload to configuration? ) or from some external > location could be allowed as well down the line. > In use, it could be argued that the patterns should be loaded from something > like the schema registry. > consumers of the service should then be able select the specific service > instance and then using dependent properties select which patterns provided > by the service to consume. > To this end, it may be nice to have the service support pattern 'groups', > such that you can select all patterns for a group at once. This would be the > easy button version of the linked multiple expressions to grok reader issue. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (NIFI-9863) Controller Service for managing custom Grok patterns
Otto Fowler created NIFI-9863: - Summary: Controller Service for managing custom Grok patterns Key: NIFI-9863 URL: https://issues.apache.org/jira/browse/NIFI-9863 Project: Apache NiFi Issue Type: New Feature Reporter: Otto Fowler -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (NIFI-7262) Add detailed logging to PutCassandraRecord
[ https://issues.apache.org/jira/browse/NIFI-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otto Fowler resolved NIFI-7262. --- Resolution: Done > Add detailed logging to PutCassandraRecord > -- > > Key: NIFI-7262 > URL: https://issues.apache.org/jira/browse/NIFI-7262 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.11.3 >Reporter: Arpad Boda >Assignee: Mike Thomsen >Priority: Minor > Time Spent: 0.5h > Remaining Estimate: 0h > > PutCassandraRecord doesn't provide any logging to log the queries generated. > Some debug lines would help integration. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-9462) JsonTreeReader schema inference only examines first record for parts of structure, causing data loss for subsequent records
[
https://issues.apache.org/jira/browse/NIFI-9462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17466546#comment-17466546
]
Otto Fowler commented on NIFI-9462:
---
Sorry [~Josiah.Johnston], I was greatly mistaken
> JsonTreeReader schema inference only examines first record for parts of
> structure, causing data loss for subsequent records
> ---
>
> Key: NIFI-9462
> URL: https://issues.apache.org/jira/browse/NIFI-9462
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.13.0
>Reporter: Josiah Johnston
>Priority: Major
> Attachments: JSON record set writer.png, JSON tree reader.png,
> NIFI-9462_example.xml, flow.png, updateRecord.png
>
>
> I use GenerateFlowFile with this JSON line content, and send it through an
> UpdateRecord processor that uses a JsonTreeReader with schema inference. The
> UpdateRecord processor adds a top level `s3_key` element with the filename.
> {"_source": {"name": "battery-voltage-changed", "metadata":
> {"voltage": 2.8}
> }}
> {"_source": {"name": "temperature-changed", "metadata":
> {"temperature": 19.54}
> }}
> {"other_L1_keys": "are_preserved", "_source": {"other_L2_keys":
> "are_preserved", "metadata":
> {"voltage": 6.3, "other_L3_keys": "are_lost"}
> }}
> In the output, the structure of `_source.metadata.*` is always strictly based
> on the first record, causing data loss for subsequent records that have
> different fields.
> {"_source":{"name":"battery-voltage-changed","metadata":
> {"voltage":2.8}
> ,"other_L2_keys":null},"other_L1_keys":null,"s3_key":"9830423c-c8b6-4a03-a4a1-427750e94d26"}
> {"_source":{"name":"temperature-changed","metadata":
> {"voltage":null}
> ,"other_L2_keys":null},"other_L1_keys":null,"s3_key":"9830423c-c8b6-4a03-a4a1-427750e94d26"}
> {"_source":{"name":null,"metadata":
> {"voltage":6.3}
> ,"other_L2_keys":"are_preserved"},"other_L1_keys":"are_preserved","s3_key":"9830423c-c8b6-4a03-a4a1-427750e94d26"}
> In general it drops all 3rd level keys weren't seen in the first record
> (_source.metadata.temperature in record 2, _source.metadata.other_L3_keys in
> record 3). This behavior only applies to keys in the 3rd level; schema
> inference works as documented (scanning through all records) for alternative
> keys in the 1st & 2nd level.
> This behavior persists whether I specify the input as JSON lines (shown in
> this example), or if I rearrange it to be a JSON array.
> I've attached screenshots of a minimal example and settings of JSON reader &
> writer. I've also attached a template of a minimal example. If you import it,
> you'll need to create & enable controller services of JsonTreeReader &
> JsonRecordSetWriter (default values for each).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (NIFI-9462) JsonTreeReader schema inference only examines first record for parts of structure, causing data loss for subsequent records
[
https://issues.apache.org/jira/browse/NIFI-9462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17466504#comment-17466504
]
Otto Fowler commented on NIFI-9462:
---
I believe this is working correctly. When you are passing multiple json
structures to the schema inference it is going to assume that they are
homogeneous and use the first one.
> JsonTreeReader schema inference only examines first record for parts of
> structure, causing data loss for subsequent records
> ---
>
> Key: NIFI-9462
> URL: https://issues.apache.org/jira/browse/NIFI-9462
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.13.0
>Reporter: Josiah Johnston
>Priority: Major
> Attachments: JSON record set writer.png, JSON tree reader.png,
> NIFI-9462_example.xml, flow.png, updateRecord.png
>
>
> I use GenerateFlowFile with this JSON line content, and send it through an
> UpdateRecord processor that uses a JsonTreeReader with schema inference. The
> UpdateRecord processor adds a top level `s3_key` element with the filename.
> {"_source": {"name": "battery-voltage-changed", "metadata":
> {"voltage": 2.8}
> }}
> {"_source": {"name": "temperature-changed", "metadata":
> {"temperature": 19.54}
> }}
> {"other_L1_keys": "are_preserved", "_source": {"other_L2_keys":
> "are_preserved", "metadata":
> {"voltage": 6.3, "other_L3_keys": "are_lost"}
> }}
> In the output, the structure of `_source.metadata.*` is always strictly based
> on the first record, causing data loss for subsequent records that have
> different fields.
> {"_source":{"name":"battery-voltage-changed","metadata":
> {"voltage":2.8}
> ,"other_L2_keys":null},"other_L1_keys":null,"s3_key":"9830423c-c8b6-4a03-a4a1-427750e94d26"}
> {"_source":{"name":"temperature-changed","metadata":
> {"voltage":null}
> ,"other_L2_keys":null},"other_L1_keys":null,"s3_key":"9830423c-c8b6-4a03-a4a1-427750e94d26"}
> {"_source":{"name":null,"metadata":
> {"voltage":6.3}
> ,"other_L2_keys":"are_preserved"},"other_L1_keys":"are_preserved","s3_key":"9830423c-c8b6-4a03-a4a1-427750e94d26"}
> In general it drops all 3rd level keys weren't seen in the first record
> (_source.metadata.temperature in record 2, _source.metadata.other_L3_keys in
> record 3). This behavior only applies to keys in the 3rd level; schema
> inference works as documented (scanning through all records) for alternative
> keys in the 1st & 2nd level.
> This behavior persists whether I specify the input as JSON lines (shown in
> this example), or if I rearrange it to be a JSON array.
> I've attached screenshots of a minimal example and settings of JSON reader &
> writer. I've also attached a template of a minimal example. If you import it,
> you'll need to create & enable controller services of JsonTreeReader &
> JsonRecordSetWriter (default values for each).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (NIFI-9518) Update mysql-binlog-connector-java library to maintained fork
[ https://issues.apache.org/jira/browse/NIFI-9518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otto Fowler updated NIFI-9518: -- Summary: Update mysql-binlog-connector-java library to maintained fork (was: Update mysql-binlog-connector-java library) > Update mysql-binlog-connector-java library to maintained fork > - > > Key: NIFI-9518 > URL: https://issues.apache.org/jira/browse/NIFI-9518 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.15.1 >Reporter: Anton Koval >Priority: Major > > I noticed that a processor CaptureChangeMySQL uses library > ([https://github.com/shyiko/mysql-binlog-connector-java]), which is no longer > maintained. It's recommend migrating to > [https://github.com/osheroff/mysql-binlog-connector-java] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-9498) Ability to report what components a flow is actually using
[ https://issues.apache.org/jira/browse/NIFI-9498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17461597#comment-17461597 ] Otto Fowler commented on NIFI-9498: --- Thanks Mark! > Ability to report what components a flow is actually using > -- > > Key: NIFI-9498 > URL: https://issues.apache.org/jira/browse/NIFI-9498 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Otto Fowler >Priority: Major > > Nifi may have 100s of NARS and their components loaded, but for any flow the > actual working set of NARS and components is a subset. > We often recommend that people remove NARS that aren't used from the nifi > working set for various reasons, but there is no concrete way to identify > what NARs you can remove. > NIFI should provide some set of features around allowing visibility into what > the working set of NARS and components are for a given flow. > There are different levels of feature around this idea to be explored: > * a cli command that produces a report document from a live instance or from > flow definition file > * a ui component that visualizes the graph and can export a report > * Actions from these things, to remove or restrict what is loaded in a > cluster ( imagine a cluster whitelist for NARS that is share in the cluster ) > * ??? > * profit!!! -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (NIFI-9498) Ability to report what components a flow is actually using
Otto Fowler created NIFI-9498: - Summary: Ability to report what components a flow is actually using Key: NIFI-9498 URL: https://issues.apache.org/jira/browse/NIFI-9498 Project: Apache NiFi Issue Type: New Feature Reporter: Otto Fowler Nifi may have 100s of NARS and their components loaded, but for any flow the actual working set of NARS and components is a subset. We often recommend that people remove NARS that aren't used from the nifi working set for various reasons, but there is no concrete way to identify what NARs you can remove. NIFI should provide some set of features around allowing visibility into what the working set of NARS and components are for a given flow. There are different levels of feature around this idea to be explored: * a cli command that produces a report document from a live instance or from flow definition file * a ui component that visualizes the graph and can export a report * Actions from these things, to remove or restrict what is loaded in a cluster ( imagine a cluster whitelist for NARS that is share in the cluster ) * ??? * profit!!! -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-8706) Add support for Redis Lists and Hashes
[ https://issues.apache.org/jira/browse/NIFI-8706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452393#comment-17452393 ] Otto Fowler commented on NIFI-8706: --- These are great examples of redis commands, can you put them in a nifi context? "I would like to write attributes to redis"? "I would like to write one or more fields from records to redis" "I would like to write fields from records that match a record query to redis" "I would like have a record writer that writes json to redis" something like that? > Add support for Redis Lists and Hashes > -- > > Key: NIFI-8706 > URL: https://issues.apache.org/jira/browse/NIFI-8706 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.8.0, 1.13.2 >Reporter: Doug Houck >Priority: Major > Labels: newbie, performance > > Nifi supports Redis interactions, but only for Keys. From a Redis > perspective, this is a poor use of resources. Lists and Hashes only required > one key per. It would be nice to have the ability to interact with Redis > Lists and Hashes in a Nifi Processor. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Closed] (NIFI-9349) InvokeHTTP - add attribute with call duration
[ https://issues.apache.org/jira/browse/NIFI-9349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otto Fowler closed NIFI-9349. - > InvokeHTTP - add attribute with call duration > - > > Key: NIFI-9349 > URL: https://issues.apache.org/jira/browse/NIFI-9349 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Fix For: 1.15.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Add an attribute to the FlowFile like "invokehttp.request.duration" to > provide in milliseconds the response time of the remote endpoint. At this > time, this can be retrieved by looking at the Event Duration value of the > corresponding provenance event. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-9349) InvokeHTTP - add attribute with call duration
[ https://issues.apache.org/jira/browse/NIFI-9349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438266#comment-17438266 ] Otto Fowler commented on NIFI-9349: --- oops, sorry > InvokeHTTP - add attribute with call duration > - > > Key: NIFI-9349 > URL: https://issues.apache.org/jira/browse/NIFI-9349 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Fix For: 1.15.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Add an attribute to the FlowFile like "invokehttp.request.duration" to > provide in milliseconds the response time of the remote endpoint. At this > time, this can be retrieved by looking at the Event Duration value of the > corresponding provenance event. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-9015) Ability to derive custom Rest API processes from InvokeHTTP
[ https://issues.apache.org/jira/browse/NIFI-9015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otto Fowler updated NIFI-9015: -- Description: In setups with custom development, there is a lot of boilerplate customization to InvokeHTTP around some known set of rest apis. As such flow developers / users have to 'know' and understand these things in order to setup possibly multiple InvokeHTTP instances with these details. Some users may instead want to create a top level derived processor with custom setup and parameters for a specific rest api for easy configuration and deployment. The [InvokeAWSGatewayApi|https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-aws-nar/1.14.0/org.apache.nifi.processors.aws.wag.InvokeAWSGatewayApi/index.html] processor ( which itself had to be copied from InvokeHTTP because of this limitation ) offers this, such that you can create a derived processor for a custom AWS Web Gateway service. With a coming processor registry, the ability for people to contribute specialized processors for certain APIs would be great. These may be a subset or a super set of the existing properties. We may also move the 'base' properties to a new UI tab in the configuration, such that you can add custom extra configuration in the first focus tab. We could also allow for removing properties as well from the base tab, etc etc This task would involve refactoring the invokeHTTP processor such that there is a reusable base, with the InvokeHTTP processor being a pass-through default implementation ( IE> a new derived from base would have all the same features as InvokeHTTP ). was: In setups with custom development, there is a lot of boilerplate customization to InvokeHTTP around some known set of rest apis. As such flow developers / users have to 'know' and understand these things in order to setup possibly multiple InvokeHTTP instances with these details. Some users may instead want to create a top level derived processor with custom setup and parameters for a specific rest api for easy configuration and deployment. The [InvokeAWSGatewayApi|https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-aws-nar/1.14.0/org.apache.nifi.processors.aws.wag.InvokeAWSGatewayApi/index.html] processor ( which itself had to be copied from InvokeHTTP because of this limitation ) offers this, such that you can create a derived processor for a custom AWS Web Gateway service. With a coming processor registry, the ability for people to contribute specialized processors for certain APIs would be great. This task would involve refactoring the invokeHTTP processor such that there is a reusable base, with the InvokeHTTP processor being a pass-through default implementation ( IE> a new derived from base would have all the same features as InvokeHTTP ). > Ability to derive custom Rest API processes from InvokeHTTP > --- > > Key: NIFI-9015 > URL: https://issues.apache.org/jira/browse/NIFI-9015 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Otto Fowler >Priority: Major > > In setups with custom development, there is a lot of boilerplate > customization to InvokeHTTP around some known set of rest apis. As such flow > developers / users have to 'know' and understand these things in order to > setup possibly multiple InvokeHTTP instances with these details. > Some users may instead want to create a top level derived processor with > custom setup and parameters for a specific rest api for easy configuration > and deployment. > The > [InvokeAWSGatewayApi|https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-aws-nar/1.14.0/org.apache.nifi.processors.aws.wag.InvokeAWSGatewayApi/index.html] > processor ( which itself had to be copied from InvokeHTTP because of this > limitation ) offers this, such that you can create a derived processor for a > custom AWS Web Gateway service. > With a coming processor registry, the ability for people to contribute > specialized processors for certain APIs would be great. These may be a > subset or a super set of the existing properties. We may also move the > 'base' properties to a new UI tab in the configuration, such that you can add > custom extra configuration in the first focus tab. We could also allow for > removing properties as well from the base tab, etc etc > This task would involve refactoring the invokeHTTP processor such that there > is a reusable base, with the InvokeHTTP processor being a pass-through > default implementation ( IE> a new derived from base would have all the same > features as InvokeHTTP ). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-9015) Ability to derive custom Rest API processes from InvokeHTTP
Otto Fowler created NIFI-9015: - Summary: Ability to derive custom Rest API processes from InvokeHTTP Key: NIFI-9015 URL: https://issues.apache.org/jira/browse/NIFI-9015 Project: Apache NiFi Issue Type: Improvement Reporter: Otto Fowler In setups with custom development, there is a lot of boilerplate customization to InvokeHTTP around some known set of rest apis. As such flow developers / users have to 'know' and understand these things in order to setup possibly multiple InvokeHTTP instances with these details. Some users may instead want to create a top level derived processor with custom setup and parameters for a specific rest api for easy configuration and deployment. The [InvokeAWSGatewayApi|https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-aws-nar/1.14.0/org.apache.nifi.processors.aws.wag.InvokeAWSGatewayApi/index.html] processor ( which itself had to be copied from InvokeHTTP because of this limitation ) offers this, such that you can create a derived processor for a custom AWS Web Gateway service. With a coming processor registry, the ability for people to contribute specialized processors for certain APIs would be great. This task would involve refactoring the invokeHTTP processor such that there is a reusable base, with the InvokeHTTP processor being a pass-through default implementation ( IE> a new derived from base would have all the same features as InvokeHTTP ). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8778) maven version requirements out of date in readme and enforcer configuration
[ https://issues.apache.org/jira/browse/NIFI-8778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17380077#comment-17380077 ] Otto Fowler commented on NIFI-8778: --- I change both, thanks [~jfrazee] > maven version requirements out of date in readme and enforcer configuration > --- > > Key: NIFI-8778 > URL: https://issues.apache.org/jira/browse/NIFI-8778 > Project: Apache NiFi > Issue Type: Bug > Components: Documentation Website >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > [ERROR] Failed to execute goal > com.github.eirslett:frontend-maven-plugin:1.12.0:install-node-and-npm > (install-node-and-npm) on project nifi-web-ui: The plugin > com.github.eirslett:frontend-maven-plugin:1.12.0 requires Maven version 3.6.0 > -> [Help 1] > I have maven version: > Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; > 2018-06-17T14:33:14-04:00) > Maven home: /usr/local/Cellar/maven@3.5/3.5.4_1/libexec > Java version: 1.8.0_292, vendor: AdoptOpenJDK, runtime: > /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre > What is the minimum version of maven required to build nifi? is it now 3.6.0? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (NIFI-8778) maven version requirements out of date in readme and enforcer configuration
[ https://issues.apache.org/jira/browse/NIFI-8778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otto Fowler reassigned NIFI-8778: - Assignee: Otto Fowler > maven version requirements out of date in readme and enforcer configuration > --- > > Key: NIFI-8778 > URL: https://issues.apache.org/jira/browse/NIFI-8778 > Project: Apache NiFi > Issue Type: Bug > Components: Documentation Website >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > [ERROR] Failed to execute goal > com.github.eirslett:frontend-maven-plugin:1.12.0:install-node-and-npm > (install-node-and-npm) on project nifi-web-ui: The plugin > com.github.eirslett:frontend-maven-plugin:1.12.0 requires Maven version 3.6.0 > -> [Help 1] > I have maven version: > Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; > 2018-06-17T14:33:14-04:00) > Maven home: /usr/local/Cellar/maven@3.5/3.5.4_1/libexec > Java version: 1.8.0_292, vendor: AdoptOpenJDK, runtime: > /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre > What is the minimum version of maven required to build nifi? is it now 3.6.0? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8778) maven version requirements out of date in readme and enforcer configuration
[ https://issues.apache.org/jira/browse/NIFI-8778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otto Fowler updated NIFI-8778: -- Summary: maven version requirements out of date in readme and enforcer configuration (was: README maven version requirements out of date) > maven version requirements out of date in readme and enforcer configuration > --- > > Key: NIFI-8778 > URL: https://issues.apache.org/jira/browse/NIFI-8778 > Project: Apache NiFi > Issue Type: Bug > Components: Documentation Website >Reporter: Otto Fowler >Priority: Major > > [ERROR] Failed to execute goal > com.github.eirslett:frontend-maven-plugin:1.12.0:install-node-and-npm > (install-node-and-npm) on project nifi-web-ui: The plugin > com.github.eirslett:frontend-maven-plugin:1.12.0 requires Maven version 3.6.0 > -> [Help 1] > I have maven version: > Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; > 2018-06-17T14:33:14-04:00) > Maven home: /usr/local/Cellar/maven@3.5/3.5.4_1/libexec > Java version: 1.8.0_292, vendor: AdoptOpenJDK, runtime: > /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre > What is the minimum version of maven required to build nifi? is it now 3.6.0? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8778) maven version requirements out of date in readme and enforcer configuration
[ https://issues.apache.org/jira/browse/NIFI-8778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17379426#comment-17379426 ] Otto Fowler commented on NIFI-8778: --- Good catch, changed the title. > maven version requirements out of date in readme and enforcer configuration > --- > > Key: NIFI-8778 > URL: https://issues.apache.org/jira/browse/NIFI-8778 > Project: Apache NiFi > Issue Type: Bug > Components: Documentation Website >Reporter: Otto Fowler >Priority: Major > > [ERROR] Failed to execute goal > com.github.eirslett:frontend-maven-plugin:1.12.0:install-node-and-npm > (install-node-and-npm) on project nifi-web-ui: The plugin > com.github.eirslett:frontend-maven-plugin:1.12.0 requires Maven version 3.6.0 > -> [Help 1] > I have maven version: > Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; > 2018-06-17T14:33:14-04:00) > Maven home: /usr/local/Cellar/maven@3.5/3.5.4_1/libexec > Java version: 1.8.0_292, vendor: AdoptOpenJDK, runtime: > /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre > What is the minimum version of maven required to build nifi? is it now 3.6.0? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-8778) README maven version requirements out of date
Otto Fowler created NIFI-8778: - Summary: README maven version requirements out of date Key: NIFI-8778 URL: https://issues.apache.org/jira/browse/NIFI-8778 Project: Apache NiFi Issue Type: Improvement Components: Documentation Website Reporter: Otto Fowler [ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.12.0:install-node-and-npm (install-node-and-npm) on project nifi-web-ui: The plugin com.github.eirslett:frontend-maven-plugin:1.12.0 requires Maven version 3.6.0 -> [Help 1] I have maven version: Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T14:33:14-04:00) Maven home: /usr/local/Cellar/maven@3.5/3.5.4_1/libexec Java version: 1.8.0_292, vendor: AdoptOpenJDK, runtime: /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre What is the minimum version of maven required to build nifi? is it now 3.6.0? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8778) README maven version requirements out of date
[ https://issues.apache.org/jira/browse/NIFI-8778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otto Fowler updated NIFI-8778: -- Issue Type: Bug (was: Improvement) > README maven version requirements out of date > - > > Key: NIFI-8778 > URL: https://issues.apache.org/jira/browse/NIFI-8778 > Project: Apache NiFi > Issue Type: Bug > Components: Documentation Website >Reporter: Otto Fowler >Priority: Major > > [ERROR] Failed to execute goal > com.github.eirslett:frontend-maven-plugin:1.12.0:install-node-and-npm > (install-node-and-npm) on project nifi-web-ui: The plugin > com.github.eirslett:frontend-maven-plugin:1.12.0 requires Maven version 3.6.0 > -> [Help 1] > I have maven version: > Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; > 2018-06-17T14:33:14-04:00) > Maven home: /usr/local/Cellar/maven@3.5/3.5.4_1/libexec > Java version: 1.8.0_292, vendor: AdoptOpenJDK, runtime: > /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre > What is the minimum version of maven required to build nifi? is it now 3.6.0? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8706) Add support for Redis Lists and Hashes
[ https://issues.apache.org/jira/browse/NIFI-8706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17364970#comment-17364970 ] Otto Fowler commented on NIFI-8706: --- Can you give any use cases for these interactions? The ability to write X to a list or hash, with records, with example? The ability to read X from list or hash to records with example? > Add support for Redis Lists and Hashes > -- > > Key: NIFI-8706 > URL: https://issues.apache.org/jira/browse/NIFI-8706 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.8.0, 1.13.2 >Reporter: Doug Houck >Priority: Major > Labels: newbie, performance > > Nifi supports Redis interactions, but only for Keys. From a Redis > perspective, this is a poor use of resources. Lists and Hashes only required > one key per. It would be nice to have the ability to interact with Redis > Lists and Hashes in a Nifi Processor. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-8685) [NIFI-SITE] nifi site list of committers and pmc is out of date
Otto Fowler created NIFI-8685: - Summary: [NIFI-SITE] nifi site list of committers and pmc is out of date Key: NIFI-8685 URL: https://issues.apache.org/jira/browse/NIFI-8685 Project: Apache NiFi Issue Type: Improvement Components: Documentation Website Reporter: Otto Fowler Looks like it hasn't been updated in a while. We need the current list and to update it -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8625) ExecuteScript processor always stuck after restart or multi thread
[ https://issues.apache.org/jira/browse/NIFI-8625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17351886#comment-17351886 ] Otto Fowler commented on NIFI-8625: --- so this is a duplicate? > ExecuteScript processor always stuck after restart or multi thread > -- > > Key: NIFI-8625 > URL: https://issues.apache.org/jira/browse/NIFI-8625 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.13.2 >Reporter: KevinSky >Priority: Critical > Labels: ExecuteScript,stcuk > Attachments: executeScript_nifi_8625.xml, > image-2021-05-21-16-22-34-775.png > > > In single thread, executeScript just stop and start, flow file always stuck > in connections. > In multi thread.executeScript always throw exception like is already marked > for transfer in or is not known in this session. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8625) ExecuteScript processor always stuck after restart or multi thread
[ https://issues.apache.org/jira/browse/NIFI-8625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17351225#comment-17351225 ] Otto Fowler commented on NIFI-8625: --- I'm going to try to get some feedback or tips from [~mburgess149] > ExecuteScript processor always stuck after restart or multi thread > -- > > Key: NIFI-8625 > URL: https://issues.apache.org/jira/browse/NIFI-8625 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.13.2 >Reporter: KevinSky >Priority: Critical > Labels: ExecuteScript,stcuk > Attachments: executeScript_nifi_8625.xml, > image-2021-05-21-16-22-34-775.png > > > In single thread, executeScript just stop and start, flow file always stuck > in connections. > In multi thread.executeScript always throw exception like is already marked > for transfer in or is not known in this session. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8625) ExecuteScript processor always stuck after restart or multi thread
[
https://issues.apache.org/jira/browse/NIFI-8625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17351179#comment-17351179
]
Otto Fowler commented on NIFI-8625:
---
also seeing : e -> {PyBaseExceptionDerived@19457} "AttributeError("'NoneType'
object has no attribute 'getAttribute'",)
> ExecuteScript processor always stuck after restart or multi thread
> --
>
> Key: NIFI-8625
> URL: https://issues.apache.org/jira/browse/NIFI-8625
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.13.2
>Reporter: KevinSky
>Priority: Critical
> Labels: ExecuteScript,stcuk
> Attachments: executeScript_nifi_8625.xml,
> image-2021-05-21-16-22-34-775.png
>
>
> In single thread, executeScript just stop and start, flow file always stuck
> in connections.
> In multi thread.executeScript always throw exception like is already marked
> for transfer in or is not known in this session.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-8625) ExecuteScript processor always stuck after restart or multi thread
[ https://issues.apache.org/jira/browse/NIFI-8625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17351176#comment-17351176 ] Otto Fowler commented on NIFI-8625: --- It looks like once I get the 'not known to session' exception, every new file fails with that and the queue is not being processed > ExecuteScript processor always stuck after restart or multi thread > -- > > Key: NIFI-8625 > URL: https://issues.apache.org/jira/browse/NIFI-8625 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.13.2 >Reporter: KevinSky >Priority: Critical > Labels: ExecuteScript,stcuk > Attachments: executeScript_nifi_8625.xml, > image-2021-05-21-16-22-34-775.png > > > In single thread, executeScript just stop and start, flow file always stuck > in connections. > In multi thread.executeScript always throw exception like is already marked > for transfer in or is not known in this session. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8625) ExecuteScript processor always stuck after restart or multi thread
[ https://issues.apache.org/jira/browse/NIFI-8625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17351171#comment-17351171 ] Otto Fowler commented on NIFI-8625: --- What I'm doing: I have 4 files in the input directory after processing the first four, I do a `cp` like from 1.txt to 5.txt etc to add a file at a time Sometimes it works, sometimes I get the exception above > ExecuteScript processor always stuck after restart or multi thread > -- > > Key: NIFI-8625 > URL: https://issues.apache.org/jira/browse/NIFI-8625 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.13.2 >Reporter: KevinSky >Priority: Critical > Labels: ExecuteScript,stcuk > Attachments: executeScript_nifi_8625.xml, > image-2021-05-21-16-22-34-775.png > > > In single thread, executeScript just stop and start, flow file always stuck > in connections. > In multi thread.executeScript always throw exception like is already marked > for transfer in or is not known in this session. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8625) ExecuteScript processor always stuck after restart or multi thread
[
https://issues.apache.org/jira/browse/NIFI-8625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17351169#comment-17351169
]
Otto Fowler commented on NIFI-8625:
---
also seeing this at times
{noformat}
xecuteScript[id=bb911eff-0bbc-346e-6a34-3875ae197ff0]
org.apache.nifi.processor.exception.FlowFileHandlingException:
StandardFlowFileRecord[uuid=4cc02d23-3763-4b09-99e2-725d67924ba4,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1621958091080-1, container=default,
section=1], offset=43, length=4],offset=0,name=5.txt,size=4] transfer
relationship not specified
at
org.apache.nifi.controller.repository.StandardProcessSession.validateCommitState(StandardProcessSession.java:245)
at
org.apache.nifi.controller.repository.StandardProcessSession.checkpoint(StandardProcessSession.java:259)
at
org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:406)
at
org.apache.nifi.controller.repository.StandardProcessSession.commitAsync(StandardProcessSession.java:362)
at
org.apache.nifi.processors.script.ExecuteScript.onTrigger(ExecuteScript.java:273)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1180)
at org.apache.nifi.controller.tasks.Con
{noformat}
> ExecuteScript processor always stuck after restart or multi thread
> --
>
> Key: NIFI-8625
> URL: https://issues.apache.org/jira/browse/NIFI-8625
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.13.2
>Reporter: KevinSky
>Priority: Critical
> Labels: ExecuteScript,stcuk
> Attachments: executeScript_nifi_8625.xml,
> image-2021-05-21-16-22-34-775.png
>
>
> In single thread, executeScript just stop and start, flow file always stuck
> in connections.
> In multi thread.executeScript always throw exception like is already marked
> for transfer in or is not known in this session.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-8625) ExecuteScript processor always stuck after restart or multi thread
[
https://issues.apache.org/jira/browse/NIFI-8625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17351162#comment-17351162
]
Otto Fowler commented on NIFI-8625:
---
ok, i'm seeing mixed results:
newest error detail is
{noformat}
org.apache.nifi.processor.exception.FlowFileHandlingException:
org.apache.nifi.processor.exception.FlowFileHandlingException:
StandardFlowFileRecord[uuid=29345e45-3fc6-4859-9449-0ce12f640e6f,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1621958091080-1, container=default,
section=1], offset=9, length=4],offset=0,name=2.txt,size=4] is already marked
for transfer
{noformat}
> ExecuteScript processor always stuck after restart or multi thread
> --
>
> Key: NIFI-8625
> URL: https://issues.apache.org/jira/browse/NIFI-8625
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.13.2
>Reporter: KevinSky
>Priority: Critical
> Labels: ExecuteScript,stcuk
> Attachments: executeScript_nifi_8625.xml,
> image-2021-05-21-16-22-34-775.png
>
>
> In single thread, executeScript just stop and start, flow file always stuck
> in connections.
> In multi thread.executeScript always throw exception like is already marked
> for transfer in or is not known in this session.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-8625) ExecuteScript processor always stuck after restart or multi thread
[
https://issues.apache.org/jira/browse/NIFI-8625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17351132#comment-17351132
]
Otto Fowler commented on NIFI-8625:
---
Strangely enough, I try to reproduce in the debugger, and it works:
{noformat}
2021-05-25 11:24:14,368 INFO [Timer-Driven Process Thread-10]
o.a.n.processors.standard.LogAttribute
LogAttribute[id=a402624a-0179-1000-3c1d-e423f684707b] logging for flow file
StandardFlowFileRecord[uuid=5a7bc46c-5a63-4a32-9fac-35913fa0c087,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1621954150437-1, container=default,
section=1], offset=38, length=4],offset=0,name=1.txt,size=4]
--
Standard FlowFile Attributes
Key: 'entryDate'
Value: 'Tue May 25 11:24:00 EDT 2021'
Key: 'lineageStartDate'
Value: 'Tue May 25 11:24:00 EDT 2021'
Key: 'fileSize'
Value: '4'
FlowFile Attribute Map Content
Key: 'absolute.path'
Value: '/Users/ottofowler/tmp/nifi-input/'
Key: 'file.creationTime'
Value: '2021-05-25T10:45:26-0400'
Key: 'file.group'
Value: 'staff'
Key: 'file.lastAccessTime'
Value: '2021-05-25T10:49:10-0400'
Key: 'file.lastModifiedTime'
Value: '2021-05-25T10:45:34-0400'
Key: 'file.owner'
Value: 'ottofowler'
Key: 'file.permissions'
Value: 'rw-r--r--'
Key: 'file.size'
Value: '4'
Key: 'filename'
Value: '1.txt'
Key: 'path'
Value: './'
Key: 'userCountName'
Value: 'D:\99_TestCase\4_MultiFile_Stuck\B5_ALL_CELL_.txt'
Key: 'uuid'
Value: '5a7bc46c-5a63-4a32-9fac-35913fa0c087'
--
one
{noformat}
> ExecuteScript processor always stuck after restart or multi thread
> --
>
> Key: NIFI-8625
> URL: https://issues.apache.org/jira/browse/NIFI-8625
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.13.2
>Reporter: KevinSky
>Priority: Critical
> Labels: ExecuteScript,stcuk
> Attachments: executeScript_nifi_8625.xml,
> image-2021-05-21-16-22-34-775.png
>
>
> In single thread, executeScript just stop and start, flow file always stuck
> in connections.
> In multi thread.executeScript always throw exception like is already marked
> for transfer in or is not known in this session.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-8625) ExecuteScript processor always stuck after restart or multi thread
[
https://issues.apache.org/jira/browse/NIFI-8625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17351112#comment-17351112
]
Otto Fowler commented on NIFI-8625:
---
Thanks!
I just ran the template against main:
* change input directory
* remove filter
* input has 4 text files
* change log message ( success ) to logattributes
I am seeing exceptions in the nifi-app.log that are rolling back the session.
Can you see if you see those as well?
{noformat}
2021-05-25 10:53:01,402 ERROR [Timer-Driven Process Thread-9]
o.a.nifi.processors.script.ExecuteScript
ExecuteScript[id=bb911eff-0bbc-346e-6a34-3875ae197ff0]
ExecuteScript[id=bb911eff-0bbc-346e-6a34-3875ae197ff0] failed to process due to
org.apache.nifi.processor.exception.ProcessException:
javax.script.ScriptException: java.lang.NullPointerException:
java.lang.NullPointerException in
[jira] [Commented] (NIFI-8625) ExecuteScript processor always stuck after restart or multi thread
[ https://issues.apache.org/jira/browse/NIFI-8625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350560#comment-17350560 ] Otto Fowler commented on NIFI-8625: --- Can you provide a template or steps to reproduce? > ExecuteScript processor always stuck after restart or multi thread > -- > > Key: NIFI-8625 > URL: https://issues.apache.org/jira/browse/NIFI-8625 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.13.2 >Reporter: KevinSky >Priority: Critical > Labels: ExecuteScript,stcuk > Attachments: image-2021-05-21-16-22-34-775.png > > > In single thread, executeScript just stop and start, flow file always stuck > in connections. > In multi thread.executeScript always throw exception like is already marked > for transfer in or is not known in this session. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8400) SystemDiagnostics throws NPE on Windows
[ https://issues.apache.org/jira/browse/NIFI-8400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316461#comment-17316461 ] Otto Fowler commented on NIFI-8400: --- If you read this description, and initially thought "That variable name doesn't seem very long", please leave a comment here. :D > SystemDiagnostics throws NPE on Windows > --- > > Key: NIFI-8400 > URL: https://issues.apache.org/jira/browse/NIFI-8400 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Matt Burgess >Priority: Major > > SystemDiagnostics includes some Long member variables such as openFileHandles > that are not populated on Windows, so they remain null and when > getOpenFileHandles() is called, the null is cast to a long which throws an > NPE. > The member variables should be long not Long, thereby getting a default value > of zero and avoiding an NPE when the values are not populated. If Long is > used elsewhere, a null check should be added to avoid possible NPEs when > calling the setter methods. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8398) Maven 3.8.1 disables support for repositories using "http" protocol
[
https://issues.apache.org/jira/browse/NIFI-8398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17315774#comment-17315774
]
Otto Fowler commented on NIFI-8398:
---
I was just looking at this. I have opened and issue upstream:
https://github.com/fluenda/ParCEFone/issues/16.
Note: This may not be the only issue here, since the build doesn't continue.
It is possible that there are other dependencies like this that will have to be
handled one after the other
> Maven 3.8.1 disables support for repositories using "http" protocol
>
>
> Key: NIFI-8398
> URL: https://issues.apache.org/jira/browse/NIFI-8398
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Paul Grey
>Priority: Minor
>
> Maven 3.8.1 (released April 2021) has disabled support (by default) for
> "http" repositories. This causes an issue with tips of NiFi project when
> doing a clean build:
> {code:java}
> mvn clean install
> ...
> [INFO] nifi-stateless-system-test-suite ... SKIPPED
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 08:18 min
> [INFO] Finished at: 2021-04-06T13:20:08-04:00
> [INFO]
>
> [ERROR] Failed to execute goal on project nifi-standard-processors: Could not
> resolve dependencies for project
> org.apache.nifi:nifi-standard-processors:jar:1.14.0-SNAPSHOT: Failed to
> collect dependencies at com.fluenda:ParCEFone:jar:1.2.6 ->
> com.martiansoftware:macnificent:jar:0.2.0: Failed to read artifact descriptor
> for com.martiansoftware:macnificent:jar:0.2.0: Could not transfer artifact
> com.martiansoftware:macnificent:pom:0.2.0 from/to maven-default-http-blocker
> (http://0.0.0.0/): Blocked mirror for repositories: [martiansoftware
> (http://mvn.martiansoftware.com, default, releases+snapshots)] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :nifi-standard-processors
> pgrey@10457 nifi % {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (NIFI-8397) Update to latest Simple-Syslog-5424 to fix BOM issue
[ https://issues.apache.org/jira/browse/NIFI-8397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otto Fowler reassigned NIFI-8397: - Assignee: Otto Fowler > Update to latest Simple-Syslog-5424 to fix BOM issue > > > Key: NIFI-8397 > URL: https://issues.apache.org/jira/browse/NIFI-8397 > Project: Apache NiFi > Issue Type: Bug >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > The current simple-syslog-5424 has a bug handling BOM when included in syslog. > release 0.0.16 resolves this. > This release is also on mvn central and not bintray. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8397) Update to latest Simple-Syslog-5424 to fix BOM issue
[ https://issues.apache.org/jira/browse/NIFI-8397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otto Fowler updated NIFI-8397: -- Issue Type: Bug (was: Improvement) > Update to latest Simple-Syslog-5424 to fix BOM issue > > > Key: NIFI-8397 > URL: https://issues.apache.org/jira/browse/NIFI-8397 > Project: Apache NiFi > Issue Type: Bug >Reporter: Otto Fowler >Priority: Major > > The current simple-syslog-5424 has a bug handling BOM when included in syslog. > release 0.0.16 resolves this. > This release is also on mvn central and not bintray. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-8397) Update to latest Simple-Syslog-5424 to fix BOM issue
Otto Fowler created NIFI-8397: - Summary: Update to latest Simple-Syslog-5424 to fix BOM issue Key: NIFI-8397 URL: https://issues.apache.org/jira/browse/NIFI-8397 Project: Apache NiFi Issue Type: Improvement Reporter: Otto Fowler The current simple-syslog-5424 has a bug handling BOM when included in syslog. release 0.0.16 resolves this. This release is also on mvn central and not bintray. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFI-8354) ExecuteStreamCommand processor doesn't delete the temp file if the process start failed
[ https://issues.apache.org/jira/browse/NIFI-8354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otto Fowler resolved NIFI-8354. --- Resolution: Fixed > ExecuteStreamCommand processor doesn't delete the temp file if the process > start failed > --- > > Key: NIFI-8354 > URL: https://issues.apache.org/jira/browse/NIFI-8354 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Hsin-Ying Lee >Assignee: Hsin-Ying Lee >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > ExecuteStreamCommand processor doesn't delete the temp file if the process > start failed > The i-node of /tmp will be exhausted, and the service will run into an > unexpected situation -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-8351) Investigate and remove build warnings on documentation generation
Otto Fowler created NIFI-8351: - Summary: Investigate and remove build warnings on documentation generation Key: NIFI-8351 URL: https://issues.apache.org/jira/browse/NIFI-8351 Project: Apache NiFi Issue Type: Improvement Reporter: Otto Fowler Warning: Could not generate extensions' documentation 2237 org.apache.maven.plugin.MojoExecutionException: Failed to create Extension Documentation 2238 at org.apache.nifi.NarMojo.generateDocumentation (NarMojo.java:596) 2239 at org.apache.nifi.NarMojo.execute (NarMojo.java:499) snip. These are seen in the GitHub action builds. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8322) build failed on AArch64, Fedora 33
[ https://issues.apache.org/jira/browse/NIFI-8322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17301692#comment-17301692 ] Otto Fowler commented on NIFI-8322: --- So, that jar looks like it is built per supported platform. So kudu doesn't support the platform you are building on. There are two issues here. 1. You should open a jira with Kudu to support your platform 2. This jira should change to disabling kudu if if it won't build on a platform in nifi ( Or some other resolution ) maybe. > build failed on AArch64, Fedora 33 > --- > > Key: NIFI-8322 > URL: https://issues.apache.org/jira/browse/NIFI-8322 > Project: Apache NiFi > Issue Type: Bug >Reporter: Lutz Weischer >Priority: Major > > [jw@cn05 nifi]$ mvn clean install -DskipTests > ... > [INFO] --- maven-remote-resources-plugin:1.7.0:process > (process-resource-bundles) @ nifi-kudu-bundle --- > [INFO] Preparing remote bundle org.apache:apache-jar-resource-bundle:1.4 > [INFO] Copying 3 resources from 1 bundle. > [INFO] > [INFO] --- maven-compiler-plugin:3.8.1:testCompile (groovy-tests) @ > nifi-kudu-bundle --- > [INFO] No sources to compile > [INFO] > [INFO] --- maven-site-plugin:3.7.1:attach-descriptor (attach-descriptor) @ > nifi-kudu-bundle --- > [INFO] No site descriptor found: nothing to attach. > [INFO] > [INFO] --- maven-install-plugin:2.5.2:install (default-install) @ > nifi-kudu-bundle --- > [INFO] Installing > /home/jw/apache/nifi/nifi-nar-bundles/nifi-kudu-bundle/pom.xml to > /home/jw/.m2/repository/org/apache/nifi/nifi-kudu-bundle/1.14.0-SNAPSHOT/nifi-kudu-bundle-1.14.0-SNAPSHOT.pom > [INFO] > [INFO] < org.apache.nifi:nifi-kudu-processors > > > [INFO] Building nifi-kudu-processors 1.14.0-SNAPSHOT > [193/506] > [INFO] [ jar > ]- > [INFO] > > [INFO] Reactor Summary for nifi 1.14.0-SNAPSHOT: > [INFO] > [INFO] nifi ... SUCCESS [ 4.040 > s] > [INFO] nifi-api ... SUCCESS [ 12.535 > s] > [INFO] nifi-commons ... SUCCESS [ 0.185 > s] > [INFO] nifi-utils . SUCCESS [ 19.467 > s] > [INFO] nifi-security-utils-api SUCCESS [ 11.197 > s] > [INFO] nifi-properties SUCCESS [ 8.688 > s] > [INFO] nifi-security-utils SUCCESS [ 30.187 > s] > [INFO] nifi-framework-api . SUCCESS [ 12.632 > s] > [INFO] nifi-nar-bundles ... SUCCESS [ 0.987 > s] > [INFO] nifi-framework-bundle .. SUCCESS [ 0.145 > s] > [INFO] nifi-framework . SUCCESS [ 0.135 > s] > [INFO] nifi-properties-loader . SUCCESS [ 13.949 > s] > [INFO] nifi-data-provenance-utils . SUCCESS [ 14.710 > s] > [INFO] nifi-parameter . SUCCESS [ 7.130 > s] > [INFO] nifi-uuid5 . SUCCESS [ 2.995 > s] > [INFO] nifi-expression-language ... SUCCESS [ 26.723 > s] > [INFO] nifi-flowfile-packager . SUCCESS [ 6.828 > s] > [INFO] nifi-hl7-query-language SUCCESS [ 10.843 > s] > [INFO] nifi-json-utils SUCCESS [ 3.206 > s] > [INFO] nifi-logging-utils . SUCCESS [ 2.965 > s] > [INFO] nifi-metrics ... SUCCESS [ 8.832 > s] > [INFO] nifi-record SUCCESS [ 11.556 > s] > [INFO] nifi-record-path ... SUCCESS [ 14.880 > s] > [INFO] nifi-rocksdb-utils . SUCCESS [ 7.284 > s] > [INFO] nifi-schema-utils .. SUCCESS [ 8.517 > s] > [INFO] nifi-client-dto SUCCESS [ 14.054 > s] > [INFO] nifi-site-to-site-client ... SUCCESS [ 24.203 > s] > [INFO] nifi-socket-utils .. SUCCESS [ 14.809 > s] > [INFO] nifi-web-utils . SUCCESS [ 5.345 > s] > [INFO] nifi-write-ahead-log ... SUCCESS [ 13.261 > s] > [INFO] nifi-server-api SUCCESS [ 3.137 > s] > [INFO] nifi-bootstrap . SUCCESS [ 16.978 > s] > [INFO] nifi-nar-utils . SUCCESS [ 15.482 > s] > [INFO]
[jira] [Commented] (NIFI-8245) Beats processors are missing
[ https://issues.apache.org/jira/browse/NIFI-8245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288371#comment-17288371 ] Otto Fowler commented on NIFI-8245: --- This documentation would be better if it listed the processors / controller services as well. People may not know the nars, or may not 'make the connection' when reading the guide. > Beats processors are missing > > > Key: NIFI-8245 > URL: https://issues.apache.org/jira/browse/NIFI-8245 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration, Core UI >Affects Versions: 1.13.0 > Environment: Canvas >Reporter: Rory Sibbley >Priority: Minor > Labels: patch > Fix For: 1.12.1 > > > The Listen Beats processor is missing from NIFI version 1.13.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8179) NiFi standalon json validator processor
[ https://issues.apache.org/jira/browse/NIFI-8179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17282538#comment-17282538 ] Otto Fowler commented on NIFI-8179: --- Validate how? Validate against json schema? if so what version? where will the schema live? Validate as parsable? just valid json at all? > NiFi standalon json validator processor > --- > > Key: NIFI-8179 > URL: https://issues.apache.org/jira/browse/NIFI-8179 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: John Carole >Priority: Major > > A standalone processor that reads a json, makes sure its a valid json and > then reroute the flow file to success or failure based on the validation -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-3882) Unable to run NiFi behind AWS Load Balancer
[ https://issues.apache.org/jira/browse/NIFI-3882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17250369#comment-17250369 ] Otto Fowler commented on NIFI-3882: --- Maybe there would need to be multiple endpoints, with different logical meanings? Cluster Health would be one, would there be others? One that says some set of flows are running? One that says if some set of queues are full or back pressure is being applied? > Unable to run NiFi behind AWS Load Balancer > --- > > Key: NIFI-3882 > URL: https://issues.apache.org/jira/browse/NIFI-3882 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Reporter: Richard St. John >Priority: Major > > Running Apache NiFi behind an AWS Elastic Load Balancer fails. > Upon closer inspection, AWS Elastic Load Balancers only support the following > headers: >X-Forwarded-For >X-Forwarded-Proto >X-Forwarded-Port > In our case, we would like our ELB to be HTTPS enabled, but our NiFi cluster > is running in a trusted environment on port 8080. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8051) NIFI ApiOperations should define nicknames to generate unique swagger operationId
[
https://issues.apache.org/jira/browse/NIFI-8051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17240777#comment-17240777
]
Otto Fowler commented on NIFI-8051:
---
>From looking at this a little bit, fixing one will make the code get junk.
>I'm sure that code gen is the main thing here. It is unfortunate. Maybe
>there is a way to get both things correct IE> code gen the way it works now
>but with operationID's unique and not used as the method names?
> NIFI ApiOperations should define nicknames to generate unique swagger
> operationId
> -
>
> Key: NIFI-8051
> URL: https://issues.apache.org/jira/browse/NIFI-8051
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Otto Fowler
>Priority: Major
>
> The swagger definitions that nifi produces have duplicate operationId's for
> many operations.
> While the swagger implementation 'works' ( IE. it can generate clients with
> all the correct operations ) it is not per the spec, where unique
> operationIds are required.
> https://github.com/OAI/OpenAPI-Specification/blob/master/versions/2.0.md#fixed-fields-5
> Thus, tools that are written to the spec throw errors when trying to generate
> against the nifi api json, such as :
> {code:json}
> {
> "type": "DUPLICATE_OPERATIONID",
> "message": "Multiple OASs share operations with the same operationId
> 'getPropertyDescriptor'",
> "mitigation": "Ignore operation and maintain preexisting operation. The
> operation from the OAS 'NiFi Rest Api' will be ignored"
> },
> {
> "type": "DUPLICATE_OPERATIONID",
> "message": "Multiple OASs share operations with the same operationId
> 'updateRunStatus'",
> "mitigation": "Ignore operation and maintain preexisting operation. The
> operation from the OAS 'NiFi Rest Api' will be ignored"
> },
> {
> "type": "DUPLICATE_OPERATIONID",
> "message": "Multiple OASs share operations with the same operationId
> 'getState'",
> "mitigation": "Ignore operation and maintain preexisting operation. The
> operation from the OAS 'NiFi Rest Api' will be ignored"
> },
> {
> "type": "DUPLICATE_OPERATIONID",
> "message": "Multiple OASs share operations with the same operationId
> 'clearState'",
> "mitigation": "Ignore operation and maintain preexisting operation. The
> operation from the OAS 'NiFi Rest Api' will be ignored"
> },
> {
> "type": "MISSING_RESPONSE_SCHEMA",
> "message": "Operation DELETE /versions/active-requests/{id} has no
> (valid) response schema. You can use the fillEmptyResponses option to create
> a placeholder schema",
> "mitigation": "Ignore operation."
> },
> {
> "type": "DUPLICATE_OPERATIONID",
> "message": "Multiple OASs share operations with the same operationId
> 'deleteUpdateRequest'",
> "mitigation": "Ignore operation and maintain preexisting operation. The
> operation from the OAS 'NiFi Rest Api' will be ignored"
> }
> {code}
> the fix for this may be to define a "nickname", that would create a unique
> operationId, such as
> {code:java}
> @GET
> @Consumes(MediaType.WILDCARD)
> @Produces(MediaType.APPLICATION_JSON)
> @Path("/{id}/state")
> @ApiOperation(
> nickname="processor_get_state"
> value = "Gets the state for a processor",
> response = ComponentStateEntity.class,
> authorizations = {
> @Authorization(value = "Write - /processors/{uuid}")
> }
> )
> {code}
> to reproduce:
> https://loopback.io/openapi-to-graphql.html against the nifi openapi json
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-8051) NIFI ApiOperations should define nicknames to generate unique swagger operationId
[
https://issues.apache.org/jira/browse/NIFI-8051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17240185#comment-17240185
]
Otto Fowler commented on NIFI-8051:
---
Note: I do not know if fixing this changes the generation in a way that
regresses or breaks existing clients
> NIFI ApiOperations should define nicknames to generate unique swagger
> operationId
> -
>
> Key: NIFI-8051
> URL: https://issues.apache.org/jira/browse/NIFI-8051
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Otto Fowler
>Priority: Major
>
> The swagger definitions that nifi produces have duplicate operationId's for
> many operations.
> While the swagger implementation 'works' ( IE. it can generate clients with
> all the correct operations ) it is not per the spec, where unique
> operationIds are required.
> https://github.com/OAI/OpenAPI-Specification/blob/master/versions/2.0.md#fixed-fields-5
> Thus, tools that are written to the spec throw errors when trying to generate
> against the nifi api json, such as :
> {code:json}
> {
> "type": "DUPLICATE_OPERATIONID",
> "message": "Multiple OASs share operations with the same operationId
> 'getPropertyDescriptor'",
> "mitigation": "Ignore operation and maintain preexisting operation. The
> operation from the OAS 'NiFi Rest Api' will be ignored"
> },
> {
> "type": "DUPLICATE_OPERATIONID",
> "message": "Multiple OASs share operations with the same operationId
> 'updateRunStatus'",
> "mitigation": "Ignore operation and maintain preexisting operation. The
> operation from the OAS 'NiFi Rest Api' will be ignored"
> },
> {
> "type": "DUPLICATE_OPERATIONID",
> "message": "Multiple OASs share operations with the same operationId
> 'getState'",
> "mitigation": "Ignore operation and maintain preexisting operation. The
> operation from the OAS 'NiFi Rest Api' will be ignored"
> },
> {
> "type": "DUPLICATE_OPERATIONID",
> "message": "Multiple OASs share operations with the same operationId
> 'clearState'",
> "mitigation": "Ignore operation and maintain preexisting operation. The
> operation from the OAS 'NiFi Rest Api' will be ignored"
> },
> {
> "type": "MISSING_RESPONSE_SCHEMA",
> "message": "Operation DELETE /versions/active-requests/{id} has no
> (valid) response schema. You can use the fillEmptyResponses option to create
> a placeholder schema",
> "mitigation": "Ignore operation."
> },
> {
> "type": "DUPLICATE_OPERATIONID",
> "message": "Multiple OASs share operations with the same operationId
> 'deleteUpdateRequest'",
> "mitigation": "Ignore operation and maintain preexisting operation. The
> operation from the OAS 'NiFi Rest Api' will be ignored"
> }
> {code}
> the fix for this may be to define a "nickname", that would create a unique
> operationId, such as
> {code:java}
> @GET
> @Consumes(MediaType.WILDCARD)
> @Produces(MediaType.APPLICATION_JSON)
> @Path("/{id}/state")
> @ApiOperation(
> nickname="processor_get_state"
> value = "Gets the state for a processor",
> response = ComponentStateEntity.class,
> authorizations = {
> @Authorization(value = "Write - /processors/{uuid}")
> }
> )
> {code}
> to reproduce:
> https://loopback.io/openapi-to-graphql.html against the nifi openapi json
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (NIFI-8051) NIFI ApiOperations should define nicknames to generate unique swagger operationId
Otto Fowler created NIFI-8051:
-
Summary: NIFI ApiOperations should define nicknames to generate
unique swagger operationId
Key: NIFI-8051
URL: https://issues.apache.org/jira/browse/NIFI-8051
Project: Apache NiFi
Issue Type: Bug
Reporter: Otto Fowler
The swagger definitions that nifi produces have duplicate operationId's for
many operations.
While the swagger implementation 'works' ( IE. it can generate clients with
all the correct operations ) it is not per the spec, where unique operationIds
are required.
https://github.com/OAI/OpenAPI-Specification/blob/master/versions/2.0.md#fixed-fields-5
Thus, tools that are written to the spec throw errors when trying to generate
against the nifi api json, such as :
{code:json}
{
"type": "DUPLICATE_OPERATIONID",
"message": "Multiple OASs share operations with the same operationId
'getPropertyDescriptor'",
"mitigation": "Ignore operation and maintain preexisting operation. The
operation from the OAS 'NiFi Rest Api' will be ignored"
},
{
"type": "DUPLICATE_OPERATIONID",
"message": "Multiple OASs share operations with the same operationId
'updateRunStatus'",
"mitigation": "Ignore operation and maintain preexisting operation. The
operation from the OAS 'NiFi Rest Api' will be ignored"
},
{
"type": "DUPLICATE_OPERATIONID",
"message": "Multiple OASs share operations with the same operationId
'getState'",
"mitigation": "Ignore operation and maintain preexisting operation. The
operation from the OAS 'NiFi Rest Api' will be ignored"
},
{
"type": "DUPLICATE_OPERATIONID",
"message": "Multiple OASs share operations with the same operationId
'clearState'",
"mitigation": "Ignore operation and maintain preexisting operation. The
operation from the OAS 'NiFi Rest Api' will be ignored"
},
{
"type": "MISSING_RESPONSE_SCHEMA",
"message": "Operation DELETE /versions/active-requests/{id} has no
(valid) response schema. You can use the fillEmptyResponses option to create a
placeholder schema",
"mitigation": "Ignore operation."
},
{
"type": "DUPLICATE_OPERATIONID",
"message": "Multiple OASs share operations with the same operationId
'deleteUpdateRequest'",
"mitigation": "Ignore operation and maintain preexisting operation. The
operation from the OAS 'NiFi Rest Api' will be ignored"
}
{code}
the fix for this may be to define a "nickname", that would create a unique
operationId, such as
{code:java}
@GET
@Consumes(MediaType.WILDCARD)
@Produces(MediaType.APPLICATION_JSON)
@Path("/{id}/state")
@ApiOperation(
nickname="processor_get_state"
value = "Gets the state for a processor",
response = ComponentStateEntity.class,
authorizations = {
@Authorization(value = "Write - /processors/{uuid}")
}
)
{code}
to reproduce:
https://loopback.io/openapi-to-graphql.html against the nifi openapi json
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7998) Parquet file reader service throws Nullpointer Exception
[ https://issues.apache.org/jira/browse/NIFI-7998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17230580#comment-17230580 ] Otto Fowler commented on NIFI-7998: --- can you create a template of the reproduction setup and attach it? > Parquet file reader service throws Nullpointer Exception > > > Key: NIFI-7998 > URL: https://issues.apache.org/jira/browse/NIFI-7998 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.12.1 > Environment: Docker build on Kubernetes >Reporter: Dennis Kliche >Priority: Major > Attachments: Jira_Parquet_Error.json, > image-2020-11-11-22-33-31-019.png, image-2020-11-11-22-41-13-099.png, > image-2020-11-11-22-41-44-188.png > > > We have a pipeline that gets an parquet file and write it into a database via > PutDatabase Record. The content of the flow file is a parquet file. If the > flow file is processed we are getting a NullPointerException. > > !image-2020-11-11-22-33-31-019.png! > After some tests I found out, that the processor works like expected but if > the flowfile is a parquet file and the reader is a parquet reader it throws > this error. > To confirm that is has to do with the parquet reader, I used a very simple > approach. > Generate flowfile with JSON transform it to parquet and then back to JSON. > This resulted in the same error message. > In the attached Jira_Parquet_Error.json you can see my simple construction. > Here you can see the settings of the parquet writer: > !image-2020-11-11-22-41-13-099.png! > > Here you can see the settings of Parquet Reader > > !image-2020-11-11-22-41-44-188.png! > > Until update from 1.11.4 to 1.12.1 it worked. Since 1.12.1 we are getting > this issue. > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (NIFI-7995) Requesting create Parameter Context via API with Parameters null generates an NPE
[ https://issues.apache.org/jira/browse/NIFI-7995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otto Fowler reassigned NIFI-7995: - Assignee: Otto Fowler > Requesting create Parameter Context via API with Parameters null generates an > NPE > - > > Key: NIFI-7995 > URL: https://issues.apache.org/jira/browse/NIFI-7995 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.10.0, 1.12.1 >Reporter: Daniel Chaffelson >Assignee: Otto Fowler >Priority: Minor > > org.apache.nifi.web.api.ParameterContextResource.validateParameterNames(ParameterContextResource.java:403) > gets very upset if you submit a JSON with null for the Parameter listing > instead of an empty or populated list. It is a minor thing, but probably > worth tidying up in the validator. > Discovered because NiPyAPI defaults to Python 'None' for unpopulated > properties. > > Full logged error is: > 2020-11-09 13:20:28,612 ERROR [NiFi Web Server-22] > o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred: > java.lang.NullPointerException. Returning Internal Server Error response. > java.lang.NullPointerException: null > at > org.apache.nifi.web.api.ParameterContextResource.validateParameterNames(ParameterContextResource.java:403) > at > org.apache.nifi.web.api.ParameterContextResource.createParameterContext(ParameterContextResource.java:199) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) > at > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) > at > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) > at > org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200) > at > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) > at > org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) > at > org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) > at > org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) > at > org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) > at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) > at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) > at org.glassfish.jersey.internal.Errors.process(Errors.java:316) > at org.glassfish.jersey.internal.Errors.process(Errors.java:298) > at org.glassfish.jersey.internal.Errors.process(Errors.java:268) > at > org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) > at > org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) > at > org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) > at > org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) > at > org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) > at > org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) > at > org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) > at > org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) > at > org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1395) > at > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617) > at > org.apache.nifi.web.filter.RequestLogger.doFilter(RequestLogger.java:66) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) > at > org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) >
[jira] [Commented] (NIFI-7995) Requesting create Parameter Context via API with Parameters null generates an NPE
[ https://issues.apache.org/jira/browse/NIFI-7995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17230031#comment-17230031 ] Otto Fowler commented on NIFI-7995: --- do you have a python script to reproduce this? > Requesting create Parameter Context via API with Parameters null generates an > NPE > - > > Key: NIFI-7995 > URL: https://issues.apache.org/jira/browse/NIFI-7995 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.10.0, 1.12.1 >Reporter: Daniel Chaffelson >Priority: Minor > > org.apache.nifi.web.api.ParameterContextResource.validateParameterNames(ParameterContextResource.java:403) > gets very upset if you submit a JSON with null for the Parameter listing > instead of an empty or populated list. It is a minor thing, but probably > worth tidying up in the validator. > Discovered because NiPyAPI defaults to Python 'None' for unpopulated > properties. > > Full logged error is: > 2020-11-09 13:20:28,612 ERROR [NiFi Web Server-22] > o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred: > java.lang.NullPointerException. Returning Internal Server Error response. > java.lang.NullPointerException: null > at > org.apache.nifi.web.api.ParameterContextResource.validateParameterNames(ParameterContextResource.java:403) > at > org.apache.nifi.web.api.ParameterContextResource.createParameterContext(ParameterContextResource.java:199) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) > at > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) > at > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) > at > org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200) > at > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) > at > org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) > at > org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) > at > org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) > at > org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) > at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) > at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) > at org.glassfish.jersey.internal.Errors.process(Errors.java:316) > at org.glassfish.jersey.internal.Errors.process(Errors.java:298) > at org.glassfish.jersey.internal.Errors.process(Errors.java:268) > at > org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) > at > org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) > at > org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) > at > org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) > at > org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) > at > org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) > at > org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) > at > org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) > at > org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1395) > at > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617) > at > org.apache.nifi.web.filter.RequestLogger.doFilter(RequestLogger.java:66) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) > at >
[jira] [Commented] (NIFI-7822) Set raw query string attribute in HandleHttpRequest processor
[ https://issues.apache.org/jira/browse/NIFI-7822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199121#comment-17199121 ] Otto Fowler commented on NIFI-7822: --- I'm not saying we shouldn't do the raw sting option, just suggesting that we implement it such that we don't have multiple copies of things. > Set raw query string attribute in HandleHttpRequest processor > - > > Key: NIFI-7822 > URL: https://issues.apache.org/jira/browse/NIFI-7822 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.11.4 >Reporter: Yuriy Flyud >Priority: Major > > HandleHttpRequest parses a query string and writes output to the following > attributes: > * http.query.string > * http.query.param.XXX > The problem is that neither of these two options works as expected if I want > to use query parameters later in my flow. > First option holds a DECODED query string, so if we are using some characters > like & or = in query parameter names or values - there is no way to resolve > this. E.g. query string 'text=abc%26notAProp%3D25' will be decoded to > 'text=abc=25' which is a completely different query string with > additional parameter. > Second option does not handle duplicating query parameters. So query like > 'name=John=Colin' will be resolved to a single attribute 'name', with > value 'Colin'. > > A simple improvement would be to add an 'http.query.raw.string' at least to > give a possibility to parse this query string manually where needed. > ' -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7822) Set raw query string attribute in HandleHttpRequest processor
[ https://issues.apache.org/jira/browse/NIFI-7822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199038#comment-17199038 ] Otto Fowler commented on NIFI-7822: --- I would think that we may want to consider having a query attribute strategy, were you could say RAW or Attributes or something maybe. I wouldn't think we would want to have multiple copies of that. As for the parameters, they are a map, so they must have unique names. If you have something that has duplicated parameters, there is no way around this, unless the parameter names are made unique before putting in attributes, so if the processor says " if we already have an attribute with this name, from this query, append a number to the name" or something. > Set raw query string attribute in HandleHttpRequest processor > - > > Key: NIFI-7822 > URL: https://issues.apache.org/jira/browse/NIFI-7822 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.11.4 >Reporter: Yuriy Flyud >Priority: Major > > HandleHttpRequest parses a query string and writes output to the following > attributes: > * http.query.string > * http.query.param.XXX > The problem is that neither of these two options works as expected if I want > to use query parameters later in my flow. > First option holds a DECODED query string, so if we are using some characters > like & or = in query parameter names or values - there is no way to resolve > this. E.g. query string 'text=abc%26notAProp%3D25' will be decoded to > 'text=abc=25' which is a completely different query string with > additional parameter. > Second option does not handle duplicating query parameters. So query like > 'name=John=Colin' will be resolved to a single attribute 'name', with > value 'Colin'. > > A simple improvement would be to add an 'http.query.raw.string' at least to > give a possibility to parse this query string manually where needed. > ' -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (NIFI-7761) Allow HandleHttpRequest to add specified form data to FlowFile attributes
[ https://issues.apache.org/jira/browse/NIFI-7761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193203#comment-17193203 ] Otto Fowler edited comment on NIFI-7761 at 9/9/20, 9:54 PM: [~grego33] if there is any way you can try or review the PR it would help getting this landed was (Author: ottobackwards): [~grego33]if there is any way you can try or review the PR it would help getting this landed > Allow HandleHttpRequest to add specified form data to FlowFile attributes > - > > Key: NIFI-7761 > URL: https://issues.apache.org/jira/browse/NIFI-7761 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.12.0 >Reporter: Matt Gregory >Assignee: Otto Fowler >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > NIFI-7420 changed the behavior of HandleHttpRequest to no longer add > http.param.* attributes for all parts of a multipart request. This was nice > for scenarios when a single file + form data was being sent in a multipart as > the form data was readily accessible as FlowFile attributes. > > A nice approach suggested by ottobackwards [in > Slack|https://apachenifi.slack.com/archives/C0L9VCD47/p1598295952058200] is > to allow a list of form fields to be specified to the HandleHttpRequest > processor as a property. I would imagine this working similarly to how > LogAttribute specifies Attributes to Log, with a simple comma-separated list. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7761) Allow HandleHttpRequest to add specified form data to FlowFile attributes
[ https://issues.apache.org/jira/browse/NIFI-7761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193203#comment-17193203 ] Otto Fowler commented on NIFI-7761: --- [~grego33]if there is any way you can try or review the PR it would help getting this landed > Allow HandleHttpRequest to add specified form data to FlowFile attributes > - > > Key: NIFI-7761 > URL: https://issues.apache.org/jira/browse/NIFI-7761 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.12.0 >Reporter: Matt Gregory >Assignee: Otto Fowler >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > NIFI-7420 changed the behavior of HandleHttpRequest to no longer add > http.param.* attributes for all parts of a multipart request. This was nice > for scenarios when a single file + form data was being sent in a multipart as > the form data was readily accessible as FlowFile attributes. > > A nice approach suggested by ottobackwards [in > Slack|https://apachenifi.slack.com/archives/C0L9VCD47/p1598295952058200] is > to allow a list of form fields to be specified to the HandleHttpRequest > processor as a property. I would imagine this working similarly to how > LogAttribute specifies Attributes to Log, with a simple comma-separated list. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7761) Allow HandleHttpRequest to add specified form data to FlowFile attributes
[ https://issues.apache.org/jira/browse/NIFI-7761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17191462#comment-17191462 ] Otto Fowler commented on NIFI-7761: --- PR is up with implementation as mentioned. > Allow HandleHttpRequest to add specified form data to FlowFile attributes > - > > Key: NIFI-7761 > URL: https://issues.apache.org/jira/browse/NIFI-7761 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.12.0 >Reporter: Matt Gregory >Assignee: Otto Fowler >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > NIFI-7420 changed the behavior of HandleHttpRequest to no longer add > http.param.* attributes for all parts of a multipart request. This was nice > for scenarios when a single file + form data was being sent in a multipart as > the form data was readily accessible as FlowFile attributes. > > A nice approach suggested by ottobackwards [in > Slack|https://apachenifi.slack.com/archives/C0L9VCD47/p1598295952058200] is > to allow a list of form fields to be specified to the HandleHttpRequest > processor as a property. I would imagine this working similarly to how > LogAttribute specifies Attributes to Log, with a simple comma-separated list. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7588) InvokeHTTP ignoring custom parameters when stop+finalize+start
[ https://issues.apache.org/jira/browse/NIFI-7588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17191148#comment-17191148 ] Otto Fowler commented on NIFI-7588: --- https://palindromicity.blogspot.com/2020/04/sending-multipart-form-data-with.html > InvokeHTTP ignoring custom parameters when stop+finalize+start > -- > > Key: NIFI-7588 > URL: https://issues.apache.org/jira/browse/NIFI-7588 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.11.4 > Environment: Amazon Linux >Reporter: Alejandro Fiel Martínez >Priority: Major > Attachments: invokeHTTP_NiFi_bug.png > > > I have an InvokeHTTP processor, with 3 custom paramenters to be passed as > headers, If I add an SSL Context Service and remove it, the processor stop > using those 3 paramenters and I have to delete and recreate then, they are > there, but I see in DEBUG that there are not used in the GET request. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7588) InvokeHTTP ignoring custom parameters when stop+finalize+start
[ https://issues.apache.org/jira/browse/NIFI-7588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17191147#comment-17191147 ] Otto Fowler commented on NIFI-7588: --- [~alejandrofm], this blog post I did has the steps to setup invoke http to handlehttprequest inside of nifi. Can you reproduce your issue using this as a guide ( to your case ) and export it as a template and attache that template to the issue? Along with steps to reproduce? > InvokeHTTP ignoring custom parameters when stop+finalize+start > -- > > Key: NIFI-7588 > URL: https://issues.apache.org/jira/browse/NIFI-7588 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.11.4 > Environment: Amazon Linux >Reporter: Alejandro Fiel Martínez >Priority: Major > Attachments: invokeHTTP_NiFi_bug.png > > > I have an InvokeHTTP processor, with 3 custom paramenters to be passed as > headers, If I add an SSL Context Service and remove it, the processor stop > using those 3 paramenters and I have to delete and recreate then, they are > there, but I see in DEBUG that there are not used in the GET request. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7588) InvokeHTTP ignoring custom parameters when stop+finalize+start
[ https://issues.apache.org/jira/browse/NIFI-7588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190938#comment-17190938 ] Otto Fowler commented on NIFI-7588: --- can you create and attach a template that reproduces the issue? When you say parameters do you mean dynamic properties or the new parameters feature? > InvokeHTTP ignoring custom parameters when stop+finalize+start > -- > > Key: NIFI-7588 > URL: https://issues.apache.org/jira/browse/NIFI-7588 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.11.4 > Environment: Amazon Linux >Reporter: Alejandro Fiel Martínez >Priority: Major > Attachments: invokeHTTP_NiFi_bug.png > > > I have an InvokeHTTP processor, with 3 custom paramenters to be passed as > headers, If I add an SSL Context Service and remove it, the processor stop > using those 3 paramenters and I have to delete and recreate then, they are > there, but I see in DEBUG that there are not used in the GET request. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7781) Bad nar crashes Jira
[ https://issues.apache.org/jira/browse/NIFI-7781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188530#comment-17188530 ] Otto Fowler commented on NIFI-7781: --- As per the comment this is intentional and by design. > Bad nar crashes Jira > > > Key: NIFI-7781 > URL: https://issues.apache.org/jira/browse/NIFI-7781 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.11.4 > Environment: Linux >Reporter: Joseph Kesselman >Priority: Major > Labels: exception-handling > Original Estimate: 24h > Remaining Estimate: 24h > > No matter what's broken about it, a bad nar should not be able to keep the > rest of NiFi from running. Catch and handle this exception. And preferably > give us more information about what the problem was than "unable to load" – > was file not found, not a valid nar, in conflict with other classes, written > on a Thursday... > > 2020-08-31 23:58:20,960 ERROR [main] -> org.apache.nifi.NiFi -> Failure to > launch NiFi due to java.lang.IllegalStateException: Unable to load NAR with co > java.lang.IllegalStateException: Unable to load NAR with coordinates > com.ibm.rtwa:nifi-random-nar:1.0-SNAPSHOT and working directory > ./work/nar/extension > at org.apache.nifi.nar.NarClassLoaders.load(NarClassLoaders.java:181) > at org.apache.nifi.nar.NarClassLoaders.init(NarClassLoaders.java:119) > at org.apache.nifi.NiFi.(NiFi.java:134) > at org.apache.nifi.NiFi.(NiFi.java:72) > at org.apache.nifi.NiFi.main(NiFi.java:301) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7781) Bad nar crashes Jira
[
https://issues.apache.org/jira/browse/NIFI-7781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188526#comment-17188526
]
Otto Fowler commented on NIFI-7781:
---
That doesn't seem to be the whole message :
{code:java}
// prevent the application from starting when there are two NARs with same
group, id, and version
final String narCoordinate =
narDetail.getCoordinate().getCoordinate();
if (narCoordinatesToWorkingDir.containsKey(narCoordinate)) {
final String existingNarWorkingDir =
narCoordinatesToWorkingDir.get(narCoordinate);
throw new IllegalStateException("Unable to load NAR with
coordinates " + narCoordinate
+ " and working directory " +
narDetail.getWorkingDirectory()
+ " because another NAR with the same coordinates
already exists at " + existingNarWorkingDir);
}
{code}
> Bad nar crashes Jira
>
>
> Key: NIFI-7781
> URL: https://issues.apache.org/jira/browse/NIFI-7781
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.11.4
> Environment: Linux
>Reporter: Joseph Kesselman
>Priority: Major
> Labels: exception-handling
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> No matter what's broken about it, a bad nar should not be able to keep the
> rest of NiFi from running. Catch and handle this exception. And preferably
> give us more information about what the problem was than "unable to load" –
> was file not found, not a valid nar, in conflict with other classes, written
> on a Thursday...
>
> 2020-08-31 23:58:20,960 ERROR [main] -> org.apache.nifi.NiFi -> Failure to
> launch NiFi due to java.lang.IllegalStateException: Unable to load NAR with co
> java.lang.IllegalStateException: Unable to load NAR with coordinates
> com.ibm.rtwa:nifi-random-nar:1.0-SNAPSHOT and working directory
> ./work/nar/extension
> at org.apache.nifi.nar.NarClassLoaders.load(NarClassLoaders.java:181)
> at org.apache.nifi.nar.NarClassLoaders.init(NarClassLoaders.java:119)
> at org.apache.nifi.NiFi.(NiFi.java:134)
> at org.apache.nifi.NiFi.(NiFi.java:72)
> at org.apache.nifi.NiFi.main(NiFi.java:301)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-7766) Getting initialize class org.apache.nifi.util.text.RegexDateTimeMatcher error in jsontreereader
[
https://issues.apache.org/jira/browse/NIFI-7766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Otto Fowler updated NIFI-7766:
--
Component/s: (was: Extensions)
Core Framework
> Getting initialize class org.apache.nifi.util.text.RegexDateTimeMatcher error
> in jsontreereader
> ---
>
> Key: NIFI-7766
> URL: https://issues.apache.org/jira/browse/NIFI-7766
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.12.0
>Reporter: Lucas Read
>Assignee: Otto Fowler
>Priority: Minor
> Attachments: nifi-app.log
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Setting the date formats in the jsontreereader causes an error. Server time
> was set for UTC and has since been changed to ETC but the issue still
> persists.
> {code:java}
> Could not initialize class
> org.apache.nifi.util.text.RegexDateTimeMatcher$Compiler
> {code}
> {code:java}
> 0174-1000-e2c7-552f72fd9f08], versionedComponentId=null,
> processGroup=StandardProcessGroup[identifier=20fdafa4-0174-1000-7885-3f4547c81e71,name=test],
> active=true] Failed to invoke @OnEnabled method due to
> java.lang.ExceptionInInitializerError: {}
> 1098 java.lang.ExceptionInInitializerError: null
> 1099 at
> org.apache.nifi.util.text.DateTimeMatcherCompiler.compile(DateTimeMatcherCompiler.java:25)
> 1100 at
> org.apache.nifi.util.text.DateTimeMatcher.compile(DateTimeMatcher.java:66)
> 1101 at
> org.apache.nifi.schema.inference.TimeValueInference.(TimeValueInference.java:39)
> 1102 at
> org.apache.nifi.json.JsonTreeReader.lambda$getSchemaAccessStrategy$1(JsonTreeReader.java:96)
> 1103 at
> org.apache.nifi.schema.inference.SchemaInferenceUtil.getSchemaAccessStrategy(SchemaInferenceUtil.java:47)
> 1104 at
> org.apache.nifi.json.JsonTreeReader.getSchemaAccessStrategy(JsonTreeReader.java:98)
> 1105 at
> org.apache.nifi.serialization.SchemaRegistryService.storeSchemaAccessStrategy(SchemaRegistryService.java:108)
> 1106 at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 1107 at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 1108 at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 1109 at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> 1110 at
> org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:142)
> at
> org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:130)
> 1112 at
> org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:75)
> 1113 at
> org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:52)
> 1114 at
> org.apache.nifi.controller.service.StandardControllerServiceNode$2.run(StandardControllerServiceNode.java:432)
> 1115 at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> 1116 at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> 1117 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> 1118 at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> 1119 at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> 1120 at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> 1121 at java.base/java.lang.Thread.run(Thread.java:834)
> 1122 Caused by: java.lang.NullPointerException: null
> 1123 at java.base/java.util.regex.Pattern.quote(Pattern.java:1352)
> 1124 at
> org.apache.nifi.util.text.RegexDateTimeMatcher$Compiler.(RegexDateTimeMatcher.java:134)
> 1125 ... 23 common frames omitted
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-7766) Getting initialize class org.apache.nifi.util.text.RegexDateTimeMatcher error in jsontreereader
[
https://issues.apache.org/jira/browse/NIFI-7766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Otto Fowler updated NIFI-7766:
--
Status: Patch Available (was: In Progress)
> Getting initialize class org.apache.nifi.util.text.RegexDateTimeMatcher error
> in jsontreereader
> ---
>
> Key: NIFI-7766
> URL: https://issues.apache.org/jira/browse/NIFI-7766
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.12.0
>Reporter: Lucas Read
>Assignee: Otto Fowler
>Priority: Minor
> Attachments: nifi-app.log
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Setting the date formats in the jsontreereader causes an error. Server time
> was set for UTC and has since been changed to ETC but the issue still
> persists.
> {code:java}
> Could not initialize class
> org.apache.nifi.util.text.RegexDateTimeMatcher$Compiler
> {code}
> {code:java}
> 0174-1000-e2c7-552f72fd9f08], versionedComponentId=null,
> processGroup=StandardProcessGroup[identifier=20fdafa4-0174-1000-7885-3f4547c81e71,name=test],
> active=true] Failed to invoke @OnEnabled method due to
> java.lang.ExceptionInInitializerError: {}
> 1098 java.lang.ExceptionInInitializerError: null
> 1099 at
> org.apache.nifi.util.text.DateTimeMatcherCompiler.compile(DateTimeMatcherCompiler.java:25)
> 1100 at
> org.apache.nifi.util.text.DateTimeMatcher.compile(DateTimeMatcher.java:66)
> 1101 at
> org.apache.nifi.schema.inference.TimeValueInference.(TimeValueInference.java:39)
> 1102 at
> org.apache.nifi.json.JsonTreeReader.lambda$getSchemaAccessStrategy$1(JsonTreeReader.java:96)
> 1103 at
> org.apache.nifi.schema.inference.SchemaInferenceUtil.getSchemaAccessStrategy(SchemaInferenceUtil.java:47)
> 1104 at
> org.apache.nifi.json.JsonTreeReader.getSchemaAccessStrategy(JsonTreeReader.java:98)
> 1105 at
> org.apache.nifi.serialization.SchemaRegistryService.storeSchemaAccessStrategy(SchemaRegistryService.java:108)
> 1106 at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 1107 at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 1108 at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 1109 at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> 1110 at
> org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:142)
> at
> org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:130)
> 1112 at
> org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:75)
> 1113 at
> org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:52)
> 1114 at
> org.apache.nifi.controller.service.StandardControllerServiceNode$2.run(StandardControllerServiceNode.java:432)
> 1115 at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> 1116 at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> 1117 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> 1118 at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> 1119 at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> 1120 at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> 1121 at java.base/java.lang.Thread.run(Thread.java:834)
> 1122 Caused by: java.lang.NullPointerException: null
> 1123 at java.base/java.util.regex.Pattern.quote(Pattern.java:1352)
> 1124 at
> org.apache.nifi.util.text.RegexDateTimeMatcher$Compiler.(RegexDateTimeMatcher.java:134)
> 1125 ... 23 common frames omitted
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7766) Getting initialize class org.apache.nifi.util.text.RegexDateTimeMatcher error in jsontreereader
[
https://issues.apache.org/jira/browse/NIFI-7766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184665#comment-17184665
]
Otto Fowler commented on NIFI-7766:
---
>From the documentation:
"The value returned is a two-dimensional array of strings of size n by m, where
m is at least 5. Each of the n rows is an entry containing the localized names
for a single TimeZone. Each such row contains (with i ranging from 0..n-1):
zoneStrings[i][0] - time zone ID
zoneStrings[i][1] - long name of zone in standard time
zoneStrings[i][2] - short name of zone in standard time
zoneStrings[i][3] - long name of zone in daylight saving time
zoneStrings[i][4] - short name of zone in daylight saving time
The zone ID is not localized; it’s one of the valid IDs of the TimeZone class
that are not custom IDs. All other entries are localized names. If a zone does
not implement daylight saving time, the daylight saving time names should not
be used."
We are looping through all the names, without checking for null, that is
incorrect in principle.
Will add null checks
> Getting initialize class org.apache.nifi.util.text.RegexDateTimeMatcher error
> in jsontreereader
> ---
>
> Key: NIFI-7766
> URL: https://issues.apache.org/jira/browse/NIFI-7766
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.12.0
>Reporter: Lucas Read
>Assignee: Otto Fowler
>Priority: Minor
> Attachments: nifi-app.log
>
>
> Setting the date formats in the jsontreereader causes an error. Server time
> was set for UTC and has since been changed to ETC but the issue still
> persists.
> {code:java}
> Could not initialize class
> org.apache.nifi.util.text.RegexDateTimeMatcher$Compiler
> {code}
> {code:java}
> 0174-1000-e2c7-552f72fd9f08], versionedComponentId=null,
> processGroup=StandardProcessGroup[identifier=20fdafa4-0174-1000-7885-3f4547c81e71,name=test],
> active=true] Failed to invoke @OnEnabled method due to
> java.lang.ExceptionInInitializerError: {}
> 1098 java.lang.ExceptionInInitializerError: null
> 1099 at
> org.apache.nifi.util.text.DateTimeMatcherCompiler.compile(DateTimeMatcherCompiler.java:25)
> 1100 at
> org.apache.nifi.util.text.DateTimeMatcher.compile(DateTimeMatcher.java:66)
> 1101 at
> org.apache.nifi.schema.inference.TimeValueInference.(TimeValueInference.java:39)
> 1102 at
> org.apache.nifi.json.JsonTreeReader.lambda$getSchemaAccessStrategy$1(JsonTreeReader.java:96)
> 1103 at
> org.apache.nifi.schema.inference.SchemaInferenceUtil.getSchemaAccessStrategy(SchemaInferenceUtil.java:47)
> 1104 at
> org.apache.nifi.json.JsonTreeReader.getSchemaAccessStrategy(JsonTreeReader.java:98)
> 1105 at
> org.apache.nifi.serialization.SchemaRegistryService.storeSchemaAccessStrategy(SchemaRegistryService.java:108)
> 1106 at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 1107 at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 1108 at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 1109 at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> 1110 at
> org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:142)
> at
> org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:130)
> 1112 at
> org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:75)
> 1113 at
> org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:52)
> 1114 at
> org.apache.nifi.controller.service.StandardControllerServiceNode$2.run(StandardControllerServiceNode.java:432)
> 1115 at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> 1116 at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> 1117 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> 1118 at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> 1119 at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> 1120 at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> 1121 at java.base/java.lang.Thread.run(Thread.java:834)
> 1122 Caused by: java.lang.NullPointerException: null
> 1123 at java.base/java.util.regex.Pattern.quote(Pattern.java:1352)
> 1124 at
> org.apache.nifi.util.text.RegexDateTimeMatcher$Compiler.(RegexDateTimeMatcher.java:134)
> 1125 ... 23 common frames omitted
> {code}
--
This message was sent by
[jira] [Assigned] (NIFI-7766) Getting initialize class org.apache.nifi.util.text.RegexDateTimeMatcher error in jsontreereader
[
https://issues.apache.org/jira/browse/NIFI-7766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Otto Fowler reassigned NIFI-7766:
-
Assignee: Otto Fowler
> Getting initialize class org.apache.nifi.util.text.RegexDateTimeMatcher error
> in jsontreereader
> ---
>
> Key: NIFI-7766
> URL: https://issues.apache.org/jira/browse/NIFI-7766
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.12.0
>Reporter: Lucas Read
>Assignee: Otto Fowler
>Priority: Minor
> Attachments: nifi-app.log
>
>
> Setting the date formats in the jsontreereader causes an error. Server time
> was set for UTC and has since been changed to ETC but the issue still
> persists.
> {code:java}
> Could not initialize class
> org.apache.nifi.util.text.RegexDateTimeMatcher$Compiler
> {code}
> {code:java}
> 0174-1000-e2c7-552f72fd9f08], versionedComponentId=null,
> processGroup=StandardProcessGroup[identifier=20fdafa4-0174-1000-7885-3f4547c81e71,name=test],
> active=true] Failed to invoke @OnEnabled method due to
> java.lang.ExceptionInInitializerError: {}
> 1098 java.lang.ExceptionInInitializerError: null
> 1099 at
> org.apache.nifi.util.text.DateTimeMatcherCompiler.compile(DateTimeMatcherCompiler.java:25)
> 1100 at
> org.apache.nifi.util.text.DateTimeMatcher.compile(DateTimeMatcher.java:66)
> 1101 at
> org.apache.nifi.schema.inference.TimeValueInference.(TimeValueInference.java:39)
> 1102 at
> org.apache.nifi.json.JsonTreeReader.lambda$getSchemaAccessStrategy$1(JsonTreeReader.java:96)
> 1103 at
> org.apache.nifi.schema.inference.SchemaInferenceUtil.getSchemaAccessStrategy(SchemaInferenceUtil.java:47)
> 1104 at
> org.apache.nifi.json.JsonTreeReader.getSchemaAccessStrategy(JsonTreeReader.java:98)
> 1105 at
> org.apache.nifi.serialization.SchemaRegistryService.storeSchemaAccessStrategy(SchemaRegistryService.java:108)
> 1106 at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 1107 at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 1108 at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 1109 at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> 1110 at
> org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:142)
> at
> org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:130)
> 1112 at
> org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:75)
> 1113 at
> org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:52)
> 1114 at
> org.apache.nifi.controller.service.StandardControllerServiceNode$2.run(StandardControllerServiceNode.java:432)
> 1115 at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> 1116 at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> 1117 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> 1118 at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> 1119 at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> 1120 at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> 1121 at java.base/java.lang.Thread.run(Thread.java:834)
> 1122 Caused by: java.lang.NullPointerException: null
> 1123 at java.base/java.util.regex.Pattern.quote(Pattern.java:1352)
> 1124 at
> org.apache.nifi.util.text.RegexDateTimeMatcher$Compiler.(RegexDateTimeMatcher.java:134)
> 1125 ... 23 common frames omitted
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (NIFI-7761) Allow HandleHttpRequest to add specified form data to FlowFile attributes
[ https://issues.apache.org/jira/browse/NIFI-7761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otto Fowler reassigned NIFI-7761: - Assignee: Otto Fowler > Allow HandleHttpRequest to add specified form data to FlowFile attributes > - > > Key: NIFI-7761 > URL: https://issues.apache.org/jira/browse/NIFI-7761 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.12.0 >Reporter: Matt Gregory >Assignee: Otto Fowler >Priority: Major > > NIFI-7420 changed the behavior of HandleHttpRequest to no longer add > http.param.* attributes for all parts of a multipart request. This was nice > for scenarios when a single file + form data was being sent in a multipart as > the form data was readily accessible as FlowFile attributes. > > A nice approach suggested by ottobackwards [in > Slack|https://apachenifi.slack.com/archives/C0L9VCD47/p1598295952058200] is > to allow a list of form fields to be specified to the HandleHttpRequest > processor as a property. I would imagine this working similarly to how > LogAttribute specifies Attributes to Log, with a simple comma-separated list. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-7747) Create Expression Editor for expression eligible Properties and stand alone
Otto Fowler created NIFI-7747: - Summary: Create Expression Editor for expression eligible Properties and stand alone Key: NIFI-7747 URL: https://issues.apache.org/jira/browse/NIFI-7747 Project: Apache NiFi Issue Type: New Feature Components: Core UI Reporter: Otto Fowler Attachments: Screen Shot 2020-08-17 at 10.09.01.png Writing and testing expressions is not a great experience for Nifi users. In the UI for a processor configuration it is basically typing text in a small text box !Screen Shot 2020-08-17 at 10.09.01.png! Nifi should have a ui for authoring expression language statements. * This UI should be available when editing a property that supports EL * This UI should be available stand alone from the main UI menu * This property should 'discover' and display the referenced variables /attribute names and allow setting values for them * This UI should allow execution of the statement, with the variables/attributes/values as configured in the UI to test the results * -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7718) create NiFi sub projects to host NiFi REST client in different languages
[
https://issues.apache.org/jira/browse/NIFI-7718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17173675#comment-17173675
]
Otto Fowler commented on NIFI-7718:
---
So, the idea would be that nifi provides a core set of libraries that these
project use instead of building themselves?
> create NiFi sub projects to host NiFi REST client in different languages
>
>
> Key: NIFI-7718
> URL: https://issues.apache.org/jira/browse/NIFI-7718
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Tools and Build
>Reporter: Simon Weng
>Priority: Minor
>
> We're seeing the need of NiFI REST clients in different languages, such as
> Python, Go, etc, so that software can be created to control and manage NiFi
> cluster and flows.
> Since the RESTful API is documented in OpenAPI spec v2, a client SDK can be
> generated via
> [openapi-generator|https://github.com/OpenAPITools/openapi-generator]
> Individual effort is seen from the community, such as:
> * [https://github.com/erdrix/nigoapi]
> * [https://github.com/simingweng/nifi-go-client]
> * [https://github.com/Chaffelson/nipyapi]
> It would be beneficial to the community to consolidate the effort and
> centrally maintain the Client SDK effort for everybody to use.
> Just like {{minifi}} being a sub project of NiFi, we can create sub project
> for different language bindings, such as:
> * apache/nifi-clients
> * apache/nifi-client-go
> A single repo like {{nifi-clients}} can house all the languages that does not
> requires separate repo for publishing, {{nifi-client-go}} is an exception
> because Go Module couples with its own repo.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-2072) Support named captures in ExtractText
[ https://issues.apache.org/jira/browse/NIFI-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163925#comment-17163925 ] Otto Fowler commented on NIFI-2072: --- The PR is up for review. The next step is that somebody reviews it. And if that person is a committer then they can +1 it and merge it. [~pvillard] is pretty busy. You are welcome to review and try etc. If that is in the realm of things you are comfortable doing > Support named captures in ExtractText > - > > Key: NIFI-2072 > URL: https://issues.apache.org/jira/browse/NIFI-2072 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Otto Fowler >Priority: Major > Labels: extracttext > > ExtractText currently captures and creates attributes using numeric indices > (e.g, attribute.name.0, attribute.name.1, etc.) whether or not the capture > groups are named, i.e., patterns like (?\w+). > In addition to being more faithful to the provided regexes, named captures > could help simplify data flows because you wouldn't have to add superfluous > UpdateAttribute steps which are just renaming the indexed captures to more > interpretable names. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-2072) Support named captures in ExtractText
[ https://issues.apache.org/jira/browse/NIFI-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163793#comment-17163793 ] Otto Fowler commented on NIFI-2072: --- [~malthe] I just pushed validation support > Support named captures in ExtractText > - > > Key: NIFI-2072 > URL: https://issues.apache.org/jira/browse/NIFI-2072 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Otto Fowler >Priority: Major > Labels: extracttext > > ExtractText currently captures and creates attributes using numeric indices > (e.g, attribute.name.0, attribute.name.1, etc.) whether or not the capture > groups are named, i.e., patterns like (?\w+). > In addition to being more faithful to the provided regexes, named captures > could help simplify data flows because you wouldn't have to add superfluous > UpdateAttribute steps which are just renaming the indexed captures to more > interpretable names. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-2072) Support named captures in ExtractText
[ https://issues.apache.org/jira/browse/NIFI-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17153069#comment-17153069 ] Otto Fowler commented on NIFI-2072: --- I'm more inclined to do the validation, since it think handling mixed, and scoped ( nested ) etc goes downhill real fast since java doesn't support it. Would would be nice is when I called group(string) i could also call getGroupIndex(string) so that I could mix and match, but you can't. > Support named captures in ExtractText > - > > Key: NIFI-2072 > URL: https://issues.apache.org/jira/browse/NIFI-2072 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Otto Fowler >Priority: Major > Labels: extracttext > > ExtractText currently captures and creates attributes using numeric indices > (e.g, attribute.name.0, attribute.name.1, etc.) whether or not the capture > groups are named, i.e., patterns like (?\w+). > In addition to being more faithful to the provided regexes, named captures > could help simplify data flows because you wouldn't have to add superfluous > UpdateAttribute steps which are just renaming the indexed captures to more > interpretable names. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-2072) Support named captures in ExtractText
[ https://issues.apache.org/jira/browse/NIFI-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152081#comment-17152081 ] Otto Fowler commented on NIFI-2072: --- That is a good question. Usually (in my experience) when a behavior of a processor is changed it is put behind a configuration property, so that is the convention I followed. [~pvillard] did this as well. > Support named captures in ExtractText > - > > Key: NIFI-2072 > URL: https://issues.apache.org/jira/browse/NIFI-2072 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Otto Fowler >Priority: Major > Labels: extracttext > > ExtractText currently captures and creates attributes using numeric indices > (e.g, attribute.name.0, attribute.name.1, etc.) whether or not the capture > groups are named, i.e., patterns like (?\w+). > In addition to being more faithful to the provided regexes, named captures > could help simplify data flows because you wouldn't have to add superfluous > UpdateAttribute steps which are just renaming the indexed captures to more > interpretable names. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-2072) Support named captures in ExtractText
[ https://issues.apache.org/jira/browse/NIFI-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otto Fowler updated NIFI-2072: -- Labels: extracttext (was: ) > Support named captures in ExtractText > - > > Key: NIFI-2072 > URL: https://issues.apache.org/jira/browse/NIFI-2072 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Otto Fowler >Priority: Major > Labels: extracttext > > ExtractText currently captures and creates attributes using numeric indices > (e.g, attribute.name.0, attribute.name.1, etc.) whether or not the capture > groups are named, i.e., patterns like (?\w+). > In addition to being more faithful to the provided regexes, named captures > could help simplify data flows because you wouldn't have to add superfluous > UpdateAttribute steps which are just renaming the indexed captures to more > interpretable names. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-2072) Support named captures in ExtractText
[ https://issues.apache.org/jira/browse/NIFI-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otto Fowler updated NIFI-2072: -- Status: Patch Available (was: In Progress) [l#4348|https://github.com/apache/nifi/pull/4384/] > Support named captures in ExtractText > - > > Key: NIFI-2072 > URL: https://issues.apache.org/jira/browse/NIFI-2072 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Otto Fowler >Priority: Major > > ExtractText currently captures and creates attributes using numeric indices > (e.g, attribute.name.0, attribute.name.1, etc.) whether or not the capture > groups are named, i.e., patterns like (?\w+). > In addition to being more faithful to the provided regexes, named captures > could help simplify data flows because you wouldn't have to add superfluous > UpdateAttribute steps which are just renaming the indexed captures to more > interpretable names. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-2072) Support named captures in ExtractText
[
https://issues.apache.org/jira/browse/NIFI-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17151064#comment-17151064
]
Otto Fowler commented on NIFI-2072:
---
OK
I have a PR just about ready for this. But just to get some feedback first:
After the PR there implicitly two ways the processor works based on the enable
named groups property.
The old way if it is not enabled.
The new way.
The new way is different in that numeric indices are not added until the second
set of matches ( if you have that enabled).
The root attribute name is used for the 0 group -or- the whole match line if
there are no groups specified.
such as :
{code:java}
@Test
public void testFindAll() throws Exception {
final TestRunner testRunner = TestRunners.newTestRunner(new
ExtractText());
testRunner.setProperty(ENABLE_NAMED_GROUPS, "true");
testRunner.setProperty(ExtractText.ENABLE_REPEATING_CAPTURE_GROUP,
"true");
final String attributeKey = "regex.result";
testRunner.setProperty(attributeKey, "(?s)(?\\w+)");
testRunner.enqueue("This is my text".getBytes(StandardCharsets.UTF_8));
testRunner.run();
testRunner.assertAllFlowFilesTransferred(ExtractText.REL_MATCH, 1);
final MockFlowFile out =
testRunner.getFlowFilesForRelationship(ExtractText.REL_MATCH).get(0);
// Ensure the zero capture group is in the resultant attributes
out.assertAttributeExists(attributeKey);
out.assertAttributeExists(attributeKey + ".W");
out.assertAttributeExists(attributeKey + ".W.1");
out.assertAttributeExists(attributeKey + ".W.2");
out.assertAttributeExists(attributeKey + ".W.3");
out.assertAttributeEquals(attributeKey, "This");
out.assertAttributeEquals(attributeKey + ".W", "This");
out.assertAttributeEquals(attributeKey + ".W.1", "is");
out.assertAttributeEquals(attributeKey + ".W.2", "my");
out.assertAttributeEquals(attributeKey + ".W.3", "text");
}
@Test
public void testFindAllPair() throws Exception {
final TestRunner testRunner = TestRunners.newTestRunner(new
ExtractText());
testRunner.setProperty(ENABLE_NAMED_GROUPS, "true");
testRunner.setProperty(ExtractText.ENABLE_REPEATING_CAPTURE_GROUP,
"true");
final String attributeKey = "regex.result";
testRunner.setProperty(attributeKey, "(?\\w+)=(?\\d+)");
testRunner.enqueue("a=1,b=10,c=100".getBytes(StandardCharsets.UTF_8));
testRunner.run();
testRunner.assertAllFlowFilesTransferred(ExtractText.REL_MATCH, 1);
final MockFlowFile out =
testRunner.getFlowFilesForRelationship(ExtractText.REL_MATCH).get(0);
// Ensure the zero capture group is in the resultant attributes
out.assertAttributeExists(attributeKey);
out.assertAttributeExists(attributeKey + ".LEFT");
out.assertAttributeExists(attributeKey + ".RIGHT");
out.assertAttributeExists(attributeKey + ".LEFT.1");
out.assertAttributeExists(attributeKey + ".RIGHT.1");
out.assertAttributeExists(attributeKey + ".LEFT.2");
out.assertAttributeExists(attributeKey + ".RIGHT.2");
out.assertAttributeNotExists(attributeKey + ".LEFT.3"); // Ensure
there's no more attributes
out.assertAttributeNotExists(attributeKey + ".RIGHT.3"); // Ensure
there's no more attributes
out.assertAttributeEquals(attributeKey , "a=1");
out.assertAttributeEquals(attributeKey + ".LEFT", "a");
out.assertAttributeEquals(attributeKey + ".RIGHT", "1");
out.assertAttributeEquals(attributeKey + ".LEFT.1", "b");
out.assertAttributeEquals(attributeKey + ".RIGHT.1", "10");
out.assertAttributeEquals(attributeKey + ".LEFT.2", "c");
out.assertAttributeEquals(attributeKey + ".RIGHT.2", "100");
}
{code}
> Support named captures in ExtractText
> -
>
> Key: NIFI-2072
> URL: https://issues.apache.org/jira/browse/NIFI-2072
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joey Frazee
>Assignee: Otto Fowler
>Priority: Major
>
> ExtractText currently captures and creates attributes using numeric indices
> (e.g, attribute.name.0, attribute.name.1, etc.) whether or not the capture
> groups are named, i.e., patterns like (?\w+).
> In addition to being more faithful to the provided regexes, named captures
> could help simplify data flows because you wouldn't have to add superfluous
> UpdateAttribute steps which are just renaming the indexed captures to more
> interpretable names.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-2072) Support named captures in ExtractText
[ https://issues.apache.org/jira/browse/NIFI-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17151065#comment-17151065 ] Otto Fowler commented on NIFI-2072: --- [~pvillard] > Support named captures in ExtractText > - > > Key: NIFI-2072 > URL: https://issues.apache.org/jira/browse/NIFI-2072 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Otto Fowler >Priority: Major > > ExtractText currently captures and creates attributes using numeric indices > (e.g, attribute.name.0, attribute.name.1, etc.) whether or not the capture > groups are named, i.e., patterns like (?\w+). > In addition to being more faithful to the provided regexes, named captures > could help simplify data flows because you wouldn't have to add superfluous > UpdateAttribute steps which are just renaming the indexed captures to more > interpretable names. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (NIFI-2072) Support named captures in ExtractText
[
https://issues.apache.org/jira/browse/NIFI-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17148956#comment-17148956
]
Otto Fowler edited comment on NIFI-2072 at 6/30/20, 9:21 PM:
-
[~pvillard]
Something like this? The restriction on the property to enable is: if you
want name groups, all your capturing groups MUST be named. You can't mix named
and unnamed captures. If they don't match, it falls back to the old way.
But I haven't written the verify yet either
{code:java}
final String SAMPLE_STRING =
"foo\r\nbar1\r\nbar2\r\nbar3\r\nhello\r\nworld\r\n";
@Test
public void testProcessorWithGroupNames() throws Exception {
final TestRunner testRunner = TestRunners.newTestRunner(new
ExtractText());
testRunner.setProperty("regex.result1", "(?s)(?.*)");
testRunner.setProperty("regex.result2", "(?s).*(?bar1).*");
testRunner.setProperty("regex.result3", "(?s).*?(?bar\\d).*");
testRunner.setProperty("regex.result4",
"(?s).*?(?:bar\\d).*?(?bar\\d).*?(?bar3).*");
testRunner.setProperty("regex.result5", "(?s).*(?bar\\d).*");
testRunner.setProperty("regex.result6", "(?s)^(?.*)$");
testRunner.setProperty("regex.result7", "(?s)(?XXX)");
testRunner.setProperty(ENABLE_NAMED_GROUPS, "true");
testRunner.enqueue(SAMPLE_STRING.getBytes("UTF-8"));
testRunner.run();
testRunner.assertAllFlowFilesTransferred(ExtractText.REL_MATCH, 1);
final MockFlowFile out =
testRunner.getFlowFilesForRelationship(ExtractText.REL_MATCH).get(0);
java.util.Map attributes = out.getAttributes();
out.assertAttributeEquals("regex.result1.all", SAMPLE_STRING);
out.assertAttributeEquals("regex.result2.bar1", "bar1");
out.assertAttributeEquals("regex.result3.bar1", "bar1");
out.assertAttributeEquals("regex.result4.bar2", "bar2");
out.assertAttributeEquals("regex.result4.bar2", "bar2");
out.assertAttributeEquals("regex.result4.bar3", "bar3");
out.assertAttributeEquals("regex.result5.bar3", "bar3");
out.assertAttributeEquals("regex.result6.all", SAMPLE_STRING);
out.assertAttributeEquals("regex.result7.miss", null);
}
{code}
was (Author: ottobackwards):
[~pvillard]
Something like this? The restriction on the property to enable is: if you
want name groups, all your capturing groups MUST be named. You can't mix named
and unnamed captures.
{code:java}
final String SAMPLE_STRING =
"foo\r\nbar1\r\nbar2\r\nbar3\r\nhello\r\nworld\r\n";
@Test
public void testProcessorWithGroupNames() throws Exception {
final TestRunner testRunner = TestRunners.newTestRunner(new
ExtractText());
testRunner.setProperty("regex.result1", "(?s)(?.*)");
testRunner.setProperty("regex.result2", "(?s).*(?bar1).*");
testRunner.setProperty("regex.result3", "(?s).*?(?bar\\d).*");
testRunner.setProperty("regex.result4",
"(?s).*?(?:bar\\d).*?(?bar\\d).*?(?bar3).*");
testRunner.setProperty("regex.result5", "(?s).*(?bar\\d).*");
testRunner.setProperty("regex.result6", "(?s)^(?.*)$");
testRunner.setProperty("regex.result7", "(?s)(?XXX)");
testRunner.setProperty(ENABLE_NAMED_GROUPS, "true");
testRunner.enqueue(SAMPLE_STRING.getBytes("UTF-8"));
testRunner.run();
testRunner.assertAllFlowFilesTransferred(ExtractText.REL_MATCH, 1);
final MockFlowFile out =
testRunner.getFlowFilesForRelationship(ExtractText.REL_MATCH).get(0);
java.util.Map attributes = out.getAttributes();
out.assertAttributeEquals("regex.result1.all", SAMPLE_STRING);
out.assertAttributeEquals("regex.result2.bar1", "bar1");
out.assertAttributeEquals("regex.result3.bar1", "bar1");
out.assertAttributeEquals("regex.result4.bar2", "bar2");
out.assertAttributeEquals("regex.result4.bar2", "bar2");
out.assertAttributeEquals("regex.result4.bar3", "bar3");
out.assertAttributeEquals("regex.result5.bar3", "bar3");
out.assertAttributeEquals("regex.result6.all", SAMPLE_STRING);
out.assertAttributeEquals("regex.result7.miss", null);
}
{code}
> Support named captures in ExtractText
> -
>
> Key: NIFI-2072
> URL: https://issues.apache.org/jira/browse/NIFI-2072
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joey Frazee
>Assignee: Otto Fowler
>Priority: Major
>
> ExtractText currently captures and creates attributes using numeric indices
> (e.g, attribute.name.0, attribute.name.1, etc.) whether or not the capture
> groups are named, i.e., patterns like (?\w+).
> In addition to being more faithful to the provided regexes, named captures
> could help simplify data flows because you wouldn't
[jira] [Commented] (NIFI-2072) Support named captures in ExtractText
[
https://issues.apache.org/jira/browse/NIFI-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17148956#comment-17148956
]
Otto Fowler commented on NIFI-2072:
---
[~pvillard]
Something like this? The restriction on the property to enable is: if you
want name groups, all your capturing groups MUST be named. You can't mix named
and unnamed captures.
{code:java}
final String SAMPLE_STRING =
"foo\r\nbar1\r\nbar2\r\nbar3\r\nhello\r\nworld\r\n";
@Test
public void testProcessorWithGroupNames() throws Exception {
final TestRunner testRunner = TestRunners.newTestRunner(new
ExtractText());
testRunner.setProperty("regex.result1", "(?s)(?.*)");
testRunner.setProperty("regex.result2", "(?s).*(?bar1).*");
testRunner.setProperty("regex.result3", "(?s).*?(?bar\\d).*");
testRunner.setProperty("regex.result4",
"(?s).*?(?:bar\\d).*?(?bar\\d).*?(?bar3).*");
testRunner.setProperty("regex.result5", "(?s).*(?bar\\d).*");
testRunner.setProperty("regex.result6", "(?s)^(?.*)$");
testRunner.setProperty("regex.result7", "(?s)(?XXX)");
testRunner.setProperty(ENABLE_NAMED_GROUPS, "true");
testRunner.enqueue(SAMPLE_STRING.getBytes("UTF-8"));
testRunner.run();
testRunner.assertAllFlowFilesTransferred(ExtractText.REL_MATCH, 1);
final MockFlowFile out =
testRunner.getFlowFilesForRelationship(ExtractText.REL_MATCH).get(0);
java.util.Map attributes = out.getAttributes();
out.assertAttributeEquals("regex.result1.all", SAMPLE_STRING);
out.assertAttributeEquals("regex.result2.bar1", "bar1");
out.assertAttributeEquals("regex.result3.bar1", "bar1");
out.assertAttributeEquals("regex.result4.bar2", "bar2");
out.assertAttributeEquals("regex.result4.bar2", "bar2");
out.assertAttributeEquals("regex.result4.bar3", "bar3");
out.assertAttributeEquals("regex.result5.bar3", "bar3");
out.assertAttributeEquals("regex.result6.all", SAMPLE_STRING);
out.assertAttributeEquals("regex.result7.miss", null);
}
{code}
> Support named captures in ExtractText
> -
>
> Key: NIFI-2072
> URL: https://issues.apache.org/jira/browse/NIFI-2072
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joey Frazee
>Assignee: Otto Fowler
>Priority: Major
>
> ExtractText currently captures and creates attributes using numeric indices
> (e.g, attribute.name.0, attribute.name.1, etc.) whether or not the capture
> groups are named, i.e., patterns like (?\w+).
> In addition to being more faithful to the provided regexes, named captures
> could help simplify data flows because you wouldn't have to add superfluous
> UpdateAttribute steps which are just renaming the indexed captures to more
> interpretable names.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-2072) Support named captures in ExtractText
[ https://issues.apache.org/jira/browse/NIFI-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147556#comment-17147556 ] Otto Fowler commented on NIFI-2072: --- I'll take a shot > Support named captures in ExtractText > - > > Key: NIFI-2072 > URL: https://issues.apache.org/jira/browse/NIFI-2072 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Otto Fowler >Priority: Major > > ExtractText currently captures and creates attributes using numeric indices > (e.g, attribute.name.0, attribute.name.1, etc.) whether or not the capture > groups are named, i.e., patterns like (?\w+). > In addition to being more faithful to the provided regexes, named captures > could help simplify data flows because you wouldn't have to add superfluous > UpdateAttribute steps which are just renaming the indexed captures to more > interpretable names. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (NIFI-2072) Support named captures in ExtractText
[ https://issues.apache.org/jira/browse/NIFI-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otto Fowler reassigned NIFI-2072: - Assignee: Otto Fowler > Support named captures in ExtractText > - > > Key: NIFI-2072 > URL: https://issues.apache.org/jira/browse/NIFI-2072 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Otto Fowler >Priority: Major > > ExtractText currently captures and creates attributes using numeric indices > (e.g, attribute.name.0, attribute.name.1, etc.) whether or not the capture > groups are named, i.e., patterns like (?\w+). > In addition to being more faithful to the provided regexes, named captures > could help simplify data flows because you wouldn't have to add superfluous > UpdateAttribute steps which are just renaming the indexed captures to more > interpretable names. -- This message was sent by Atlassian Jira (v8.3.4#803005)
