[jira] [Commented] (NIFI-5327) NetFlow Processors
[ https://issues.apache.org/jira/browse/NIFI-5327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556827#comment-16556827 ] ASF GitHub Bot commented on NIFI-5327: -- Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2820 OK. Contrib check and build passes. Everything runs the way it should. My question is on the data. The src and dest addresses are just the numbers. They are supposed to be ip addresses. I think that formatting them as IP addresses rather than just as numbers would be more usable. Is there something I'm missing? I don't have an example of something else that shows net flow data. Are there any other fields that are output 'raw' that could be formatted? > NetFlow Processors > -- > > Key: NIFI-5327 > URL: https://issues.apache.org/jira/browse/NIFI-5327 > Project: Apache NiFi > Issue Type: New Feature > Components: Core Framework >Affects Versions: 1.6.0 >Reporter: Prashanth Venkatesan >Assignee: Prashanth Venkatesan >Priority: Major > > As network traffic data scopes for the big data use case, would like NiFi to > have processors to support parsing of those protocols. > Netflow is a protocol introduced by Cisco that provides the ability to > collect IP network traffic as it enters or exits an interface and is > described in detail in here: > [https://www.cisco.com/c/en/us/td/docs/net_mgmt/netflow_collection_engine/3-6/user/guide/format.html] > > Currently, I have created the following processor: > *ParseNetflowv5*: Parses the ingress netflowv5 bytes and ingest as either > NiFi flowfile attributes or as a JSON content. This also sends > one-time-template. > > Further ahead, we can add many processor specific to network protocols in > this nar bundle. > I will create a pull request. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2820 OK. Contrib check and build passes. Everything runs the way it should. My question is on the data. The src and dest addresses are just the numbers. They are supposed to be ip addresses. I think that formatting them as IP addresses rather than just as numbers would be more usable. Is there something I'm missing? I don't have an example of something else that shows net flow data. Are there any other fields that are output 'raw' that could be formatted? ---
[jira] [Commented] (NIFI-5327) NetFlow Processors
[ https://issues.apache.org/jira/browse/NIFI-5327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556794#comment-16556794 ] ASF GitHub Bot commented on NIFI-5327: -- Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/nifi/pull/2820#discussion_r205314387 --- Diff: nifi-nar-bundles/nifi-network-bundle/nifi-network-processors/src/main/resources/docs/org.apache.nifi.processors.network.ParseNetflowv5/additionalDetails.html --- @@ -0,0 +1,74 @@ + + + + + +Netflowv5Parser + + + + + + Netflowv5Parser processor parses the ingress netflowv5 datagram format + and transfers it either as flowfile attributes or JSON object. + Netflowv5 format has predefined schema named "template" for parsing + the netflowv5 record. More information:https://www.cisco.com/c/en/us/td/docs/net_mgmt/netflow_collection_engine/3-6/user/guide/format.html;>RFC-netflowv5 + --- End diff -- I am sorry @PrashanthVenkatesan, I've been on vacation the last couple of days. I'm fine with the review, I just want to run everything again. I will try as soon as I can. I am not a committer though, so even with my potential +1 you will still need a committer to sign off and merge. > NetFlow Processors > -- > > Key: NIFI-5327 > URL: https://issues.apache.org/jira/browse/NIFI-5327 > Project: Apache NiFi > Issue Type: New Feature > Components: Core Framework >Affects Versions: 1.6.0 >Reporter: Prashanth Venkatesan >Assignee: Prashanth Venkatesan >Priority: Major > > As network traffic data scopes for the big data use case, would like NiFi to > have processors to support parsing of those protocols. > Netflow is a protocol introduced by Cisco that provides the ability to > collect IP network traffic as it enters or exits an interface and is > described in detail in here: > [https://www.cisco.com/c/en/us/td/docs/net_mgmt/netflow_collection_engine/3-6/user/guide/format.html] > > Currently, I have created the following processor: > *ParseNetflowv5*: Parses the ingress netflowv5 bytes and ingest as either > NiFi flowfile attributes or as a JSON content. This also sends > one-time-template. > > Further ahead, we can add many processor specific to network protocols in > this nar bundle. > I will create a pull request. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/nifi/pull/2820#discussion_r205314387 --- Diff: nifi-nar-bundles/nifi-network-bundle/nifi-network-processors/src/main/resources/docs/org.apache.nifi.processors.network.ParseNetflowv5/additionalDetails.html --- @@ -0,0 +1,74 @@ + + + + + +Netflowv5Parser + + + + + + Netflowv5Parser processor parses the ingress netflowv5 datagram format + and transfers it either as flowfile attributes or JSON object. + Netflowv5 format has predefined schema named "template" for parsing + the netflowv5 record. More information:https://www.cisco.com/c/en/us/td/docs/net_mgmt/netflow_collection_engine/3-6/user/guide/format.html;>RFC-netflowv5 + --- End diff -- I am sorry @PrashanthVenkatesan, I've been on vacation the last couple of days. I'm fine with the review, I just want to run everything again. I will try as soon as I can. I am not a committer though, so even with my potential +1 you will still need a committer to sign off and merge. ---
[jira] [Commented] (NIFI-5327) NetFlow Processors
[ https://issues.apache.org/jira/browse/NIFI-5327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556778#comment-16556778 ] ASF GitHub Bot commented on NIFI-5327: -- Github user PrashanthVenkatesan commented on a diff in the pull request: https://github.com/apache/nifi/pull/2820#discussion_r205312374 --- Diff: nifi-nar-bundles/nifi-network-bundle/nifi-network-processors/src/main/resources/docs/org.apache.nifi.processors.network.ParseNetflowv5/additionalDetails.html --- @@ -0,0 +1,74 @@ + + + + + +Netflowv5Parser + + + + + + Netflowv5Parser processor parses the ingress netflowv5 datagram format + and transfers it either as flowfile attributes or JSON object. + Netflowv5 format has predefined schema named "template" for parsing + the netflowv5 record. More information:https://www.cisco.com/c/en/us/td/docs/net_mgmt/netflow_collection_engine/3-6/user/guide/format.html;>RFC-netflowv5 + --- End diff -- any other changes required?? > NetFlow Processors > -- > > Key: NIFI-5327 > URL: https://issues.apache.org/jira/browse/NIFI-5327 > Project: Apache NiFi > Issue Type: New Feature > Components: Core Framework >Affects Versions: 1.6.0 >Reporter: Prashanth Venkatesan >Assignee: Prashanth Venkatesan >Priority: Major > > As network traffic data scopes for the big data use case, would like NiFi to > have processors to support parsing of those protocols. > Netflow is a protocol introduced by Cisco that provides the ability to > collect IP network traffic as it enters or exits an interface and is > described in detail in here: > [https://www.cisco.com/c/en/us/td/docs/net_mgmt/netflow_collection_engine/3-6/user/guide/format.html] > > Currently, I have created the following processor: > *ParseNetflowv5*: Parses the ingress netflowv5 bytes and ingest as either > NiFi flowfile attributes or as a JSON content. This also sends > one-time-template. > > Further ahead, we can add many processor specific to network protocols in > this nar bundle. > I will create a pull request. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user PrashanthVenkatesan commented on a diff in the pull request: https://github.com/apache/nifi/pull/2820#discussion_r205312374 --- Diff: nifi-nar-bundles/nifi-network-bundle/nifi-network-processors/src/main/resources/docs/org.apache.nifi.processors.network.ParseNetflowv5/additionalDetails.html --- @@ -0,0 +1,74 @@ + + + + + +Netflowv5Parser + + + + + + Netflowv5Parser processor parses the ingress netflowv5 datagram format + and transfers it either as flowfile attributes or JSON object. + Netflowv5 format has predefined schema named "template" for parsing + the netflowv5 record. More information:https://www.cisco.com/c/en/us/td/docs/net_mgmt/netflow_collection_engine/3-6/user/guide/format.html;>RFC-netflowv5 + --- End diff -- any other changes required?? ---
[jira] [Commented] (NIFI-5443) Improve cluster configuration for dynamic scaling
[
https://issues.apache.org/jira/browse/NIFI-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556775#comment-16556775
]
Prashanth Venkatesan commented on NIFI-5443:
[~alopresto] Scaling in running NiFi secure cluster expects empty
authorizers.xml from new node to join the cluster. Say, node-0 & node-1
running as secure nifi cluster. I modified policies and flow in that cluster.
Now, if new fresh node-3 to join the cluster, we need to pass *empty
authorizers.xml.* Is there any other elegant way of handling this because we
need to handle this in docker container?
> Improve cluster configuration for dynamic scaling
> -
>
> Key: NIFI-5443
> URL: https://issues.apache.org/jira/browse/NIFI-5443
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Andy LoPresto
>Priority: Critical
> Labels: cluster, docker, kubernetes, rkt, scale, security
>
> Currently, NiFi is designed for static clusters, with frequent references in
> configuration files to a priori knowledge of node hostnames, ports, etc.
> Efforts should be taken to make NiFi easier to dynamically scale. This can
> involve containerization improvements via Docker/rkt, deployment improvements
> via Kubernetes, and abstraction of the configuration values needed to stand
> up the cluster. A node should be able to join the cluster, and, given the
> correct keystore and truststore, immediately communicate with other existing
> nodes in the cluster without requiring direct configuration changes to them,
> or a restart of any node.
> * {{authorizers.xml}}
> * node identities
> * permissions ({{RW}} on {{/proxy}})
> * ZooKeeper configuration
> * etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (NIFI-5463) Add a "durationNanos" field to provenance events
Matt Burgess created NIFI-5463: -- Summary: Add a "durationNanos" field to provenance events Key: NIFI-5463 URL: https://issues.apache.org/jira/browse/NIFI-5463 Project: Apache NiFi Issue Type: New Feature Components: Core Framework Reporter: Matt Burgess This Jira proposes to add a "durationNanos" field to provenance events, possibly replacing "durationMillis" in NiFi 2.0, to give better precision for how long a FlowFile spends in a particular processor. This may depend on and/or build off NIFI-5420 to have the ProcessSession keep track of the duration of a FlowFile spent in its session. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (NIFI-5457) apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0
[ https://issues.apache.org/jira/browse/NIFI-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall resolved NIFI-5457. Resolution: Fixed Not a problem, thanks for the assist [~aldrin] > apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0 > - > > Key: NIFI-5457 > URL: https://issues.apache.org/jira/browse/NIFI-5457 > Project: Apache NiFi > Issue Type: Task >Reporter: Joseph Percivall >Assignee: Joseph Percivall >Priority: Major > > Currently, on DockerHub the tagged image for Apache NiFi 1.7.1 instead pulls > 1.7.0. This can be verified by looking at the DockerFile for the tagged > commit[1], looking at the pushed DockerFile[2], and verifying locally by > running "docker run apache/nifi:1.7.1" and looking at the logs. > > > [1] > [https://github.com/apache/nifi/blob/rel/nifi-1.7.1/nifi-docker/dockerhub/Dockerfile#L25] > [2] [https://hub.docker.com/r/apache/nifi/~/dockerfile/] > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5457) apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0
[ https://issues.apache.org/jira/browse/NIFI-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556426#comment-16556426 ] Aldrin Piri commented on NIFI-5457: --- [~JPercivall] Looks great, thanks much! > apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0 > - > > Key: NIFI-5457 > URL: https://issues.apache.org/jira/browse/NIFI-5457 > Project: Apache NiFi > Issue Type: Task >Reporter: Joseph Percivall >Assignee: Joseph Percivall >Priority: Major > > Currently, on DockerHub the tagged image for Apache NiFi 1.7.1 instead pulls > 1.7.0. This can be verified by looking at the DockerFile for the tagged > commit[1], looking at the pushed DockerFile[2], and verifying locally by > running "docker run apache/nifi:1.7.1" and looking at the logs. > > > [1] > [https://github.com/apache/nifi/blob/rel/nifi-1.7.1/nifi-docker/dockerhub/Dockerfile#L25] > [2] [https://hub.docker.com/r/apache/nifi/~/dockerfile/] > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5457) apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0
[ https://issues.apache.org/jira/browse/NIFI-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556402#comment-16556402 ] Joseph Percivall commented on NIFI-5457: [~aldrin] done > apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0 > - > > Key: NIFI-5457 > URL: https://issues.apache.org/jira/browse/NIFI-5457 > Project: Apache NiFi > Issue Type: Task >Reporter: Joseph Percivall >Assignee: Joseph Percivall >Priority: Major > > Currently, on DockerHub the tagged image for Apache NiFi 1.7.1 instead pulls > 1.7.0. This can be verified by looking at the DockerFile for the tagged > commit[1], looking at the pushed DockerFile[2], and verifying locally by > running "docker run apache/nifi:1.7.1" and looking at the logs. > > > [1] > [https://github.com/apache/nifi/blob/rel/nifi-1.7.1/nifi-docker/dockerhub/Dockerfile#L25] > [2] [https://hub.docker.com/r/apache/nifi/~/dockerfile/] > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5457) apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0
[ https://issues.apache.org/jira/browse/NIFI-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556399#comment-16556399 ] Joseph Percivall commented on NIFI-5457: Ah yup, can do > apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0 > - > > Key: NIFI-5457 > URL: https://issues.apache.org/jira/browse/NIFI-5457 > Project: Apache NiFi > Issue Type: Task >Reporter: Joseph Percivall >Assignee: Joseph Percivall >Priority: Major > > Currently, on DockerHub the tagged image for Apache NiFi 1.7.1 instead pulls > 1.7.0. This can be verified by looking at the DockerFile for the tagged > commit[1], looking at the pushed DockerFile[2], and verifying locally by > running "docker run apache/nifi:1.7.1" and looking at the logs. > > > [1] > [https://github.com/apache/nifi/blob/rel/nifi-1.7.1/nifi-docker/dockerhub/Dockerfile#L25] > [2] [https://hub.docker.com/r/apache/nifi/~/dockerfile/] > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5442) Message Page uses raw X-ProxyContextPath
[
https://issues.apache.org/jira/browse/NIFI-5442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556393#comment-16556393
]
ASF GitHub Bot commented on NIFI-5442:
--
Github user alopresto commented on the issue:
https://github.com/apache/nifi/pull/2908
I believe there are some other locations in the code that forward requests
to `/message` rather than `.../message-page.jsp`, so I am performing a refactor
where either the `CatchAllFilter` or a service/utility method for the
sanitization will be moved to `nifi-web-utils` and then both `nifi-web-ui` and
`nifi-web-error` will consume this and register (the single filter or a filter
per URL) in their respective `web.xml` files.
This PR is not ready to be merged.
> Message Page uses raw X-ProxyContextPath
>
>
> Key: NIFI-5442
> URL: https://issues.apache.org/jira/browse/NIFI-5442
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.6.0
>Reporter: Dan Fike
>Assignee: Andy LoPresto
>Priority: Major
>
> It looks like {{message-page.jsp}} uses {{X-ProxyContextPath}} verbatim
> without sanitizing it or anything. See
> [https://github.com/apache/nifi/blob/66783c18b24b1c6b1cfd662c58ca9df1e60b866e/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/WEB-INF/pages/message-page.jsp#L21]
>
> I verified this by hitting {{/nifi-api/access/oidc/callback}} on an unsecured
> NiFi host to get the *User authentication/authorization is only supported
> when running over HTTPS* message page.
>
> {code:java}
> $ curl http://hostname/nifi-api/access/oidc/callback
> ...
> type="text/css" />
> ...
> $ curl --header "X-ProxyContextPath: /nifi/assets/reset.css/reset.css\"
> type=\"text/css\" /> type=\"text/javascript\">alert(\"omg\"); href=\"" http://hostname/nifi-api/access/oidc/callback
> ...
> type="text/css" />alert("omg"); rel="stylesheet" href="/nifi/assets/reset.css/reset.css" type="text/css" />
> ...{code}
>
> Presumably we want to do something like this:
> [https://github.com/apache/nifi/commit/5d643edfaba4f5369c94ee1b4eaa5c59e3a9f37a#diff-91119fe15bb6f3b931662093e367b671R20]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #2908: NIFI-5442 Get X-ProxyContextPath value from request attrib...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2908 I believe there are some other locations in the code that forward requests to `/message` rather than `.../message-page.jsp`, so I am performing a refactor where either the `CatchAllFilter` or a service/utility method for the sanitization will be moved to `nifi-web-utils` and then both `nifi-web-ui` and `nifi-web-error` will consume this and register (the single filter or a filter per URL) in their respective `web.xml` files. This PR is not ready to be merged. ---
[jira] [Reopened] (NIFI-5457) apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0
[ https://issues.apache.org/jira/browse/NIFI-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri reopened NIFI-5457: --- Could you also tag the image you pushed up as latest? Tag the image you made locally and then push that as well. docker tag apache/nifi:1.7.1 apache/nifi:latest docker push apache/nifi:latest > apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0 > - > > Key: NIFI-5457 > URL: https://issues.apache.org/jira/browse/NIFI-5457 > Project: Apache NiFi > Issue Type: Task >Reporter: Joseph Percivall >Assignee: Joseph Percivall >Priority: Major > > Currently, on DockerHub the tagged image for Apache NiFi 1.7.1 instead pulls > 1.7.0. This can be verified by looking at the DockerFile for the tagged > commit[1], looking at the pushed DockerFile[2], and verifying locally by > running "docker run apache/nifi:1.7.1" and looking at the logs. > > > [1] > [https://github.com/apache/nifi/blob/rel/nifi-1.7.1/nifi-docker/dockerhub/Dockerfile#L25] > [2] [https://hub.docker.com/r/apache/nifi/~/dockerfile/] > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5459) Integrate TLS Toolkit with external CA / certificate providers
[
https://issues.apache.org/jira/browse/NIFI-5459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556380#comment-16556380
]
Andy LoPresto commented on NIFI-5459:
-
This effort should investigate the ACME protocol and the Let's Encrypt certbot
tool as examples, if not for direct integration.
* https://tools.ietf.org/html/draft-ietf-acme-acme-13
* https://en.wikipedia.org/wiki/Automated_Certificate_Management_Environment
* https://letsencrypt.org/docs/client-options/
*
https://community.letsencrypt.org/t/acme-v2-production-environment-wildcards/55578
> Integrate TLS Toolkit with external CA / certificate providers
> --
>
> Key: NIFI-5459
> URL: https://issues.apache.org/jira/browse/NIFI-5459
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions, Security, Tools and Build
>Affects Versions: 1.7.1
>Reporter: Andy LoPresto
>Priority: Major
> Labels: certificate, security, tls, tls-toolkit
>
> The TLS toolkit was designed to be a sandbox-level assistant for deployments
> which did not have access to a security team / dedicated certificate provider
> to replace manual generation of certificates with complicated {{openssl}} and
> {{keytool}} commands. It has grown to be depended on by many users, as well
> as Apache Ambari deployments.
> Because of these uses, it should be made more robust, including integration
> with external certificate providers. Multiple commercial certificate
> providers have APIs for automating the submission of CSRs and requesting
> issuance of signed certificates. In the *standalone* or *client*/*server*
> model, the toolkit should be able to define an external certificate provider,
> and if the proper "definition" (API mapping) is available, use provided
> credentials to request and receive a signed certificate. There should be an
> intermediate mapping between a "toolkit-standard communication method" and
> the communication with the external provider, so that the toolkit requestor
> (*standalone*/*client*) does not need to change; either the *server* maps the
> standard request to a specific provider, or the *standalone* component uses
> the pluggable mapping to do the same.
> Only two definitions should be required to mark this ticket as complete:
> * the internal NiFi provider (map toolkit commands to internal {{openssl}}
> generation commands)
> * an external provider (suggest
> [letsencrypt.org|https://letsencrypt.org/how-it-works/] as it is free,
> non-profit, and domain validation (DV) is automated)
> Additional definitions can be provided as follow-on tickets independent of
> this ticket. If internal organization/enterprise certificate providers are
> needed, they can follow the well-defined extension point which should result
> from this effort.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (NIFI-5462) Refactor TLS Toolkit
Andy LoPresto created NIFI-5462: --- Summary: Refactor TLS Toolkit Key: NIFI-5462 URL: https://issues.apache.org/jira/browse/NIFI-5462 Project: Apache NiFi Issue Type: Improvement Components: Security, Tools and Build Affects Versions: 1.7.1 Reporter: Andy LoPresto The TLS Toolkit was originally designed with limited scope and has since grown organically to meet additional requirements. Because of this, its components can be disjoint, there is unnecessary code and communication interfaces, and it can be difficult to extend. A complete refactor should be performed to remove superfluous code, make it easier to extend, separate concerns (command-line interaction, certificate generation and signing, and component communication [client/server, internal/external]). Along with this refactor, it should be extended to handle additional components which require certificate generation, signing, and trust allocation, such as the Apache NiFi Registry. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5461) TLS Toolkit should populate generated client certificate DN into authorizers.xml if requested
Andy LoPresto created NIFI-5461:
---
Summary: TLS Toolkit should populate generated client certificate
DN into authorizers.xml if requested
Key: NIFI-5461
URL: https://issues.apache.org/jira/browse/NIFI-5461
Project: Apache NiFi
Issue Type: New Feature
Components: Configuration, Security, Tools and Build
Affects Versions: 1.7.1
Reporter: Andy LoPresto
When initially configuring the {{authorizers.xml}}, it can be difficult because
the *Initial Admin Identity* must be exactly the same as the Distinguished Name
(DN) from the generated client certificate. Spaces in the DN can make this
especially hard. The TLS Toolkit should have a command-line option which
instructs it to generate an {{authorizers.xml}} with the client DN populated as
the *IAI* in every necessary location.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (NIFI-5460) TLS Toolkit should allow custom CAs to be added to generated truststores
Andy LoPresto created NIFI-5460: --- Summary: TLS Toolkit should allow custom CAs to be added to generated truststores Key: NIFI-5460 URL: https://issues.apache.org/jira/browse/NIFI-5460 Project: Apache NiFi Issue Type: New Feature Components: Extensions, Security, Tools and Build Affects Versions: 1.7.1 Reporter: Andy LoPresto The TLS Toolkit should allow a command-line flag to accept custom signing authorities so that a generated truststore can contain multiple trusted certificates rather than requiring manual joining of separate truststores. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5459) Integrate TLS Toolkit with external CA / certificate providers
Andy LoPresto created NIFI-5459:
---
Summary: Integrate TLS Toolkit with external CA / certificate
providers
Key: NIFI-5459
URL: https://issues.apache.org/jira/browse/NIFI-5459
Project: Apache NiFi
Issue Type: New Feature
Components: Extensions, Security, Tools and Build
Affects Versions: 1.7.1
Reporter: Andy LoPresto
The TLS toolkit was designed to be a sandbox-level assistant for deployments
which did not have access to a security team / dedicated certificate provider
to replace manual generation of certificates with complicated {{openssl}} and
{{keytool}} commands. It has grown to be depended on by many users, as well as
Apache Ambari deployments.
Because of these uses, it should be made more robust, including integration
with external certificate providers. Multiple commercial certificate providers
have APIs for automating the submission of CSRs and requesting issuance of
signed certificates. In the *standalone* or *client*/*server* model, the
toolkit should be able to define an external certificate provider, and if the
proper "definition" (API mapping) is available, use provided credentials to
request and receive a signed certificate. There should be an intermediate
mapping between a "toolkit-standard communication method" and the communication
with the external provider, so that the toolkit requestor
(*standalone*/*client*) does not need to change; either the *server* maps the
standard request to a specific provider, or the *standalone* component uses the
pluggable mapping to do the same.
Only two definitions should be required to mark this ticket as complete:
* the internal NiFi provider (map toolkit commands to internal {{openssl}}
generation commands)
* an external provider (suggest
[letsencrypt.org|https://letsencrypt.org/how-it-works/] as it is free,
non-profit, and domain validation (DV) is automated)
Additional definitions can be provided as follow-on tickets independent of this
ticket. If internal organization/enterprise certificate providers are needed,
they can follow the well-defined extension point which should result from this
effort.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5458) Improve NiFi TLS and certificate management
[ https://issues.apache.org/jira/browse/NIFI-5458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andy LoPresto updated NIFI-5458: Summary: Improve NiFi TLS and certificate management (was: NiFi security configuration requires substantial knowledge and effort to deploy) > Improve NiFi TLS and certificate management > --- > > Key: NIFI-5458 > URL: https://issues.apache.org/jira/browse/NIFI-5458 > Project: Apache NiFi > Issue Type: Epic > Components: Configuration, Configuration Management, Core Framework, > Docker, Security >Affects Versions: 1.7.1 >Reporter: Andy LoPresto >Assignee: Andy LoPresto >Priority: Major > Labels: certificate, cluster, encryption, security, tls, > tls-toolkit > > To securely deploy Apache NiFi requires substantial background knowledge, > applied familiarity with a disparate set of tools and operating systems, and > disjoint manual effort. The NiFi TLS Toolkit and Encrypt Config Toolkits aim > to help, but the former is designed for development/sandbox environments, not > integration with enterprise certificate authorities (CA). In addition, NiFi > requires tightly coupled security configuration when deploying in a cluster > environment, and dynamic horizontal scaling is difficult. > This epic will serve as an aggregator for all individual tickets related to > an ongoing, holistic effort to streamline, automate, and lower the barrier to > entry to configuring a secure NiFi deployment. > * Generating or acquiring signed certificates and converting them to the > proper format (JKS, PEM, P12, etc.) > * Integrating with external certificate providers > * Securing the sensitive configuration values > * Automating deployment of configuration values > * Encapsulating/delegating security configuration for containerization efforts > * Automating deployment of TLS cipher suites and protocol versions > * Automating mitigation of TLS vulnerabilities -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5458) Improve NiFi TLS and certificate management
[ https://issues.apache.org/jira/browse/NIFI-5458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andy LoPresto updated NIFI-5458: Epic Name: NiFi security configuration requires substantial knowledge and effort to deploy (was: Improve NiFi TLS and certificate management) > Improve NiFi TLS and certificate management > --- > > Key: NIFI-5458 > URL: https://issues.apache.org/jira/browse/NIFI-5458 > Project: Apache NiFi > Issue Type: Epic > Components: Configuration, Configuration Management, Core Framework, > Docker, Security >Affects Versions: 1.7.1 >Reporter: Andy LoPresto >Assignee: Andy LoPresto >Priority: Major > Labels: certificate, cluster, encryption, security, tls, > tls-toolkit > > To securely deploy Apache NiFi requires substantial background knowledge, > applied familiarity with a disparate set of tools and operating systems, and > disjoint manual effort. The NiFi TLS Toolkit and Encrypt Config Toolkits aim > to help, but the former is designed for development/sandbox environments, not > integration with enterprise certificate authorities (CA). In addition, NiFi > requires tightly coupled security configuration when deploying in a cluster > environment, and dynamic horizontal scaling is difficult. > This epic will serve as an aggregator for all individual tickets related to > an ongoing, holistic effort to streamline, automate, and lower the barrier to > entry to configuring a secure NiFi deployment. > * Generating or acquiring signed certificates and converting them to the > proper format (JKS, PEM, P12, etc.) > * Integrating with external certificate providers > * Securing the sensitive configuration values > * Automating deployment of configuration values > * Encapsulating/delegating security configuration for containerization efforts > * Automating deployment of TLS cipher suites and protocol versions > * Automating mitigation of TLS vulnerabilities -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (NIFI-5457) apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0
[ https://issues.apache.org/jira/browse/NIFI-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall resolved NIFI-5457. Resolution: Fixed > apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0 > - > > Key: NIFI-5457 > URL: https://issues.apache.org/jira/browse/NIFI-5457 > Project: Apache NiFi > Issue Type: Task >Reporter: Joseph Percivall >Assignee: Joseph Percivall >Priority: Major > > Currently, on DockerHub the tagged image for Apache NiFi 1.7.1 instead pulls > 1.7.0. This can be verified by looking at the DockerFile for the tagged > commit[1], looking at the pushed DockerFile[2], and verifying locally by > running "docker run apache/nifi:1.7.1" and looking at the logs. > > > [1] > [https://github.com/apache/nifi/blob/rel/nifi-1.7.1/nifi-docker/dockerhub/Dockerfile#L25] > [2] [https://hub.docker.com/r/apache/nifi/~/dockerfile/] > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5457) apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0
[ https://issues.apache.org/jira/browse/NIFI-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556350#comment-16556350 ] Joseph Percivall commented on NIFI-5457: [~aldrin] I checked out rel/nifi-1.7.1, changed the Dockerfile, then built and pushed the 1.7.1 tag. I then verified by pulling the tagged image down on another box. That said, I did misunderstand the push command at first which caused the 1.4.0 image to be overwritten. To make sure I didn't mess anything up, rebuilt using the tagged rel/nifi-1.4.0 commit and pushed out the resulting image. The only difference is that now 1.4.0 is listed as updated recently. Let me know if there's anything for me to fix but I'll be closing this issue as done. > apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0 > - > > Key: NIFI-5457 > URL: https://issues.apache.org/jira/browse/NIFI-5457 > Project: Apache NiFi > Issue Type: Task >Reporter: Joseph Percivall >Assignee: Joseph Percivall >Priority: Major > > Currently, on DockerHub the tagged image for Apache NiFi 1.7.1 instead pulls > 1.7.0. This can be verified by looking at the DockerFile for the tagged > commit[1], looking at the pushed DockerFile[2], and verifying locally by > running "docker run apache/nifi:1.7.1" and looking at the logs. > > > [1] > [https://github.com/apache/nifi/blob/rel/nifi-1.7.1/nifi-docker/dockerhub/Dockerfile#L25] > [2] [https://hub.docker.com/r/apache/nifi/~/dockerfile/] > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5458) NiFi security configuration requires substantial knowledge and effort to deploy
Andy LoPresto created NIFI-5458: --- Summary: NiFi security configuration requires substantial knowledge and effort to deploy Key: NIFI-5458 URL: https://issues.apache.org/jira/browse/NIFI-5458 Project: Apache NiFi Issue Type: Epic Components: Security, Configuration, Configuration Management, Core Framework, Docker Affects Versions: 1.7.1 Reporter: Andy LoPresto Assignee: Andy LoPresto To securely deploy Apache NiFi requires substantial background knowledge, applied familiarity with a disparate set of tools and operating systems, and disjoint manual effort. The NiFi TLS Toolkit and Encrypt Config Toolkits aim to help, but the former is designed for development/sandbox environments, not integration with enterprise certificate authorities (CA). In addition, NiFi requires tightly coupled security configuration when deploying in a cluster environment, and dynamic horizontal scaling is difficult. This epic will serve as an aggregator for all individual tickets related to an ongoing, holistic effort to streamline, automate, and lower the barrier to entry to configuring a secure NiFi deployment. * Generating or acquiring signed certificates and converting them to the proper format (JKS, PEM, P12, etc.) * Integrating with external certificate providers * Securing the sensitive configuration values * Automating deployment of configuration values * Encapsulating/delegating security configuration for containerization efforts * Automating deployment of TLS cipher suites and protocol versions * Automating mitigation of TLS vulnerabilities -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (NIFI-5457) apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0
[ https://issues.apache.org/jira/browse/NIFI-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall reassigned NIFI-5457: -- Assignee: Joseph Percivall > apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0 > - > > Key: NIFI-5457 > URL: https://issues.apache.org/jira/browse/NIFI-5457 > Project: Apache NiFi > Issue Type: Task >Reporter: Joseph Percivall >Assignee: Joseph Percivall >Priority: Major > > Currently, on DockerHub the tagged image for Apache NiFi 1.7.1 instead pulls > 1.7.0. This can be verified by looking at the DockerFile for the tagged > commit[1], looking at the pushed DockerFile[2], and verifying locally by > running "docker run apache/nifi:1.7.1" and looking at the logs. > > > [1] > [https://github.com/apache/nifi/blob/rel/nifi-1.7.1/nifi-docker/dockerhub/Dockerfile#L25] > [2] [https://hub.docker.com/r/apache/nifi/~/dockerfile/] > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5457) apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0
[ https://issues.apache.org/jira/browse/NIFI-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556311#comment-16556311 ] Joseph Percivall commented on NIFI-5457: Ok cool, I'll work on overwriting the image > apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0 > - > > Key: NIFI-5457 > URL: https://issues.apache.org/jira/browse/NIFI-5457 > Project: Apache NiFi > Issue Type: Task >Reporter: Joseph Percivall >Priority: Major > > Currently, on DockerHub the tagged image for Apache NiFi 1.7.1 instead pulls > 1.7.0. This can be verified by looking at the DockerFile for the tagged > commit[1], looking at the pushed DockerFile[2], and verifying locally by > running "docker run apache/nifi:1.7.1" and looking at the logs. > > > [1] > [https://github.com/apache/nifi/blob/rel/nifi-1.7.1/nifi-docker/dockerhub/Dockerfile#L25] > [2] [https://hub.docker.com/r/apache/nifi/~/dockerfile/] > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5456) PutKinesisStream - Fails to work with AWS Private Link endpoint
[ https://issues.apache.org/jira/browse/NIFI-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ariel Godinez updated NIFI-5456: Priority: Major (was: Minor) > PutKinesisStream - Fails to work with AWS Private Link endpoint > --- > > Key: NIFI-5456 > URL: https://issues.apache.org/jira/browse/NIFI-5456 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.6.0, 1.7.1 > Environment: RedHat 6 >Reporter: Ariel Godinez >Priority: Major > Labels: easyfix > > NiFi version: 1.6.0 > PutKinesisStream fails to put due to invalid signing information when using > an AWS Private Link as the endpoint override URL. The endpoint override URL > pattern for private links is like below along with the error that NiFi > outputs when we attempt to use this type of URL as the 'Endpoint Override > URL' property value. > Endpoint Override URL: > [https://vpce-|https://vpce-/].kinesis.us-east-2.vpce.amazonaws.com > ERROR [Timer-Driven Process Thread-11] "o.a.n.p.a.k.stream.PutKinesisStream" > PutKinesisStream[id=4c314e25-0164-1000--9bd79c77] Failed to publish > due to exception com.amazonaws.services.kinesis.model.AmazonKinesisException: > Credential should be scoped to a valid region, not 'vpce'. (Service: > AmazonKinesis; Status Code: 400; Error Code: InvalidSignatureException; > Request ID: 6330b83c-a64e-4acf-b892-a505621cf78e) flowfiles > [StandardFlowFileRecord[uuid=ba299cec-7cbf-4750-a766-c348b5cd9c73,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1532469012962-1, > container=content002, section=1], offset=2159750, > length=534625],offset=0,name=900966573101260,size=534625]] > > It looks like 'vpce' is being extracted from the url as the region name when > it should be getting 'us-east-2'. We were able to get this processor to work > correctly by explicitly passing in the region and service using > 'setEndpoint(String endpoint, String serviceName, String regionId)' instead > of 'setEndpoint(String endpoint)' in > 'nifi/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java' > line 289 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5456) PutKinesisStream - Fails to work with AWS Private Link endpoint
[ https://issues.apache.org/jira/browse/NIFI-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ariel Godinez updated NIFI-5456: Priority: Minor (was: Major) > PutKinesisStream - Fails to work with AWS Private Link endpoint > --- > > Key: NIFI-5456 > URL: https://issues.apache.org/jira/browse/NIFI-5456 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.6.0, 1.7.1 > Environment: RedHat 6 >Reporter: Ariel Godinez >Priority: Minor > Labels: easyfix > > NiFi version: 1.6.0 > PutKinesisStream fails to put due to invalid signing information when using > an AWS Private Link as the endpoint override URL. The endpoint override URL > pattern for private links is like below along with the error that NiFi > outputs when we attempt to use this type of URL as the 'Endpoint Override > URL' property value. > Endpoint Override URL: > [https://vpce-|https://vpce-/].kinesis.us-east-2.vpce.amazonaws.com > ERROR [Timer-Driven Process Thread-11] "o.a.n.p.a.k.stream.PutKinesisStream" > PutKinesisStream[id=4c314e25-0164-1000--9bd79c77] Failed to publish > due to exception com.amazonaws.services.kinesis.model.AmazonKinesisException: > Credential should be scoped to a valid region, not 'vpce'. (Service: > AmazonKinesis; Status Code: 400; Error Code: InvalidSignatureException; > Request ID: 6330b83c-a64e-4acf-b892-a505621cf78e) flowfiles > [StandardFlowFileRecord[uuid=ba299cec-7cbf-4750-a766-c348b5cd9c73,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1532469012962-1, > container=content002, section=1], offset=2159750, > length=534625],offset=0,name=900966573101260,size=534625]] > > It looks like 'vpce' is being extracted from the url as the region name when > it should be getting 'us-east-2'. We were able to get this processor to work > correctly by explicitly passing in the region and service using > 'setEndpoint(String endpoint, String serviceName, String regionId)' instead > of 'setEndpoint(String endpoint)' in > 'nifi/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java' > line 289 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5456) PutKinesisStream - Fails to work with AWS Private Link endpoint
[ https://issues.apache.org/jira/browse/NIFI-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ariel Godinez updated NIFI-5456: Priority: Major (was: Minor) Summary: PutKinesisStream - Fails to work with AWS Private Link endpoint (was: PutKinesisStream fails to put due to invalid signing information) > PutKinesisStream - Fails to work with AWS Private Link endpoint > --- > > Key: NIFI-5456 > URL: https://issues.apache.org/jira/browse/NIFI-5456 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.6.0, 1.7.1 > Environment: RedHat 6 >Reporter: Ariel Godinez >Priority: Major > Labels: easyfix > > NiFi version: 1.6.0 > PutKinesisStream fails to put due to invalid signing information when using > an AWS Private Link as the endpoint override URL. The endpoint override URL > pattern for private links is like below along with the error that NiFi > outputs when we attempt to use this type of URL as the 'Endpoint Override > URL' property value. > Endpoint Override URL: > [https://vpce-|https://vpce-/].kinesis.us-east-2.vpce.amazonaws.com > ERROR [Timer-Driven Process Thread-11] "o.a.n.p.a.k.stream.PutKinesisStream" > PutKinesisStream[id=4c314e25-0164-1000--9bd79c77] Failed to publish > due to exception com.amazonaws.services.kinesis.model.AmazonKinesisException: > Credential should be scoped to a valid region, not 'vpce'. (Service: > AmazonKinesis; Status Code: 400; Error Code: InvalidSignatureException; > Request ID: 6330b83c-a64e-4acf-b892-a505621cf78e) flowfiles > [StandardFlowFileRecord[uuid=ba299cec-7cbf-4750-a766-c348b5cd9c73,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1532469012962-1, > container=content002, section=1], offset=2159750, > length=534625],offset=0,name=900966573101260,size=534625]] > > It looks like 'vpce' is being extracted from the url as the region name when > it should be getting 'us-east-2'. We were able to get this processor to work > correctly by explicitly passing in the region and service using > 'setEndpoint(String endpoint, String serviceName, String regionId)' instead > of 'setEndpoint(String endpoint)' in > 'nifi/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java' > line 289 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5457) apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0
[ https://issues.apache.org/jira/browse/NIFI-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556297#comment-16556297 ] Aldrin Piri commented on NIFI-5457: --- Hey [~JPercivall], The issue is that the Dockerfile was never updated to reflect 1.7.1 as per https://github.com/apache/nifi/blob/rel/nifi-1.7.1/nifi-docker/dockerhub/Dockerfile#L25. The best choice for a solution is to check out the release, update the Dockerfile manually, build locally, and push the resultant image to overwrite the 1.7.1 tag. > apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0 > - > > Key: NIFI-5457 > URL: https://issues.apache.org/jira/browse/NIFI-5457 > Project: Apache NiFi > Issue Type: Task >Reporter: Joseph Percivall >Priority: Major > > Currently, on DockerHub the tagged image for Apache NiFi 1.7.1 instead pulls > 1.7.0. This can be verified by looking at the DockerFile for the tagged > commit[1], looking at the pushed DockerFile[2], and verifying locally by > running "docker run apache/nifi:1.7.1" and looking at the logs. > > > [1] > [https://github.com/apache/nifi/blob/rel/nifi-1.7.1/nifi-docker/dockerhub/Dockerfile#L25] > [2] [https://hub.docker.com/r/apache/nifi/~/dockerfile/] > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5456) PutKinesisStream fails to put due to invalid signing information
[ https://issues.apache.org/jira/browse/NIFI-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ariel Godinez updated NIFI-5456: Affects Version/s: 1.6.0 1.7.1 > PutKinesisStream fails to put due to invalid signing information > > > Key: NIFI-5456 > URL: https://issues.apache.org/jira/browse/NIFI-5456 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.6.0, 1.7.1 > Environment: RedHat 6 >Reporter: Ariel Godinez >Priority: Minor > Labels: easyfix > > NiFi version: 1.6.0 > PutKinesisStream fails to put due to invalid signing information when using > an AWS Private Link as the endpoint override URL. The endpoint override URL > pattern for private links is like below along with the error that NiFi > outputs when we attempt to use this type of URL as the 'Endpoint Override > URL' property value. > Endpoint Override URL: > [https://vpce-|https://vpce-/].kinesis.us-east-2.vpce.amazonaws.com > ERROR [Timer-Driven Process Thread-11] "o.a.n.p.a.k.stream.PutKinesisStream" > PutKinesisStream[id=4c314e25-0164-1000--9bd79c77] Failed to publish > due to exception com.amazonaws.services.kinesis.model.AmazonKinesisException: > Credential should be scoped to a valid region, not 'vpce'. (Service: > AmazonKinesis; Status Code: 400; Error Code: InvalidSignatureException; > Request ID: 6330b83c-a64e-4acf-b892-a505621cf78e) flowfiles > [StandardFlowFileRecord[uuid=ba299cec-7cbf-4750-a766-c348b5cd9c73,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1532469012962-1, > container=content002, section=1], offset=2159750, > length=534625],offset=0,name=900966573101260,size=534625]] > > It looks like 'vpce' is being extracted from the url as the region name when > it should be getting 'us-east-2'. We were able to get this processor to work > correctly by explicitly passing in the region and service using > 'setEndpoint(String endpoint, String serviceName, String regionId)' instead > of 'setEndpoint(String endpoint)' in > 'nifi/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java' > line 289 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5456) PutKinesisStream fails to put due to invalid signing information
[ https://issues.apache.org/jira/browse/NIFI-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556294#comment-16556294 ] Randy Bovay commented on NIFI-5456: --- While we have only tested with 1.6.0 and 1.7.1, I anticipate this will fail on all versions of NiFi. May want to title this one as 'PutKinesisStream - Fails to work with Private Link' I would also bump the Priority higher as we cannot use this processor w/o re-compiling it ourselves to make this work. > PutKinesisStream fails to put due to invalid signing information > > > Key: NIFI-5456 > URL: https://issues.apache.org/jira/browse/NIFI-5456 > Project: Apache NiFi > Issue Type: Bug > Environment: RedHat 6 >Reporter: Ariel Godinez >Priority: Minor > Labels: easyfix > > NiFi version: 1.6.0 > PutKinesisStream fails to put due to invalid signing information when using > an AWS Private Link as the endpoint override URL. The endpoint override URL > pattern for private links is like below along with the error that NiFi > outputs when we attempt to use this type of URL as the 'Endpoint Override > URL' property value. > Endpoint Override URL: > [https://vpce-|https://vpce-/].kinesis.us-east-2.vpce.amazonaws.com > ERROR [Timer-Driven Process Thread-11] "o.a.n.p.a.k.stream.PutKinesisStream" > PutKinesisStream[id=4c314e25-0164-1000--9bd79c77] Failed to publish > due to exception com.amazonaws.services.kinesis.model.AmazonKinesisException: > Credential should be scoped to a valid region, not 'vpce'. (Service: > AmazonKinesis; Status Code: 400; Error Code: InvalidSignatureException; > Request ID: 6330b83c-a64e-4acf-b892-a505621cf78e) flowfiles > [StandardFlowFileRecord[uuid=ba299cec-7cbf-4750-a766-c348b5cd9c73,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1532469012962-1, > container=content002, section=1], offset=2159750, > length=534625],offset=0,name=900966573101260,size=534625]] > > It looks like 'vpce' is being extracted from the url as the region name when > it should be getting 'us-east-2'. We were able to get this processor to work > correctly by explicitly passing in the region and service using > 'setEndpoint(String endpoint, String serviceName, String regionId)' instead > of 'setEndpoint(String endpoint)' in > 'nifi/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java' > line 289 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5457) apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0
[ https://issues.apache.org/jira/browse/NIFI-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556269#comment-16556269 ] Joseph Percivall commented on NIFI-5457: Looking at the release guide, it's not as simple as just rebuilding and pushing to dockerhub. It's based on a tagged git push to ASF[1]. [~aldrin] any thoughts to fix it? [1] https://nifi.apache.org/release-guide.html > apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0 > - > > Key: NIFI-5457 > URL: https://issues.apache.org/jira/browse/NIFI-5457 > Project: Apache NiFi > Issue Type: Task >Reporter: Joseph Percivall >Priority: Major > > Currently, on DockerHub the tagged image for Apache NiFi 1.7.1 instead pulls > 1.7.0. This can be verified by looking at the DockerFile for the tagged > commit[1], looking at the pushed DockerFile[2], and verifying locally by > running "docker run apache/nifi:1.7.1" and looking at the logs. > > > [1] > [https://github.com/apache/nifi/blob/rel/nifi-1.7.1/nifi-docker/dockerhub/Dockerfile#L25] > [2] [https://hub.docker.com/r/apache/nifi/~/dockerfile/] > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5457) apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0
Joseph Percivall created NIFI-5457: -- Summary: apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0 Key: NIFI-5457 URL: https://issues.apache.org/jira/browse/NIFI-5457 Project: Apache NiFi Issue Type: Task Reporter: Joseph Percivall Currently, on DockerHub the tagged image for Apache NiFi 1.7.1 instead pulls 1.7.0. This can be verified by looking at the DockerFile for the tagged commit[1], looking at the pushed DockerFile[2], and verifying locally by running "docker run apache/nifi:1.7.1" and looking at the logs. [1] [https://github.com/apache/nifi/blob/rel/nifi-1.7.1/nifi-docker/dockerhub/Dockerfile#L25] [2] [https://hub.docker.com/r/apache/nifi/~/dockerfile/] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4731) BigQuery processors
[ https://issues.apache.org/jira/browse/NIFI-4731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556189#comment-16556189 ] ASF GitHub Bot commented on NIFI-4731: -- Github user danieljimenez commented on the issue: https://github.com/apache/nifi/pull/2682 Hi, sorry for the rebases. This is ready now. > BigQuery processors > --- > > Key: NIFI-4731 > URL: https://issues.apache.org/jira/browse/NIFI-4731 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Mikhail Sosonkin >Priority: Major > > NIFI should have processors for putting data into BigQuery (Streaming and > Batch). > Initial working processors can be found this repository: > https://github.com/nologic/nifi/tree/NIFI-4731/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/bigquery > I'd like to get them into Nifi proper. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2682: NIFI-4731: BQ Processors and GCP library update.
Github user danieljimenez commented on the issue: https://github.com/apache/nifi/pull/2682 Hi, sorry for the rebases. This is ready now. ---
[jira] [Commented] (MINIFICPP-573) break apart the component manifest into separated bundles
[ https://issues.apache.org/jira/browse/MINIFICPP-573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556127#comment-16556127 ] ASF GitHub Bot commented on MINIFICPP-573: -- Github user phrocker commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/381 @achristianson Ah those pieces were recently added with the intention that they *may* be removed to facilitate SIT in a docker container, which is not part of this build. As a result their removal has no unit tests. > break apart the component manifest into separated bundles > - > > Key: MINIFICPP-573 > URL: https://issues.apache.org/jira/browse/MINIFICPP-573 > Project: NiFi MiNiFi C++ > Issue Type: Sub-task >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp issue #381: MINIFICPP-573: Remove lower level build informat...
Github user phrocker commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/381 @achristianson Ah those pieces were recently added with the intention that they *may* be removed to facilitate SIT in a docker container, which is not part of this build. As a result their removal has no unit tests. ---
[jira] [Commented] (MINIFICPP-556) Create snap build
[ https://issues.apache.org/jira/browse/MINIFICPP-556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556115#comment-16556115 ] ASF GitHub Bot commented on MINIFICPP-556: -- Github user achristianson commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/371 Added snap build instructions. Will merge unless there are any further objections. > Create snap build > - > > Key: MINIFICPP-556 > URL: https://issues.apache.org/jira/browse/MINIFICPP-556 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Andrew Christianson >Assignee: Andrew Christianson >Priority: Major > > Package nifi-minifi-cpp into a snap package for easy distribution to users > who use snap. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp issue #371: MINIFICPP-556 Added initial snapcraft build
Github user achristianson commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/371 Added snap build instructions. Will merge unless there are any further objections. ---
[jira] [Updated] (NIFI-5456) PutKinesisStream fails to put due to invalid signing information
[ https://issues.apache.org/jira/browse/NIFI-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ariel Godinez updated NIFI-5456: Description: NiFi version: 1.6.0 PutKinesisStream fails to put due to invalid signing information when using an AWS Private Link as the endpoint override URL. The endpoint override URL pattern for private links is like below along with the error that NiFi outputs when we attempt to use this type of URL as the 'Endpoint Override URL' property value. Endpoint Override URL: [https://vpce-|https://vpce-/].kinesis.us-east-2.vpce.amazonaws.com ERROR [Timer-Driven Process Thread-11] "o.a.n.p.a.k.stream.PutKinesisStream" PutKinesisStream[id=4c314e25-0164-1000--9bd79c77] Failed to publish due to exception com.amazonaws.services.kinesis.model.AmazonKinesisException: Credential should be scoped to a valid region, not 'vpce'. (Service: AmazonKinesis; Status Code: 400; Error Code: InvalidSignatureException; Request ID: 6330b83c-a64e-4acf-b892-a505621cf78e) flowfiles [StandardFlowFileRecord[uuid=ba299cec-7cbf-4750-a766-c348b5cd9c73,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1532469012962-1, container=content002, section=1], offset=2159750, length=534625],offset=0,name=900966573101260,size=534625]] It looks like 'vpce' is being extracted from the url as the region name when it should be getting 'us-east-2'. We were able to get this processor to work correctly by explicitly passing in the region and service using 'setEndpoint(String endpoint, String serviceName, String regionId)' instead of 'setEndpoint(String endpoint)' in 'nifi/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java' line 289 was: NiFi version: 1.6.0 PutKinesisStream fails to put due to invalid signing information when using an AWS Private Link as the endpoint override URL. The endpoint override URL pattern for private links is like below along with the error that NiFi outputs when we attempt to use this type of URL as the 'Endpoint Override URL' property value. Endpoint Override URL: https://vpce-.kinesis.us-east-2.vpce.amazonaws.com ERROR [Timer-Driven Process Thread-11] "o.a.n.p.a.k.stream.PutKinesisStream" PutKinesisStream[id=4c314e25-0164-1000--9bd79c77] Failed to publish due to exception com.amazonaws.services.kinesis.model.AmazonKinesisException: Credential should be scoped to a valid region, not 'vpce'. (Service: AmazonKinesis; Status Code: 400; Error Code: InvalidSignatureException; Request ID: 6330b83c-a64e-4acf-b892-a505621cf78e) flowfiles [StandardFlowFileRecord[uuid=ba299cec-7cbf-4750-a766-c348b5cd9c73,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1532469012962-1, container=content002, section=1], offset=2159750, length=534625],offset=0,name=900966573101260,size=534625]] It looks like 'vpce' is being extracted from the url as the region name when it should be getting 'us-east-2'. I was able to get this processor to work correctly by explicitly passing in the region and service using 'setEndpoint(String endpoint, String serviceName, String regionId)' instead of 'setEndpoint(String endpoint)' in 'nifi/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java' line 289 > PutKinesisStream fails to put due to invalid signing information > > > Key: NIFI-5456 > URL: https://issues.apache.org/jira/browse/NIFI-5456 > Project: Apache NiFi > Issue Type: Bug > Environment: RedHat 6 >Reporter: Ariel Godinez >Priority: Minor > Labels: easyfix > > NiFi version: 1.6.0 > PutKinesisStream fails to put due to invalid signing information when using > an AWS Private Link as the endpoint override URL. The endpoint override URL > pattern for private links is like below along with the error that NiFi > outputs when we attempt to use this type of URL as the 'Endpoint Override > URL' property value. > Endpoint Override URL: > [https://vpce-|https://vpce-/].kinesis.us-east-2.vpce.amazonaws.com > ERROR [Timer-Driven Process Thread-11] "o.a.n.p.a.k.stream.PutKinesisStream" > PutKinesisStream[id=4c314e25-0164-1000--9bd79c77] Failed to publish > due to exception com.amazonaws.services.kinesis.model.AmazonKinesisException: > Credential should be scoped to a valid region, not 'vpce'. (Service: > AmazonKinesis; Status Code: 400; Error Code: InvalidSignatureException; > Request ID: 6330b83c-a64e-4acf-b892-a505621cf78e) flowfiles > [StandardFlowFileRecord[uuid=ba299cec-7cbf-4750-a766-c348b5cd9c73,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1532469012962-1, > container=content002, section=1],
[jira] [Created] (NIFI-5456) PutKinesisStream fails to put due to invalid signing information
Ariel Godinez created NIFI-5456: --- Summary: PutKinesisStream fails to put due to invalid signing information Key: NIFI-5456 URL: https://issues.apache.org/jira/browse/NIFI-5456 Project: Apache NiFi Issue Type: Bug Environment: RedHat 6 Reporter: Ariel Godinez NiFi version: 1.6.0 PutKinesisStream fails to put due to invalid signing information when using an AWS Private Link as the endpoint override URL. The endpoint override URL pattern for private links is like below along with the error that NiFi outputs when we attempt to use this type of URL as the 'Endpoint Override URL' property value. Endpoint Override URL: https://vpce-.kinesis.us-east-2.vpce.amazonaws.com ERROR [Timer-Driven Process Thread-11] "o.a.n.p.a.k.stream.PutKinesisStream" PutKinesisStream[id=4c314e25-0164-1000--9bd79c77] Failed to publish due to exception com.amazonaws.services.kinesis.model.AmazonKinesisException: Credential should be scoped to a valid region, not 'vpce'. (Service: AmazonKinesis; Status Code: 400; Error Code: InvalidSignatureException; Request ID: 6330b83c-a64e-4acf-b892-a505621cf78e) flowfiles [StandardFlowFileRecord[uuid=ba299cec-7cbf-4750-a766-c348b5cd9c73,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1532469012962-1, container=content002, section=1], offset=2159750, length=534625],offset=0,name=900966573101260,size=534625]] It looks like 'vpce' is being extracted from the url as the region name when it should be getting 'us-east-2'. I was able to get this processor to work correctly by explicitly passing in the region and service using 'setEndpoint(String endpoint, String serviceName, String regionId)' instead of 'setEndpoint(String endpoint)' in 'nifi/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java' line 289 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5455) Add a ParquetRecordReader controller service
Matt Burgess created NIFI-5455: -- Summary: Add a ParquetRecordReader controller service Key: NIFI-5455 URL: https://issues.apache.org/jira/browse/NIFI-5455 Project: Apache NiFi Issue Type: New Feature Components: Extensions Reporter: Matt Burgess Currently we have Fetch and PutParquet, the former of which allows a Parquet file to be ingested into NiFi, and the latter of which allows conversion of a record set into Parquet to be placed on a filesystem such as HDFS. However there is no way to convert from Parquet into another format on which additional operations (transformation, conversion to another format, e.g.) may be performed. This Jira proposes to add a ParquetRecordReader controller service that can be used by the record-aware processors for reading in records in Parquet format. I'm not including a ParquetRecordSetWriter in this Jira since there is a PutParquet and any additional operations on the records are probably (at least at present) better handled via a "faster" format w.r.t. NiFi, rather than serializing back to Parquet at each step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (MINIFICPP-576) Roll system component manifest into a top level bundle
Mr TheSegfault created MINIFICPP-576: Summary: Roll system component manifest into a top level bundle Key: MINIFICPP-576 URL: https://issues.apache.org/jira/browse/MINIFICPP-576 Project: NiFi MiNiFi C++ Issue Type: Bug Reporter: Mr TheSegfault Assignee: Mr TheSegfault Roll system component manifest into a top level bundle so that all processors and controller services are contained with a bundle. Those that aren't built within an extension can be placed into a system or default bundle. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5454) Add ability to specify per-flowfile values via DuplicateFlowFile
[ https://issues.apache.org/jira/browse/NIFI-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-5454: --- Status: Patch Available (was: In Progress) > Add ability to specify per-flowfile values via DuplicateFlowFile > > > Key: NIFI-5454 > URL: https://issues.apache.org/jira/browse/NIFI-5454 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Minor > > DuplicateFlowFile is currently useful mostly for load testing, as it > generates a fixed number of flow files as copies of the incoming flow file. > However there is no way to distinguish the flow files from each other, so for > non-test purposes, it would be nice to have a way to treat copies > differently, either for routing purposes or to access other values, such as > attributes with delimited fields, etc. > This Jira proposes to add the following improvements to DuplicateFlowFile: > - add a "copy.index" attribute to the outgoing flow files, setting the > attribute to zero (0) on the original flow file, and incrementally setting it > on the copies. > - Add support for Expression Language to the Number of Copies property -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5454) Add ability to specify per-flowfile values via DuplicateFlowFile
[ https://issues.apache.org/jira/browse/NIFI-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556057#comment-16556057 ] ASF GitHub Bot commented on NIFI-5454: -- GitHub user mattyb149 opened a pull request: https://github.com/apache/nifi/pull/2917 NIFI-5454: Added EL support and copy.index attribute to DuplicateFlowFile Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [x] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mattyb149/nifi NIFI-5454 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2917.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2917 commit ea3aada4faba631ec0dc7c2a209626f512d1343b Author: Matthew Burgess Date: 2018-07-25T17:55:10Z NIFI-5454: Added EL support and copy.index attribute to DuplicateFlowFile > Add ability to specify per-flowfile values via DuplicateFlowFile > > > Key: NIFI-5454 > URL: https://issues.apache.org/jira/browse/NIFI-5454 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Minor > > DuplicateFlowFile is currently useful mostly for load testing, as it > generates a fixed number of flow files as copies of the incoming flow file. > However there is no way to distinguish the flow files from each other, so for > non-test purposes, it would be nice to have a way to treat copies > differently, either for routing purposes or to access other values, such as > attributes with delimited fields, etc. > This Jira proposes to add the following improvements to DuplicateFlowFile: > - add a "copy.index" attribute to the outgoing flow files, setting the > attribute to zero (0) on the original flow file, and incrementally setting it > on the copies. > - Add support for Expression Language to the Number of Copies property -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2917: NIFI-5454: Added EL support and copy.index attribut...
GitHub user mattyb149 opened a pull request: https://github.com/apache/nifi/pull/2917 NIFI-5454: Added EL support and copy.index attribute to DuplicateFlowFile Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [x] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mattyb149/nifi NIFI-5454 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2917.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2917 commit ea3aada4faba631ec0dc7c2a209626f512d1343b Author: Matthew Burgess Date: 2018-07-25T17:55:10Z NIFI-5454: Added EL support and copy.index attribute to DuplicateFlowFile ---
[jira] [Commented] (MINIFICPP-465) Regex property validation
[ https://issues.apache.org/jira/browse/MINIFICPP-465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555874#comment-16555874 ] ASF GitHub Bot commented on MINIFICPP-465: -- Github user achristianson commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/350 Rebased & all issues addressed. Ready for review. > Regex property validation > - > > Key: MINIFICPP-465 > URL: https://issues.apache.org/jira/browse/MINIFICPP-465 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Andrew Christianson >Assignee: Andrew Christianson >Priority: Major > > Support validation of processor properties via regular expression. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5451) Modify NiFiGroovyTest.groovy to avoid needing unlimited strength JCE policy
[ https://issues.apache.org/jira/browse/NIFI-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555875#comment-16555875 ] ASF GitHub Bot commented on NIFI-5451: -- Github user mosermw commented on the issue: https://github.com/apache/nifi/pull/2916 Reviewed and the issue is resolved, +1 will merge. > Modify NiFiGroovyTest.groovy to avoid needing unlimited strength JCE policy > --- > > Key: NIFI-5451 > URL: https://issues.apache.org/jira/browse/NIFI-5451 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.8.0 >Reporter: Michael Moser >Assignee: Andy LoPresto >Priority: Minor > Labels: encryption, jce_policy > > When running NiFiGroovyTest unit test when using a JDK that lacks unlimited > strength JCE policy, the test fails. > > [ERROR] > testInitializePropertiesShouldSetBootstrapKeyFromFile(org.apache.nifi.NiFiGroovyTest) > Time elapsed: 0.121 s <<< ERROR! > java.lang.IllegalArgumentException: There was an issue decrypting protected > properties > at > org.apache.nifi.NiFiGroovyTest.testInitializePropertiesShouldSetBootstrapKeyFromFile(NiFiGroovyTest.groovy:166) > Caused by: org.apache.nifi.properties.SensitivePropertyProtectionException: > The key must be a valid hexadecimal key > at > org.apache.nifi.NiFiGroovyTest.testInitializePropertiesShouldSetBootstrapKeyFromFile(NiFiGroovyTest.groovy:166) > > The test uses a nifi.properties resource that does aes/gcm/256 encryption. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5454) Add ability to specify per-flowfile values via DuplicateFlowFile
[ https://issues.apache.org/jira/browse/NIFI-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-5454: --- Description: DuplicateFlowFile is currently useful mostly for load testing, as it generates a fixed number of flow files as copies of the incoming flow file. However there is no way to distinguish the flow files from each other, so for non-test purposes, it would be nice to have a way to treat copies differently, either for routing purposes or to access other values, such as attributes with delimited fields, etc. This Jira proposes to add the following improvements to DuplicateFlowFile: - add a "copy.index" attribute to the outgoing flow files, setting the attribute to zero (0) on the original flow file, and incrementally setting it on the copies. - Add support for Expression Language to the Number of Copies property was: DuplicateFlowFile is currently useful mostly for load testing, as it generates a fixed number of flow files as copies of the incoming flow file. However there is no way to distinguish the flow files from each other, so for non-test purposes, it would be nice to have a way to treat copies differently, either for routing purposes or to access other values, such as attributes with delimited fields, etc. This Jira proposes to add the following improvements to DuplicateFlowFile: - add a "copy.index" attribute to the outgoing flow files, setting the attribute to zero (0) on the original flow file, and incrementally setting it on the copies. - Add support for Expression Language to the Number of Copies property - Remove the requirement on the Number of Copies property (and remove its default value), and add an optional "Index Values" property, which is expected to be a comma-separated list of values for the copy.index attribute. Exactly one of these two properties must be set. If Number of Copies is set and Index Values is not, then the behavior remains the same as it has been. If Number of Copies is not set and Index Values is set, then the number of values in Index Values list is the Number of Copies that will be produced, and the individual values will be set as the copy.index attribute value. > Add ability to specify per-flowfile values via DuplicateFlowFile > > > Key: NIFI-5454 > URL: https://issues.apache.org/jira/browse/NIFI-5454 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Priority: Minor > > DuplicateFlowFile is currently useful mostly for load testing, as it > generates a fixed number of flow files as copies of the incoming flow file. > However there is no way to distinguish the flow files from each other, so for > non-test purposes, it would be nice to have a way to treat copies > differently, either for routing purposes or to access other values, such as > attributes with delimited fields, etc. > This Jira proposes to add the following improvements to DuplicateFlowFile: > - add a "copy.index" attribute to the outgoing flow files, setting the > attribute to zero (0) on the original flow file, and incrementally setting it > on the copies. > - Add support for Expression Language to the Number of Copies property -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (NIFI-5454) Add ability to specify per-flowfile values via DuplicateFlowFile
[ https://issues.apache.org/jira/browse/NIFI-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess reassigned NIFI-5454: -- Assignee: Matt Burgess > Add ability to specify per-flowfile values via DuplicateFlowFile > > > Key: NIFI-5454 > URL: https://issues.apache.org/jira/browse/NIFI-5454 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Minor > > DuplicateFlowFile is currently useful mostly for load testing, as it > generates a fixed number of flow files as copies of the incoming flow file. > However there is no way to distinguish the flow files from each other, so for > non-test purposes, it would be nice to have a way to treat copies > differently, either for routing purposes or to access other values, such as > attributes with delimited fields, etc. > This Jira proposes to add the following improvements to DuplicateFlowFile: > - add a "copy.index" attribute to the outgoing flow files, setting the > attribute to zero (0) on the original flow file, and incrementally setting it > on the copies. > - Add support for Expression Language to the Number of Copies property -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5454) Add ability to specify per-flowfile values via DuplicateFlowFile
Matt Burgess created NIFI-5454: -- Summary: Add ability to specify per-flowfile values via DuplicateFlowFile Key: NIFI-5454 URL: https://issues.apache.org/jira/browse/NIFI-5454 Project: Apache NiFi Issue Type: Improvement Components: Extensions Reporter: Matt Burgess DuplicateFlowFile is currently useful mostly for load testing, as it generates a fixed number of flow files as copies of the incoming flow file. However there is no way to distinguish the flow files from each other, so for non-test purposes, it would be nice to have a way to treat copies differently, either for routing purposes or to access other values, such as attributes with delimited fields, etc. This Jira proposes to add the following improvements to DuplicateFlowFile: - add a "copy.index" attribute to the outgoing flow files, setting the attribute to zero (0) on the original flow file, and incrementally setting it on the copies. - Add support for Expression Language to the Number of Copies property - Remove the requirement on the Number of Copies property (and remove its default value), and add an optional "Index Values" property, which is expected to be a comma-separated list of values for the copy.index attribute. Exactly one of these two properties must be set. If Number of Copies is set and Index Values is not, then the behavior remains the same as it has been. If Number of Copies is not set and Index Values is set, then the number of values in Index Values list is the Number of Copies that will be produced, and the individual values will be set as the copy.index attribute value. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (MINIFICPP-514) Incorporate regex validation information into agent manifest
[ https://issues.apache.org/jira/browse/MINIFICPP-514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Christianson resolved MINIFICPP-514. --- Resolution: Fixed > Incorporate regex validation information into agent manifest > > > Key: MINIFICPP-514 > URL: https://issues.apache.org/jira/browse/MINIFICPP-514 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Andrew Christianson >Assignee: Andrew Christianson >Priority: Major > > Allow agent to report to c2 regex validation rules for properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (NIFI-5453) Test failure on build for nifi-nar-utils on master
[ https://issues.apache.org/jira/browse/NIFI-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andy LoPresto resolved NIFI-5453. - Resolution: Duplicate > Test failure on build for nifi-nar-utils on master > -- > > Key: NIFI-5453 > URL: https://issues.apache.org/jira/browse/NIFI-5453 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework > Environment: Apache Maven 3.5.0 > (ff8f5e7444045639af65f6095c62210b5713f426; 2017-04-03T15:39:06-04:00) > Maven home: /development/apache-maven-3.5.0 > Java version: 1.8.0_144, vendor: Oracle Corporation > Java home: /usr/java/jdk1.8.0_144/jre > Default locale: en_US, platform encoding: UTF-8 > OS name: "linux", version: "4.17.7-200.fc28.x86_64", arch: "amd64", family: > "unix" >Reporter: Joseph Witt >Priority: Blocker > Fix For: 1.8.0 > > > [INFO] Installing > /development/code/nifi.git/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-nar-utils/target/nifi-framework-nar-utils-1.8.0-SNAPSHOT.jar > to > /development/maven-repo/org/apache/nifi/nifi-framework-nar-utils/1.8.0-SNAPSHOT/nifi-framework-nar-utils-1.8.0-SNAPSHOT.jar > [INFO] Installing > /development/code/nifi.git/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-nar-utils/pom.xml > to > /development/maven-repo/org/apache/nifi/nifi-framework-nar-utils/1.8.0-SNAPSHOT/nifi-framework-nar-utils-1.8.0-SNAPSHOT.pom > [ERROR] Tests run: 8, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.613 > s <<< FAILURE! - in org.apache.nifi.NiFiGroovyTest > [ERROR] > testInitializePropertiesShouldSetBootstrapKeyFromFile(org.apache.nifi.NiFiGroovyTest) > Time elapsed: 0.06 s <<< ERROR! > java.lang.IllegalArgumentException: There was an issue decrypting protected > properties > at > org.apache.nifi.NiFiGroovyTest.testInitializePropertiesShouldSetBootstrapKeyFromFile(NiFiGroovyTest.groovy:166) > Caused by: org.apache.nifi.properties.SensitivePropertyProtectionException: > The key must be a valid hexadecimal key > at > org.apache.nifi.NiFiGroovyTest.testInitializePropertiesShouldSetBootstrapKeyFromFile(NiFiGroovyTest.groovy:166) > [INFO] > [INFO] Results: > [INFO] > [ERROR] Errors: > [ERROR] > NiFiGroovyTest.testInitializePropertiesShouldSetBootstrapKeyFromFile:166 » > IllegalArgument > [INFO] > [ERROR] Tests run: 8, Failures: 0, Errors: 1, Skipped: 0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp issue #347: MINIFIPP-514 Incorporated regex property validat...
Github user apiri commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/347 It looks like this went in before we had outstanding items tended to, the least of which would be an issue which I do not currently see in JIRA. We should either get those tasks captured/logged or revert this commit until we can reach a general consensus. ---
[jira] [Commented] (NIFI-5083) Support for Expression Language in Group ID field of Kafka Consumer Processors
[ https://issues.apache.org/jira/browse/NIFI-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555632#comment-16555632 ] ASF GitHub Bot commented on NIFI-5083: -- Github user bbende commented on the issue: https://github.com/apache/nifi/pull/2913 Actually, I take back my last comment, I now see where the value is obtained and it does have the logic: ``` String propertyValue = propertyDescriptor.isExpressionLanguageSupported() ? context.getProperty(propertyDescriptor).evaluateAttributeExpressions().getValue() : context.getProperty(propertyDescriptor).getValue(); ``` Sorry about that. > Support for Expression Language in Group ID field of Kafka Consumer Processors > -- > > Key: NIFI-5083 > URL: https://issues.apache.org/jira/browse/NIFI-5083 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.6.0 >Reporter: Robert Bruno >Priority: Minor > Labels: kafka > Attachments: NIFI-5083.mp4 > > > I have the use case where I have two separate NiFi clusters reading data from > the same topics but need to have unique group ids so they each get separate > copies of the data. It would be useful if the group id field supported > expression language so I could use a processor group variable to define the > group id to use. I have a large enough number of kafka consumers to make > this tedious to manually change. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (MINIFICPP-570) Move external project builds into separate CMake module(s)
[ https://issues.apache.org/jira/browse/MINIFICPP-570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Christianson updated MINIFICPP-570: -- Priority: Minor (was: Blocker) > Move external project builds into separate CMake module(s) > -- > > Key: MINIFICPP-570 > URL: https://issues.apache.org/jira/browse/MINIFICPP-570 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Andrew Christianson >Assignee: Andrew Christianson >Priority: Minor > > The main CMakeLists.txt is getting cluttered with external project builds. It > is time to move these out into their own space. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5453) Test failure on build for nifi-nar-utils on master
[ https://issues.apache.org/jira/browse/NIFI-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555912#comment-16555912 ] Joseph Witt commented on NIFI-5453: --- probably a dupe of https://issues.apache.org/jira/browse/NIFI-5451 > Test failure on build for nifi-nar-utils on master > -- > > Key: NIFI-5453 > URL: https://issues.apache.org/jira/browse/NIFI-5453 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework > Environment: Apache Maven 3.5.0 > (ff8f5e7444045639af65f6095c62210b5713f426; 2017-04-03T15:39:06-04:00) > Maven home: /development/apache-maven-3.5.0 > Java version: 1.8.0_144, vendor: Oracle Corporation > Java home: /usr/java/jdk1.8.0_144/jre > Default locale: en_US, platform encoding: UTF-8 > OS name: "linux", version: "4.17.7-200.fc28.x86_64", arch: "amd64", family: > "unix" >Reporter: Joseph Witt >Priority: Blocker > Fix For: 1.8.0 > > > [INFO] Installing > /development/code/nifi.git/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-nar-utils/target/nifi-framework-nar-utils-1.8.0-SNAPSHOT.jar > to > /development/maven-repo/org/apache/nifi/nifi-framework-nar-utils/1.8.0-SNAPSHOT/nifi-framework-nar-utils-1.8.0-SNAPSHOT.jar > [INFO] Installing > /development/code/nifi.git/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-nar-utils/pom.xml > to > /development/maven-repo/org/apache/nifi/nifi-framework-nar-utils/1.8.0-SNAPSHOT/nifi-framework-nar-utils-1.8.0-SNAPSHOT.pom > [ERROR] Tests run: 8, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.613 > s <<< FAILURE! - in org.apache.nifi.NiFiGroovyTest > [ERROR] > testInitializePropertiesShouldSetBootstrapKeyFromFile(org.apache.nifi.NiFiGroovyTest) > Time elapsed: 0.06 s <<< ERROR! > java.lang.IllegalArgumentException: There was an issue decrypting protected > properties > at > org.apache.nifi.NiFiGroovyTest.testInitializePropertiesShouldSetBootstrapKeyFromFile(NiFiGroovyTest.groovy:166) > Caused by: org.apache.nifi.properties.SensitivePropertyProtectionException: > The key must be a valid hexadecimal key > at > org.apache.nifi.NiFiGroovyTest.testInitializePropertiesShouldSetBootstrapKeyFromFile(NiFiGroovyTest.groovy:166) > [INFO] > [INFO] Results: > [INFO] > [ERROR] Errors: > [ERROR] > NiFiGroovyTest.testInitializePropertiesShouldSetBootstrapKeyFromFile:166 » > IllegalArgument > [INFO] > [ERROR] Tests run: 8, Failures: 0, Errors: 1, Skipped: 0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5451) Modify NiFiGroovyTest.groovy to avoid needing unlimited strength JCE policy
[ https://issues.apache.org/jira/browse/NIFI-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Moser updated NIFI-5451: Resolution: Fixed Fix Version/s: 1.8.0 Status: Resolved (was: Patch Available) Thank you [~alopresto]! > Modify NiFiGroovyTest.groovy to avoid needing unlimited strength JCE policy > --- > > Key: NIFI-5451 > URL: https://issues.apache.org/jira/browse/NIFI-5451 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.8.0 >Reporter: Michael Moser >Assignee: Andy LoPresto >Priority: Minor > Labels: encryption, jce_policy > Fix For: 1.8.0 > > > When running NiFiGroovyTest unit test when using a JDK that lacks unlimited > strength JCE policy, the test fails. > > [ERROR] > testInitializePropertiesShouldSetBootstrapKeyFromFile(org.apache.nifi.NiFiGroovyTest) > Time elapsed: 0.121 s <<< ERROR! > java.lang.IllegalArgumentException: There was an issue decrypting protected > properties > at > org.apache.nifi.NiFiGroovyTest.testInitializePropertiesShouldSetBootstrapKeyFromFile(NiFiGroovyTest.groovy:166) > Caused by: org.apache.nifi.properties.SensitivePropertyProtectionException: > The key must be a valid hexadecimal key > at > org.apache.nifi.NiFiGroovyTest.testInitializePropertiesShouldSetBootstrapKeyFromFile(NiFiGroovyTest.groovy:166) > > The test uses a nifi.properties resource that does aes/gcm/256 encryption. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5451) Modify NiFiGroovyTest.groovy to avoid needing unlimited strength JCE policy
[ https://issues.apache.org/jira/browse/NIFI-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555876#comment-16555876 ] ASF subversion and git services commented on NIFI-5451: --- Commit 0ad30e188f1b8c39d0598656793d7682f7fc8dd7 in nifi's branch refs/heads/master from [~alopresto] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=0ad30e1 ] NIFI-5451 Added test resources for 128 bit encryption. Fixed unit test to perform properly without JCE unlimited strength policy installed. This closes #2916. Signed-off-by: Mike Moser > Modify NiFiGroovyTest.groovy to avoid needing unlimited strength JCE policy > --- > > Key: NIFI-5451 > URL: https://issues.apache.org/jira/browse/NIFI-5451 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.8.0 >Reporter: Michael Moser >Assignee: Andy LoPresto >Priority: Minor > Labels: encryption, jce_policy > > When running NiFiGroovyTest unit test when using a JDK that lacks unlimited > strength JCE policy, the test fails. > > [ERROR] > testInitializePropertiesShouldSetBootstrapKeyFromFile(org.apache.nifi.NiFiGroovyTest) > Time elapsed: 0.121 s <<< ERROR! > java.lang.IllegalArgumentException: There was an issue decrypting protected > properties > at > org.apache.nifi.NiFiGroovyTest.testInitializePropertiesShouldSetBootstrapKeyFromFile(NiFiGroovyTest.groovy:166) > Caused by: org.apache.nifi.properties.SensitivePropertyProtectionException: > The key must be a valid hexadecimal key > at > org.apache.nifi.NiFiGroovyTest.testInitializePropertiesShouldSetBootstrapKeyFromFile(NiFiGroovyTest.groovy:166) > > The test uses a nifi.properties resource that does aes/gcm/256 encryption. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5451) Modify NiFiGroovyTest.groovy to avoid needing unlimited strength JCE policy
[ https://issues.apache.org/jira/browse/NIFI-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555879#comment-16555879 ] ASF GitHub Bot commented on NIFI-5451: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2916 > Modify NiFiGroovyTest.groovy to avoid needing unlimited strength JCE policy > --- > > Key: NIFI-5451 > URL: https://issues.apache.org/jira/browse/NIFI-5451 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.8.0 >Reporter: Michael Moser >Assignee: Andy LoPresto >Priority: Minor > Labels: encryption, jce_policy > > When running NiFiGroovyTest unit test when using a JDK that lacks unlimited > strength JCE policy, the test fails. > > [ERROR] > testInitializePropertiesShouldSetBootstrapKeyFromFile(org.apache.nifi.NiFiGroovyTest) > Time elapsed: 0.121 s <<< ERROR! > java.lang.IllegalArgumentException: There was an issue decrypting protected > properties > at > org.apache.nifi.NiFiGroovyTest.testInitializePropertiesShouldSetBootstrapKeyFromFile(NiFiGroovyTest.groovy:166) > Caused by: org.apache.nifi.properties.SensitivePropertyProtectionException: > The key must be a valid hexadecimal key > at > org.apache.nifi.NiFiGroovyTest.testInitializePropertiesShouldSetBootstrapKeyFromFile(NiFiGroovyTest.groovy:166) > > The test uses a nifi.properties resource that does aes/gcm/256 encryption. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5453) Test failure on build for nifi-nar-utils on master
Joseph Witt created NIFI-5453: - Summary: Test failure on build for nifi-nar-utils on master Key: NIFI-5453 URL: https://issues.apache.org/jira/browse/NIFI-5453 Project: Apache NiFi Issue Type: Bug Components: Core Framework Environment: Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; 2017-04-03T15:39:06-04:00) Maven home: /development/apache-maven-3.5.0 Java version: 1.8.0_144, vendor: Oracle Corporation Java home: /usr/java/jdk1.8.0_144/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "4.17.7-200.fc28.x86_64", arch: "amd64", family: "unix" Reporter: Joseph Witt Fix For: 1.8.0 [INFO] Installing /development/code/nifi.git/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-nar-utils/target/nifi-framework-nar-utils-1.8.0-SNAPSHOT.jar to /development/maven-repo/org/apache/nifi/nifi-framework-nar-utils/1.8.0-SNAPSHOT/nifi-framework-nar-utils-1.8.0-SNAPSHOT.jar [INFO] Installing /development/code/nifi.git/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-nar-utils/pom.xml to /development/maven-repo/org/apache/nifi/nifi-framework-nar-utils/1.8.0-SNAPSHOT/nifi-framework-nar-utils-1.8.0-SNAPSHOT.pom [ERROR] Tests run: 8, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.613 s <<< FAILURE! - in org.apache.nifi.NiFiGroovyTest [ERROR] testInitializePropertiesShouldSetBootstrapKeyFromFile(org.apache.nifi.NiFiGroovyTest) Time elapsed: 0.06 s <<< ERROR! java.lang.IllegalArgumentException: There was an issue decrypting protected properties at org.apache.nifi.NiFiGroovyTest.testInitializePropertiesShouldSetBootstrapKeyFromFile(NiFiGroovyTest.groovy:166) Caused by: org.apache.nifi.properties.SensitivePropertyProtectionException: The key must be a valid hexadecimal key at org.apache.nifi.NiFiGroovyTest.testInitializePropertiesShouldSetBootstrapKeyFromFile(NiFiGroovyTest.groovy:166) [INFO] [INFO] Results: [INFO] [ERROR] Errors: [ERROR] NiFiGroovyTest.testInitializePropertiesShouldSetBootstrapKeyFromFile:166 » IllegalArgument [INFO] [ERROR] Tests run: 8, Failures: 0, Errors: 1, Skipped: 0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2916: NIFI-5451 Added test resources for 128 bit encrypti...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2916 ---
[GitHub] nifi issue #2916: NIFI-5451 Added test resources for 128 bit encryption.
Github user mosermw commented on the issue: https://github.com/apache/nifi/pull/2916 Reviewed and the issue is resolved, +1 will merge. ---
[GitHub] nifi-minifi-cpp issue #350: MINIFICPP-465 Implemented regex validation of pr...
Github user achristianson commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/350 Rebased & all issues addressed. Ready for review. ---
[jira] [Commented] (NIFI-5442) Message Page uses raw X-ProxyContextPath
[
https://issues.apache.org/jira/browse/NIFI-5442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555768#comment-16555768
]
ASF GitHub Bot commented on NIFI-5442:
--
Github user danfike commented on the issue:
https://github.com/apache/nifi/pull/2908
@alopresto I'm not able to validate right now, but @patwhitey2007 has
agreed to take a look for me.
> Message Page uses raw X-ProxyContextPath
>
>
> Key: NIFI-5442
> URL: https://issues.apache.org/jira/browse/NIFI-5442
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.6.0
>Reporter: Dan Fike
>Assignee: Andy LoPresto
>Priority: Major
>
> It looks like {{message-page.jsp}} uses {{X-ProxyContextPath}} verbatim
> without sanitizing it or anything. See
> [https://github.com/apache/nifi/blob/66783c18b24b1c6b1cfd662c58ca9df1e60b866e/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/WEB-INF/pages/message-page.jsp#L21]
>
> I verified this by hitting {{/nifi-api/access/oidc/callback}} on an unsecured
> NiFi host to get the *User authentication/authorization is only supported
> when running over HTTPS* message page.
>
> {code:java}
> $ curl http://hostname/nifi-api/access/oidc/callback
> ...
> type="text/css" />
> ...
> $ curl --header "X-ProxyContextPath: /nifi/assets/reset.css/reset.css\"
> type=\"text/css\" /> type=\"text/javascript\">alert(\"omg\"); href=\"" http://hostname/nifi-api/access/oidc/callback
> ...
> type="text/css" />alert("omg"); rel="stylesheet" href="/nifi/assets/reset.css/reset.css" type="text/css" />
> ...{code}
>
> Presumably we want to do something like this:
> [https://github.com/apache/nifi/commit/5d643edfaba4f5369c94ee1b4eaa5c59e3a9f37a#diff-91119fe15bb6f3b931662093e367b671R20]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #347: MINIFIPP-514 Incorporated regex property ...
Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/347 ---
[GitHub] nifi issue #2908: NIFI-5442 Get X-ProxyContextPath value from request attrib...
Github user danfike commented on the issue: https://github.com/apache/nifi/pull/2908 @alopresto I'm not able to validate right now, but @patwhitey2007 has agreed to take a look for me. ---
[jira] [Commented] (NIFI-5083) Support for Expression Language in Group ID field of Kafka Consumer Processors
[ https://issues.apache.org/jira/browse/NIFI-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555630#comment-16555630 ] ASF GitHub Bot commented on NIFI-5083: -- Github user bbende commented on the issue: https://github.com/apache/nifi/pull/2913 I think you will also need to find where the group id property is used and ensure that evaluateAttributeExpressions() is called when obtaining the value. > Support for Expression Language in Group ID field of Kafka Consumer Processors > -- > > Key: NIFI-5083 > URL: https://issues.apache.org/jira/browse/NIFI-5083 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.6.0 >Reporter: Robert Bruno >Priority: Minor > Labels: kafka > Attachments: NIFI-5083.mp4 > > > I have the use case where I have two separate NiFi clusters reading data from > the same topics but need to have unique group ids so they each get separate > copies of the data. It would be useful if the group id field supported > expression language so I could use a processor group variable to define the > group id to use. I have a large enough number of kafka consumers to make > this tedious to manually change. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #347: MINIFIPP-514 Incorporated regex property ...
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/347#discussion_r205109263
--- Diff: extensions/http-curl/processors/InvokeHTTP.cpp ---
@@ -65,7 +65,13 @@ core::Property InvokeHTTP::FollowRedirects("Follow
Redirects", "Follow HTTP redi
core::Property InvokeHTTP::AttributesToSend("Attributes to Send", "Regular
expression that defines which attributes to send as HTTP"
" headers in the request. If
not defined, no attributes are sent as headers.",
"");
-core::Property InvokeHTTP::SSLContext("SSL Context Service", "The SSL
Context Service used to provide client certificate information for TLS/SSL
(https) connections.", "");
+core::Property InvokeHTTP::SSLContext("SSL Context Service",
+ "The SSL Context Service used to
provide client certificate "
+ "information for TLS/SSL (https)
connections.",
+ "",
+ false,
+ {},
+ {{"Remote URL", "^http:.*$"}});
--- End diff --
I see this comment chain can be a bit confusing -- there were offline
discussions that make it a bit worse. My request for a separate ticket was that
this and other PRs have made the usage of these classes very difficult to read
and use. I believe that it's bad enough that it warrants resolution in this
ticket or subsequent work. To make code worse and then fix it doesn't seem
correct; however, I'm okay with a subsequent ticket. I'm -1 with my opinion
merit especially since there was no discussion of this. I think Aldrin and
others can disagree and move forward with their +1s, through which I'll provide
an implicit agreement to move forward -- I'm also open to the possibility that
I'm completely wrong so let's actually have that discussion. My goal is not to
stop this but there was no discussion regarding how this and others have made
the code difficult to read and use.
---
[jira] [Commented] (MINIFICPP-570) Move external project builds into separate CMake module(s)
[ https://issues.apache.org/jira/browse/MINIFICPP-570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555621#comment-16555621 ] Andrew Christianson commented on MINIFICPP-570: --- This is a minor organization issue and is in no way a blocker. > Move external project builds into separate CMake module(s) > -- > > Key: MINIFICPP-570 > URL: https://issues.apache.org/jira/browse/MINIFICPP-570 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Andrew Christianson >Assignee: Andrew Christianson >Priority: Minor > > The main CMakeLists.txt is getting cluttered with external project builds. It > is time to move these out into their own space. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5452) Enable HDFS-13448 in HDFS Sink
[ https://issues.apache.org/jira/browse/NIFI-5452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated NIFI-5452: -- Description: Now that [HDFS-13448] is available, add a new boolean configuration to the HDFS Sink configuration that enabled this. The basic issue is, as it currently stands, is the following: Imagine a cluster has four racks of hardware # Rack A is half management nodes and half datanodes # Rack B, C, D are all datanodes Now consider the following scenarios: If an instance of NiFi is located on a server outside of these racks, the data will be evenly distributed to each DataNode. If an instance of NiFi is running on Rack A, and is running co-located with a DataNode, then all of the HDFS Sink writes will first go to the local DataNode, thus overloading this single DataNode and filling it faster than all other DataNodes in the cluster. If an instance of NiFi is running on Rack A, on its own server, then all of the HDFS Sink writes will first go to a DataNode on Rack A, thus overloading the DataNodes on Rack A and filling those DataNodes faster than all other DataNodes in the cluster. The issue here is compounded using many racks. Rack A will always receive one copy of the each block, and the other two copies are scattered equally across the other racks. [HDFS-13448] adds a new flag to the HDFS client that requests to the NameNode that the first block should always be randomly placed. Thus, if a NiFi instance is located on Rack A, the local node (or local rack) will not be overloaded. was:Now that [HDFS-13448] is available, add a new boolean configuration to the HDFS Sink configuration that enabled this. > Enable HDFS-13448 in HDFS Sink > -- > > Key: NIFI-5452 > URL: https://issues.apache.org/jira/browse/NIFI-5452 > Project: Apache NiFi > Issue Type: New Feature >Reporter: BELUGA BEHR >Priority: Major > > Now that [HDFS-13448] is available, add a new boolean configuration to the > HDFS Sink configuration that enabled this. > The basic issue is, as it currently stands, is the following: > Imagine a cluster has four racks of hardware > # Rack A is half management nodes and half datanodes > # Rack B, C, D are all datanodes > Now consider the following scenarios: > If an instance of NiFi is located on a server outside of these racks, the > data will be evenly distributed to each DataNode. > If an instance of NiFi is running on Rack A, and is running co-located with a > DataNode, then all of the HDFS Sink writes will first go to the local > DataNode, thus overloading this single DataNode and filling it faster than > all other DataNodes in the cluster. > If an instance of NiFi is running on Rack A, on its own server, then all of > the HDFS Sink writes will first go to a DataNode on Rack A, thus overloading > the DataNodes on Rack A and filling those DataNodes faster than all other > DataNodes in the cluster. The issue here is compounded using many racks. > Rack A will always receive one copy of the each block, and the other two > copies are scattered equally across the other racks. > [HDFS-13448] adds a new flag to the HDFS client that requests to the NameNode > that the first block should always be randomly placed. Thus, if a NiFi > instance is located on Rack A, the local node (or local rack) will not be > overloaded. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4521) MS SQL CDC Processor
[ https://issues.apache.org/jira/browse/NIFI-4521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555612#comment-16555612 ] ASF GitHub Bot commented on NIFI-4521: -- Github user mattyb149 commented on a diff in the pull request: https://github.com/apache/nifi/pull/2231#discussion_r205098598 --- Diff: nifi-assembly/pom.xml --- @@ -637,6 +637,12 @@ language governing permissions and limitations under the License. --> 1.7.0-SNAPSHOT nar + +org.apache.nifi +nifi-cdc-mssql-nar +1.7.0-SNAPSHOT --- End diff -- Sorry this has taken so long, can you rebase against the latest master and update the version(s) to 1.8.0-SNAPSHOT? Please and thanks! > MS SQL CDC Processor > > > Key: NIFI-4521 > URL: https://issues.apache.org/jira/browse/NIFI-4521 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Peter Wicks >Assignee: Peter Wicks >Priority: Major > > Creation of a new processor that reads Change Data Capture details from > Microsoft SQL Server and outputs the changes a Records. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2913: NIFI-5083: enabled EL support for processor configuration ...
Github user bbende commented on the issue: https://github.com/apache/nifi/pull/2913 Actually, I take back my last comment, I now see where the value is obtained and it does have the logic: ``` String propertyValue = propertyDescriptor.isExpressionLanguageSupported() ? context.getProperty(propertyDescriptor).evaluateAttributeExpressions().getValue() : context.getProperty(propertyDescriptor).getValue(); ``` Sorry about that. ---
[GitHub] nifi issue #2913: NIFI-5083: enabled EL support for processor configuration ...
Github user bbende commented on the issue: https://github.com/apache/nifi/pull/2913 I think you will also need to find where the group id property is used and ensure that evaluateAttributeExpressions() is called when obtaining the value. ---
[GitHub] nifi-minifi-cpp issue #347: MINIFIPP-514 Incorporated regex property validat...
Github user achristianson commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/347 @phrocker Please elaborate what exactly your issues are. As you can see from 20 days ago, all concerns were addressed to the best of my knowledge. ---
[GitHub] nifi pull request #2231: NIFI-4521 MS SQL CDC Processor
Github user mattyb149 commented on a diff in the pull request: https://github.com/apache/nifi/pull/2231#discussion_r205098598 --- Diff: nifi-assembly/pom.xml --- @@ -637,6 +637,12 @@ language governing permissions and limitations under the License. --> 1.7.0-SNAPSHOT nar + +org.apache.nifi +nifi-cdc-mssql-nar +1.7.0-SNAPSHOT --- End diff -- Sorry this has taken so long, can you rebase against the latest master and update the version(s) to 1.8.0-SNAPSHOT? Please and thanks! ---
