[jira] [Created] (NIFI-6466) PutHBaseJSON should be capable of handling output of ScanHBase or GetHBase
Julian Gimbel created NIFI-6466: --- Summary: PutHBaseJSON should be capable of handling output of ScanHBase or GetHBase Key: NIFI-6466 URL: https://issues.apache.org/jira/browse/NIFI-6466 Project: Apache NiFi Issue Type: Improvement Affects Versions: 1.9.2 Reporter: Julian Gimbel Currently it is not possible to simply put the extracted JSON from either ScanHBase or GetHBase into a PutHBase processor without parsing the data. There should be a common format between those processors which makes it easy to copy tables or tables extracts with only two processors (or three for the ScanHBase because it needs input). -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (NIFI-6024) Registry Buckets Inconsistently Sorted
[ https://issues.apache.org/jira/browse/NIFI-6024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831675#comment-16831675 ] Julian Gimbel commented on NIFI-6024: - I realized that it is in order in Firefox but sorted randomly for Chrome. Maybe that helps in solving the issue > Registry Buckets Inconsistently Sorted > -- > > Key: NIFI-6024 > URL: https://issues.apache.org/jira/browse/NIFI-6024 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.8.0 >Reporter: Alan Jackoway >Priority: Major > Labels: SDLC > > When importing a process group from a registry with multiple buckets, they > should always be in alphanumeric order. Currently they are inconsistent in > what order comes back. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-6240) Proxy Support for AzureEventHub processor via Websockets
[ https://issues.apache.org/jira/browse/NIFI-6240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825135#comment-16825135 ] Julian Gimbel commented on NIFI-6240: - Also check: https://stackoverflow.com/questions/55499848/nifi-consume-azure-event-hub-with-an-enterprise-proxy-environment > Proxy Support for AzureEventHub processor via Websockets > > > Key: NIFI-6240 > URL: https://issues.apache.org/jira/browse/NIFI-6240 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Julian Gimbel >Priority: Major > > The ConsumeAzureEventHub, GetAzureEventHub and PutAzureEventHub processors > should be upgraded to a newer > [version|https://github.com/Azure/azure-event-hubs-java/releases/] of the > Azure Client and support AMQP over Websockets which can use a proxy with > basic authentication as described in here: > [https://github.com/Azure/azure-event-hubs-java/releases/tag/1.2.0] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-6240) Proxy Support for AzureEventHub processor via Websockets
[ https://issues.apache.org/jira/browse/NIFI-6240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Gimbel updated NIFI-6240: Description: The ConsumeAzureEventHub, GetAzureEventHub and PutAzureEventHub processors should be upgraded to a newer [version|https://github.com/Azure/azure-event-hubs-java/releases/] of the Azure Client and support AMQP over Websockets which can use a proxy with basic authentication as described in here: [https://github.com/Azure/azure-event-hubs-java/releases/tag/1.2.0] was: The ConsumeAzureEventHub, GetAzureEventHub and PutAzureEventHub processors should be upgraded to a newer [version|https://github.com/Azure/azure-event-hubs-java/releases/] of the Azure Client and support AMQP over Websockets which can use a proxy with basic authentication as described in here: [https://blogs.msdn.microsoft.com/eventhubs/2018/09/21/azure-event-hubs-websockets-and-proxy-support/] > Proxy Support for AzureEventHub processor via Websockets > > > Key: NIFI-6240 > URL: https://issues.apache.org/jira/browse/NIFI-6240 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Julian Gimbel >Priority: Major > > The ConsumeAzureEventHub, GetAzureEventHub and PutAzureEventHub processors > should be upgraded to a newer > [version|https://github.com/Azure/azure-event-hubs-java/releases/] of the > Azure Client and support AMQP over Websockets which can use a proxy with > basic authentication as described in here: > [https://github.com/Azure/azure-event-hubs-java/releases/tag/1.2.0] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-6240) Proxy Support for AzureEventHub processor via Websockets
Julian Gimbel created NIFI-6240: --- Summary: Proxy Support for AzureEventHub processor via Websockets Key: NIFI-6240 URL: https://issues.apache.org/jira/browse/NIFI-6240 Project: Apache NiFi Issue Type: Improvement Reporter: Julian Gimbel The ConsumeAzureEventHub, GetAzureEventHub and PutAzureEventHub processors should be upgraded to a newer [version|https://github.com/Azure/azure-event-hubs-java/releases/] of the Azure Client and support AMQP over Websockets which can use a proxy with basic authentication as described in here: [https://blogs.msdn.microsoft.com/eventhubs/2018/09/21/azure-event-hubs-websockets-and-proxy-support/] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5931) GetHDFS Property File Filter Regex should use Expression Language
[
https://issues.apache.org/jira/browse/NIFI-5931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Gimbel updated NIFI-5931:
Description:
Currently if we want to get a 1 day old file every day where the filename
contains the date we need to get the whole folder of data and then filter the
files with routeOnAttribute. It would be good to use expression language to
create a file filter regex like:
.**${now():minus(8640):format('MMdd')}.**
was:
Currently if we want to get a 2 day old file every day where the filename
contains the date we need to get the whole folder of data and then filter the
files with routeOnAttribute. It would be good to use expression language to
create a file filter regex like:
.**${now():minus(8640):format('MMdd')}.**
> GetHDFS Property File Filter Regex should use Expression Language
> -
>
> Key: NIFI-5931
> URL: https://issues.apache.org/jira/browse/NIFI-5931
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.8.0
>Reporter: Julian Gimbel
>Priority: Minor
>
> Currently if we want to get a 1 day old file every day where the filename
> contains the date we need to get the whole folder of data and then filter the
> files with routeOnAttribute. It would be good to use expression language to
> create a file filter regex like:
> .**${now():minus(8640):format('MMdd')}.**
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5931) GetHDFS Property File Filter Regex should use Expression Language
[
https://issues.apache.org/jira/browse/NIFI-5931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Gimbel updated NIFI-5931:
Description:
Currently if we want to get a 2 day old file every day where the filename
contains the date we need to get the whole folder of data and then filter the
files with routeOnAttribute. It would be good to use expression language to
create a file filter regex like:
.**${now():minus(8640):format('MMdd')}.**
was:
Currently if we want to get a 2 day old file every day where the filename
contains the date we need to get the whole folder of data and then filter the
files with routeOnAttribute. It would be good to use expression language to
create a file filter regex like:
.*${now():minus(8640):format('MMdd')}.*
> GetHDFS Property File Filter Regex should use Expression Language
> -
>
> Key: NIFI-5931
> URL: https://issues.apache.org/jira/browse/NIFI-5931
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.8.0
>Reporter: Julian Gimbel
>Priority: Minor
>
> Currently if we want to get a 2 day old file every day where the filename
> contains the date we need to get the whole folder of data and then filter the
> files with routeOnAttribute. It would be good to use expression language to
> create a file filter regex like:
> .**${now():minus(8640):format('MMdd')}.**
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (NIFI-5931) GetHDFS Property File Filter Regex should use Expression Language
Julian Gimbel created NIFI-5931:
---
Summary: GetHDFS Property File Filter Regex should use Expression
Language
Key: NIFI-5931
URL: https://issues.apache.org/jira/browse/NIFI-5931
Project: Apache NiFi
Issue Type: Improvement
Components: Core Framework
Affects Versions: 1.8.0
Reporter: Julian Gimbel
Currently if we want to get a 2 day old file every day where the filename
contains the date we need to get the whole folder of data and then filter the
files with routeOnAttribute. It would be good to use expression language to
create a file filter regex like:
.*${now():minus(8640):format('MMdd')}.*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (NIFIREG-139) Nifi registry should pick up the controller service definitions while deploying a flow
[ https://issues.apache.org/jira/browse/NIFIREG-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605506#comment-16605506 ] Julian Gimbel edited comment on NIFIREG-139 at 9/6/18 9:18 AM: --- Two other possibilities come into my mind: 1. Add a new property to the CS that names something like "Mapping Name". That one can be used to check for equality within the different environments. That would mean changes in Nifi as well as in Nifi Registry. If that property is evaluated with the expression language it might give quite powerful deployment possibilities and much more flexibility. 2. As some of the CS (e.g. KerberosCredentialService) already allow for dynamic attributes it would also be possible to use a specifically named dynamic attribute. The name of that specific attribute can be provided to the CLI as an optional parameter that would be used to check for name equality. The disadvantage is, that dynamic attributes do not allow for expression language and not all CS support dynamic attributes. Let me know what you think was (Author: jugi): Two other possibilities come into my mind: 1. Add a new property to the CS that names something like "Mapping Name". That one can be used to check for equality within the different environments. That would mean changes in Nifi as well as in Nifi Registry. If that property is evaluated with the expression language it might give quite powerful deployment possibilities and much more flexibility. 2. As some of the CS already allow for dynamic attributes it would also be possible to use a specifically named dynamic attribute. The name of that specific attribute can be provided to the CLI as an optional parameter that would be used to check for name equality. The disadvantage is, that dynamic attributes do not allow for expression language and not all CS support dynamic attributes. Let me know what you think > Nifi registry should pick up the controller service definitions while > deploying a flow > -- > > Key: NIFIREG-139 > URL: https://issues.apache.org/jira/browse/NIFIREG-139 > Project: NiFi Registry > Issue Type: Bug >Affects Versions: 0.1.0 >Reporter: ohad israeli >Priority: Critical > > I am trying to use the registry to copy a simple flow for example of > httprequest->joltconversion->httpresponse. > This flow is using the internal http context service which is installed and > configured on the source (dev env). While deploying the flow to the target > (prod env) the flow cant find the service on the target (prod end) > to summarize, on deployment of a flow if the flow is using a service > controller its setting should also be deployed on the target machine -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (NIFIREG-139) Nifi registry should pick up the controller service definitions while deploying a flow
[ https://issues.apache.org/jira/browse/NIFIREG-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605506#comment-16605506 ] Julian Gimbel edited comment on NIFIREG-139 at 9/6/18 9:16 AM: --- Two other possibilities come into my mind: 1. Add a new property to the CS that names something like "Mapping Name". That one can be used to check for equality within the different environments. That would mean changes in Nifi as well as in Nifi Registry. If that property is evaluated with the expression language it might give quite powerful deployment possibilities and much more flexibility. 2. As some of the CS already allow for dynamic attributes it would also be possible to use a specifically named dynamic attribute. The name of that specific attribute can be provided to the CLI as an optional parameter that would be used to check for name equality. The disadvantage is, that dynamic attributes do not allow for expression language and not all CS support dynamic attributes. Let me know what you think was (Author: jugi): Two other possibilities come into my mind: # is to add a new property to the CS that names something like "Mapping Name". That one can be used to check for equality within the different environments. That would mean changes in Nifi as well as in Nifi Registry. If that property is evaluated with the expression language it might give quite powerful deployment possibilities and much more flexibility. # As some of the CS already allow for Dynamic Attributes it would also be possible to use a specifically named dynamic attribute. The name of that specific attribute can be provided to the CLI as an optional parameter that would be used to check for name equality. The disadvantage is, that Dynamic Attributes do not allow > Nifi registry should pick up the controller service definitions while > deploying a flow > -- > > Key: NIFIREG-139 > URL: https://issues.apache.org/jira/browse/NIFIREG-139 > Project: NiFi Registry > Issue Type: Bug >Affects Versions: 0.1.0 >Reporter: ohad israeli >Priority: Critical > > I am trying to use the registry to copy a simple flow for example of > httprequest->joltconversion->httpresponse. > This flow is using the internal http context service which is installed and > configured on the source (dev env). While deploying the flow to the target > (prod env) the flow cant find the service on the target (prod end) > to summarize, on deployment of a flow if the flow is using a service > controller its setting should also be deployed on the target machine -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFIREG-139) Nifi registry should pick up the controller service definitions while deploying a flow
[ https://issues.apache.org/jira/browse/NIFIREG-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605506#comment-16605506 ] Julian Gimbel commented on NIFIREG-139: --- Two other possibilities come into my mind: # is to add a new property to the CS that names something like "Mapping Name". That one can be used to check for equality within the different environments. That would mean changes in Nifi as well as in Nifi Registry. If that property is evaluated with the expression language it might give quite powerful deployment possibilities and much more flexibility. # As some of the CS already allow for Dynamic Attributes it would also be possible to use a specifically named dynamic attribute. The name of that specific attribute can be provided to the CLI as an optional parameter that would be used to check for name equality. The disadvantage is, that Dynamic Attributes do not allow > Nifi registry should pick up the controller service definitions while > deploying a flow > -- > > Key: NIFIREG-139 > URL: https://issues.apache.org/jira/browse/NIFIREG-139 > Project: NiFi Registry > Issue Type: Bug >Affects Versions: 0.1.0 >Reporter: ohad israeli >Priority: Critical > > I am trying to use the registry to copy a simple flow for example of > httprequest->joltconversion->httpresponse. > This flow is using the internal http context service which is installed and > configured on the source (dev env). While deploying the flow to the target > (prod env) the flow cant find the service on the target (prod end) > to summarize, on deployment of a flow if the flow is using a service > controller its setting should also be deployed on the target machine -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFIREG-139) Nifi registry should pick up the controller service definitions while deploying a flow
[ https://issues.apache.org/jira/browse/NIFIREG-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602697#comment-16602697 ] Julian Gimbel commented on NIFIREG-139: --- Hi, we are versioning and deploying some process groups that reference controller services (CS) that are defined outside of the process group e.g. KeytabCredentialsServices for QA and Production. Now if we want to deploy the process group we still need to manually select the correct CS in the processors on production like for example in PutHDFS. Would it be possible to include a “map cs by name” parameter to the CLI so that it walks through the processors and if it finds a CS with the same name as in the source it will use that one? I know that currently only the ID is stored, but that one differs between the two Nifi instances. Best regards Julian > Nifi registry should pick up the controller service definitions while > deploying a flow > -- > > Key: NIFIREG-139 > URL: https://issues.apache.org/jira/browse/NIFIREG-139 > Project: NiFi Registry > Issue Type: Bug >Affects Versions: 0.1.0 >Reporter: ohad israeli >Priority: Critical > > I am trying to use the registry to copy a simple flow for example of > httprequest->joltconversion->httpresponse. > This flow is using the internal http context service which is installed and > configured on the source (dev env). While deploying the flow to the target > (prod env) the flow cant find the service on the target (prod end) > to summarize, on deployment of a flow if the flow is using a service > controller its setting should also be deployed on the target machine -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5532) CRON Scheduling with timezone
Julian Gimbel created NIFI-5532: --- Summary: CRON Scheduling with timezone Key: NIFI-5532 URL: https://issues.apache.org/jira/browse/NIFI-5532 Project: Apache NiFi Issue Type: Improvement Reporter: Julian Gimbel Currently it is possible to schedule processors based on a CRON trigger. This is fine as long as we only work in one timezone or in timezones that do not shift between summer and winter time. If processes should be scheduled at the same time the processors need to be rescheduled when timezones are switching from summer to winter time. As pointed out in [https://stackoverflow.com/questions/40212696/apache-nifi-how-to-pass-the-timezone-into-the-crontab-string] Nifi uses quartz scheduler under the hood which already supports Timezones that could be used. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4933) GCS Google Cloud Storage Processor requires Project ID even if it is not necessary
[ https://issues.apache.org/jira/browse/NIFI-4933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576161#comment-16576161 ] Julian Gimbel commented on NIFI-4933: - This issue is solved in NIFI-4731 > GCS Google Cloud Storage Processor requires Project ID even if it is not > necessary > -- > > Key: NIFI-4933 > URL: https://issues.apache.org/jira/browse/NIFI-4933 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.5.0 >Reporter: Julian Gimbel >Priority: Minor > > Google Cloud Storage Processsors like List or Fetch require the Project ID, > but in the Google Cloud Storage Code the Project ID is only required when > creating a new Bucket: > [https://github.com/GoogleCloudPlatform/google-cloud-java/blob/e3908826d28d24a5dd68866f1177994050dbe766/google-cloud-storage/src/main/java/com/google/cloud/storage/StorageOptions.java#L109] > With the project ID required we can not download data from buckets that we do > not own like for example public data sets: > [https://console.cloud.google.com/storage/browser/gcp-public-data-landsat/?_ga=2.37550372.-565124473.1518597165] > Solution would be to make the Project ID optional and enable the download of > such data sets. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFIREG-188) Login by hitting Enter in the password field in web UI
Julian Gimbel created NIFIREG-188: - Summary: Login by hitting Enter in the password field in web UI Key: NIFIREG-188 URL: https://issues.apache.org/jira/browse/NIFIREG-188 Project: NiFi Registry Issue Type: Improvement Affects Versions: 0.2.0, 0.1.0 Reporter: Julian Gimbel While logging in to Nifi Registry it should be possible to hit enter in the password field to login instead of using the mouse or tab on to the login button. Unfortunately I am not experienced enough with angular to do it myself. It is not as simple as just adding a tag around the dialog fields and buttons. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFIREG-187) Add to Migration Guidance: Copy flow_storage directory
Julian Gimbel created NIFIREG-187:
-
Summary: Add to Migration Guidance: Copy flow_storage directory
Key: NIFIREG-187
URL: https://issues.apache.org/jira/browse/NIFIREG-187
Project: NiFi Registry
Issue Type: Improvement
Affects Versions: 0.2.0, 0.1.0
Reporter: Julian Gimbel
When we tried to migrate H2 Database from 0.1.0 to 0.2.0 we got the problem
that the nifi-registry-app.log showed that all our buckets flows and flow
snapshots were successfully copied from legacy to new database.
However when we wanted to import or refresh the repository we could select the
flows from the dropdown list, but got an error when we clicked on import.
The log showed the following:
{code:java}
2018-07-26 07:51:21,859 WARN [NiFi Registry Web Server-36]
o.a.n.registry.service.RegistryService The specified flow id
[d7d8ed4b-68e4-4d88-932c-486863a9f194] does not exist.
2018-07-26 07:51:21,860 INFO [NiFi Registry Web Server-36]
o.a.n.r.w.m.ResourceNotFoundExceptionMapper
org.apache.nifi.registry.exception.ResourceNotFoundException: The specified
flow ID does not exist in this bucket.. Returning Not Found response.{code}
Only after we found this conversation in the [Upgrade Mailing List
|http://apache-nifi-developer-list.39713.n7.nabble.com/Upgrading-NiFi-Registry-tp19098p19215.html]
we figured out that we also need to copy HOME/flow_storage directory as the
database only contains the metadata but the flow_storage contains the actual
data.
It might help people if this peace of information would be added to the
[migration
guidance|https://cwiki.apache.org/confluence/display/NIFIREG/Migration+Guidance].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (NIFIREG-179) Move Flow to other buckets and rename them
Julian Gimbel created NIFIREG-179: - Summary: Move Flow to other buckets and rename them Key: NIFIREG-179 URL: https://issues.apache.org/jira/browse/NIFIREG-179 Project: NiFi Registry Issue Type: Improvement Affects Versions: 0.2.0, 0.1.0 Reporter: Julian Gimbel It should be possible to move a versioned flow from one bucket to another in the UI. In addition it should be possible to change the Flow Name for versioned flows. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5282) GCPProcessor with HTTP Proxy with Authentication
Julian Gimbel created NIFI-5282: --- Summary: GCPProcessor with HTTP Proxy with Authentication Key: NIFI-5282 URL: https://issues.apache.org/jira/browse/NIFI-5282 Project: Apache NiFi Issue Type: Improvement Affects Versions: 1.6.0 Reporter: Julian Gimbel The [AbstractGCPProcessor |https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/AbstractGCPProcessor.java] already accepts http proxy settings but it but be even better if it accepts authenticated proxies with user and password aswell. In the best case it would support the ProxyService introduced in [NIFI-4199|https://issues.apache.org/jira/projects/NIFI/issues/NIFI-4199] and all of its options. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-4991) H2 Database Repository should be moved from default directory
Julian Gimbel created NIFI-4991: --- Summary: H2 Database Repository should be moved from default directory Key: NIFI-4991 URL: https://issues.apache.org/jira/browse/NIFI-4991 Project: Apache NiFi Issue Type: Improvement Components: Documentation Website Affects Versions: 1.5.0 Reporter: Julian Gimbel In the admin-guide it should be mentioned that property: nifi.database.directory should also be changed to some other folder from its default location <_root-level-nifi-dir>_/database_repository It should be marked with an asterisk according to this post: [https://community.hortonworks.com/articles/7882/hdfnifi-best-practices-for-setting-up-a-high-perfo.html] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-4933) GCS Google Cloud Storage Processor requires Project ID even if it is not necessary
Julian Gimbel created NIFI-4933: --- Summary: GCS Google Cloud Storage Processor requires Project ID even if it is not necessary Key: NIFI-4933 URL: https://issues.apache.org/jira/browse/NIFI-4933 Project: Apache NiFi Issue Type: Improvement Components: Core Framework Affects Versions: 1.5.0 Reporter: Julian Gimbel Google Cloud Storage Processsors like List or Fetch require the Project ID, but in the Google Cloud Storage Code the Project ID is only required when creating a new Bucket: [https://github.com/GoogleCloudPlatform/google-cloud-java/blob/e3908826d28d24a5dd68866f1177994050dbe766/google-cloud-storage/src/main/java/com/google/cloud/storage/StorageOptions.java#L109] With the project ID required we can not download data from buckets that we do not own like for example public data sets: [https://console.cloud.google.com/storage/browser/gcp-public-data-landsat/?_ga=2.37550372.-565124473.1518597165] Solution would be to make the Project ID optional and enable the download of such data sets. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4918) JMS Connection Factory setting the dynamic Properties wrong
[ https://issues.apache.org/jira/browse/NIFI-4918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386108#comment-16386108 ] Julian Gimbel commented on NIFI-4918: - https://github.com/apache/nifi/pull/2499 > JMS Connection Factory setting the dynamic Properties wrong > --- > > Key: NIFI-4918 > URL: https://issues.apache.org/jira/browse/NIFI-4918 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.3.0, 1.4.0, 1.5.0 >Reporter: Julian Gimbel >Priority: Minor > > When trying to set the Property setSSLTrustedCertificate for the tibco jms > Connection Factory the process will sometimes fail, because this Method is > implemented three times and accepts different parameters. Therefor we should > implement a fix that checks through the methods if one of it requires the > type of parameter that was provided by the user. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-4918) JMS Connection Factory setting the dynamic Properties wrong
Julian Gimbel created NIFI-4918: --- Summary: JMS Connection Factory setting the dynamic Properties wrong Key: NIFI-4918 URL: https://issues.apache.org/jira/browse/NIFI-4918 Project: Apache NiFi Issue Type: Improvement Components: Core Framework Affects Versions: 1.5.0, 1.4.0, 1.3.0 Reporter: Julian Gimbel When trying to set the Property setSSLTrustedCertificate for the tibco jms Connection Factory the process will sometimes fail, because this Method is implemented three times and accepts different parameters. Therefor we should implement a fix that checks through the methods if one of it requires the type of parameter that was provided by the user. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (NIFI-4197) *GCP processors do not expose proxy settings
[
https://issues.apache.org/jira/browse/NIFI-4197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365504#comment-16365504
]
Julian Gimbel edited comment on NIFI-4197 at 2/15/18 1:14 PM:
--
Two additional attributes proxy_user and proxy_password for authenticated proxy
usage would be awesome:
{code:java}
final String authUser = "user";
final String authPassword = "password";
Authenticator.setDefault(
new Authenticator() {
@Override
public PasswordAuthentication getPasswordAuthentication() {
return new PasswordAuthentication(
authUser, authPassword.toCharArray()
);
}
}
);
{code}
was (Author: jugi):
Two additional attributes proxy_user and proxy_password for authenticated proxy
usage would be awesome:
final String authUser = "user";
final String authPassword = "password";
Authenticator.setDefault( new Authenticator() {
@Override public PasswordAuthentication getPasswordAuthentication() {
return new PasswordAuthentication( authUser,
authPassword.toCharArray()); } } );
> *GCP processors do not expose proxy settings
>
>
> Key: NIFI-4197
> URL: https://issues.apache.org/jira/browse/NIFI-4197
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Andre F de Miranda
>Assignee: Andre F de Miranda
>Priority: Major
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-4197) *GCP processors do not expose proxy settings
[
https://issues.apache.org/jira/browse/NIFI-4197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365504#comment-16365504
]
Julian Gimbel commented on NIFI-4197:
-
Two additional attributes proxy_user and proxy_password for authenticated proxy
usage would be awesome:
final String authUser = "user";
final String authPassword = "password";
Authenticator.setDefault( new Authenticator() {
@Override public PasswordAuthentication getPasswordAuthentication() {
return new PasswordAuthentication( authUser,
authPassword.toCharArray()); } } );
> *GCP processors do not expose proxy settings
>
>
> Key: NIFI-4197
> URL: https://issues.apache.org/jira/browse/NIFI-4197
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Andre F de Miranda
>Assignee: Andre F de Miranda
>Priority: Major
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-4747) GetHDFS Scheduling via CRON shows an error before it is scheduled
[ https://issues.apache.org/jira/browse/NIFI-4747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318136#comment-16318136 ] Julian Gimbel commented on NIFI-4747: - [~pvillard] I would prefer if you could submit a PR. Thanks in advance > GetHDFS Scheduling via CRON shows an error before it is scheduled > - > > Key: NIFI-4747 > URL: https://issues.apache.org/jira/browse/NIFI-4747 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.2.0, 1.4.0 > Environment: Linux, Windows >Reporter: Julian Gimbel >Priority: Blocker > Attachments: gethdfs_cron_scheduling.PNG, > gethdfs_cron_scheduling_error.png > > > We schedule a getHDFS Processor CRON driven at "0 0 15 * * ?" so that it > should run at 15:00:00 every day. If we start the processor it gives an error > message even before CRON should have triggered the Processor as shown below. > The question is why that is happening and how can it be fixed? > 2018-01-08 14:43:01,242 ERROR [StandardProcessScheduler Thread-5] > o.a.n.controller.StandardProcessorNode Failed to invoke @OnScheduled method > due to java.lang.RuntimeException: Failed while executing one of processor's > OnScheduled task. > java.lang.RuntimeException: Failed while executing one of processor's > OnScheduled task. > at > org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1483) > at > org.apache.nifi.controller.StandardProcessorNode.access$000(StandardProcessorNode.java:103) > at > org.apache.nifi.controller.StandardProcessorNode$1.run(StandardProcessorNode.java:1302) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.util.concurrent.ExecutionException: > java.lang.reflect.InvocationTargetException > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:206) > at > org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1466) > ... 9 common frames omitted > Caused by: java.lang.reflect.InvocationTargetException: null > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:137) > at > org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:125) > at > org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:70) > at > org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:47) > at > org.apache.nifi.controller.StandardProcessorNode$1$1.call(StandardProcessorNode.java:1306) > at > org.apache.nifi.controller.StandardProcessorNode$1$1.call(StandardProcessorNode.java:1302) > ... 6 common frames omitted > Caused by: java.io.IOException: PropertyDescriptor > PropertyDescriptor[Directory] has invalid value > /nifi/pele/pele2rtdp/20180108. The directory does not exist. > at > org.apache.nifi.processors.hadoop.GetHDFS.onScheduled(GetHDFS.java:242) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-4747) GetHDFS Scheduling via CRON shows an error before it is scheduled
[ https://issues.apache.org/jira/browse/NIFI-4747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16316631#comment-16316631 ] Julian Gimbel commented on NIFI-4747: - I understand that when we execute the processor and the folder does not exist it needs to throw an error. But I guess there are several use cases where the folder contains the date and is only created once the new data is available. Therefor this processor throws more errors then necessary and imho it would be better to evaluate the path only when the processor is really supposed to run. If that is not possible it would be fine to evaluate the path once when the processor is stored and not all the time that it is in running mode but not really scheduled. What do you think? > GetHDFS Scheduling via CRON shows an error before it is scheduled > - > > Key: NIFI-4747 > URL: https://issues.apache.org/jira/browse/NIFI-4747 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.2.0, 1.4.0 > Environment: Linux, Windows >Reporter: Julian Gimbel >Priority: Blocker > Attachments: gethdfs_cron_scheduling.PNG, > gethdfs_cron_scheduling_error.png > > > We schedule a getHDFS Processor CRON driven at "0 0 15 * * ?" so that it > should run at 15:00:00 every day. If we start the processor it gives an error > message even before CRON should have triggered the Processor as shown below. > The question is why that is happening and how can it be fixed? > 2018-01-08 14:43:01,242 ERROR [StandardProcessScheduler Thread-5] > o.a.n.controller.StandardProcessorNode Failed to invoke @OnScheduled method > due to java.lang.RuntimeException: Failed while executing one of processor's > OnScheduled task. > java.lang.RuntimeException: Failed while executing one of processor's > OnScheduled task. > at > org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1483) > at > org.apache.nifi.controller.StandardProcessorNode.access$000(StandardProcessorNode.java:103) > at > org.apache.nifi.controller.StandardProcessorNode$1.run(StandardProcessorNode.java:1302) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.util.concurrent.ExecutionException: > java.lang.reflect.InvocationTargetException > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:206) > at > org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1466) > ... 9 common frames omitted > Caused by: java.lang.reflect.InvocationTargetException: null > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:137) > at > org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:125) > at > org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:70) > at > org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:47) > at > org.apache.nifi.controller.StandardProcessorNode$1$1.call(StandardProcessorNode.java:1306) > at > org.apache.nifi.controller.StandardProcessorNode$1$1.call(StandardProcessorNode.java:1302) > ... 6 common frames omitted > Caused by: java.io.IOException: PropertyDescriptor > PropertyDescriptor[Directory] has invalid value > /nifi/pele/pele2rtdp/20180108. The directory does not exist. > at > org.apache.nifi.processors.hadoop.GetHDFS.onScheduled(GetHDFS.java:242) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (NIFI-4747) GetHDFS Scheduling via CRON shows an error before it is scheduled
Julian Gimbel created NIFI-4747: --- Summary: GetHDFS Scheduling via CRON shows an error before it is scheduled Key: NIFI-4747 URL: https://issues.apache.org/jira/browse/NIFI-4747 Project: Apache NiFi Issue Type: Bug Components: Core Framework Affects Versions: 1.4.0, 1.2.0 Environment: Linux, Windows Reporter: Julian Gimbel Priority: Blocker Attachments: gethdfs_cron_scheduling.PNG, gethdfs_cron_scheduling_error.png We schedule a getHDFS Processor CRON driven at "0 0 15 * * ?" so that it should run at 15:00:00 every day. If we start the processor it gives an error message even before CRON should have triggered the Processor as shown below. The question is why that is happening and how can it be fixed? 2018-01-08 14:43:01,242 ERROR [StandardProcessScheduler Thread-5] o.a.n.controller.StandardProcessorNode Failed to invoke @OnScheduled method due to java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task. java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task. at org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1483) at org.apache.nifi.controller.StandardProcessorNode.access$000(StandardProcessorNode.java:103) at org.apache.nifi.controller.StandardProcessorNode$1.run(StandardProcessorNode.java:1302) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:206) at org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1466) ... 9 common frames omitted Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:137) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:125) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:70) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:47) at org.apache.nifi.controller.StandardProcessorNode$1$1.call(StandardProcessorNode.java:1306) at org.apache.nifi.controller.StandardProcessorNode$1$1.call(StandardProcessorNode.java:1302) ... 6 common frames omitted Caused by: java.io.IOException: PropertyDescriptor PropertyDescriptor[Directory] has invalid value /nifi/pele/pele2rtdp/20180108. The directory does not exist. at org.apache.nifi.processors.hadoop.GetHDFS.onScheduled(GetHDFS.java:242) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (NIFI-4568) Working Dir Property not shown nor mentioned in ExecuteProcess Processor
Julian Gimbel created NIFI-4568: --- Summary: Working Dir Property not shown nor mentioned in ExecuteProcess Processor Key: NIFI-4568 URL: https://issues.apache.org/jira/browse/NIFI-4568 Project: Apache NiFi Issue Type: Bug Components: Configuration Affects Versions: 1.4.0, 1.3.0 Reporter: Julian Gimbel When executing a process with the executeProcess Operator the default working directory is the home directory of the user that started Nifi. In the code there is a Working directory Property that seems to work if one adds it. The documentation does not mention that it reads that argument and it is not shown in the Properties of the Operator. It should either be added in the documentation of the operator or directly in the Properties with home directory as the default value. See Line 100 to 110 in [https://github.com/apache/nifi/blob/6279fd41844205a14cf1f2fb225ead8881b197eb/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteProcess.java] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-4175) Add Proxy Properties to SFTP Processors
[ https://issues.apache.org/jira/browse/NIFI-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16207664#comment-16207664 ] Julian Gimbel commented on NIFI-4175: - Hi, We would really like to see that the development goes into the direction that SOCKS Proxy with authentication will also be available. In many bigger companies it is a basic requirement and for us it is not possible to provide those parameters through the -D Options as mentioned in this comment: https://community.hortonworks.com/questions/30339/how-to-configure-proxy-server-details-with-user-an.html As JSCH offers all the possibilities and a reference implementation it should be possible to integrate it and broaden the functionality of Nifi and thus make it more attractive to companies. Please feel free to discuss my opinion. Best regards > Add Proxy Properties to SFTP Processors > --- > > Key: NIFI-4175 > URL: https://issues.apache.org/jira/browse/NIFI-4175 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Grant Langlois >Assignee: Andre F de Miranda >Priority: Minor > > Add proxy server configuration as properties to the Nifi SFTP components. > Specifically add properties for: > Proxy Type: JSCH supported proxies including SOCKS4, SOCKS5 and HTTP > Proxy Host > Proxy Port > Proxy Username > Proxy Password > This would allow these properties to be configured for each processor. These > properties would align with what is configurable for the JSCH session and > shouldn't require any additional dependencies. > This proposal is similar to what is already implemented for the FTP processors -- This message was sent by Atlassian JIRA (v6.4.14#64029)
