SIGSEGV in Jetty

Hi there, I am getting regular (maybe every ten minutes?) crashes in NiFi 1.3.0. This just started happening unrelated to any change to the software environment (i.e.: we haven't installed new code/processors). It roughly coincides with an increase in flow file throughout and my subsequent use

Re: Apache Nifi - How to pass maven contrib-check after adding text file to resources

Mans See here for an example [1] The Apache RAT Plugin is what actually would detect the files and check their licenses/etc.. In the provided example we're excluding a couple test files because they cannot have headers but they are legit. You'd want to do the same most likely. Thanks [1]

Re: SIGSEGV in Jetty

Phil, I'd definitely say not any kind of known issue. One thing you might want to check is whether you're using G1 GC. We have in most cases moved away from that in Java8 because of some long standing bugs that our Lucene indexing for our provenance data which could trigger it. We've switch to

Is there a configuration to limit the size of nifi's flowfile repository

Hi guys, I checked NIFI's system administrator guide trying to find a configuration item so that the size of the flowfile repository could be limited similar to the other repositories(e.g. content repository), but I didn't find such configuration items, is there currently any configuration to

Custom Controller Service

Hi, Basically, there are 'n' number of databases that we are dealing with. We need to fetch the data from the source database into HDFS. Now since we are dealing with many databases, the source database is not static and changes every now and then. And every time the source database changes we

Pushing flows to Registry with Sensitive Information

Hi Guys, so I was playing with the registry and If I pushed a Processor that has sensitive information like a password it will be discarded when pulling it from the Registry, which is fine. Now comes the but. But if I put a variable there IMHO I think it should save it on the registry. What

Re: Custom Controller Service

Shot in the dark here, but what you try to do is create a custom connection pool service that uses dynamic properties to build a "pool of connection pools." You could then use the property names as hints for where to send the queries. On Wed, Apr 25, 2018 at 6:19 AM Rishab Prasad

Re: Custom Controller Service

Hello, Others who have worked on the DB related services and processors can correct me if I'm wrong here, but... In general the idea of a connection pool is that creating connections is somewhat expensive, and for a high-volume of operations you don't want to create a connection for each DB

Re: Pushing flows to Registry with Sensitive Information

Jorge, Currently variables are not meant to store sensitive information, the reason has to do with how users access variables... The way a user accesses a variable is via expression language, and since EL is just free from text entered into a property descriptor, it is impossible to restrict

Re: Is there a configuration to limit the size of nifi's flowfile repository

Hi Ben, Since the flow file repository contains the information of the flow files currently being processed by NiFi, you don't want to limit that repository in size since it would prevent the workflows to create new flow files. Besides this repository is very lightweight, I'm not sure it'd need

Re: Custom Controller Service

Or you could just call every time you needed properties more likely. This would still be custom unless integrated…. On April 25, 2018 at 08:26:57, Otto Fowler (ottobackwa...@gmail.com) wrote: Can services work with other controller services? Maybe a PropertiesControllerService,

Search for Controller Service UUID

When I search for a Controller Service by UUID using the search on the toolbar, only processors which reference the service are listed, not the service itself. Similarly, when selecting the UUID from the Bulletin Board, it reports "Error: Unable to find the specified component". Is this by

Re: Custom Controller Service

Option 2 and 3 seem to be a probable approach. However creating varying number of connections based on *each* flowfile still sounds to be suboptimal. If the requirement still demands to take that road, then it’s better to do some prep-work.. as in the list of probable connections that are required

Re: status bar counts on a cluster

This seems to have slipped through the cracks; I haven't seen a response. Does anyone have input? Thanks, Mark On Fri, Apr 20, 2018 at 10:40 AM, Mark Bean wrote: > On a cluster, the status bar reports 4 invalid processors. However, on > some nodes there are actually 6

Re: Custom Controller Service

To Otto's question... For simplicity sake, there is a new implementation of DBCPConnectionPool that behind the scenes has two connection pools, one for DB A and one for DB B, it doesn't matter how these are configured. Now a flow file comes into the ExecuteSQL and it goes to

Re: Custom Controller Service

Can services work with other controller services? Maybe a PropertiesControllerService, FilePropertiesControllerService could work with your service? the PCS could fire events on property changes etc. On April 25, 2018 at 08:05:27, Mike Thomsen (mikerthom...@gmail.com) wrote: Shot in the dark

Re: Custom Controller Service

If any controller service optionally supported this external service ( like the AWS processors optional support the credentials service ) then there is no need for the processor to change though right? On April 25, 2018 at 08:37:50, Bryan Bende (bbe...@gmail.com) wrote: The issue here is more

Re: Is there a configuration to limit the size of nifi's flowfile repository

I have a feeling that what Ben meant was how to limit the content repository size. On Wed, Apr 25, 2018 at 8:26 AM Pierre Villard wrote: > Hi Ben, > > Since the flow file repository contains the information of the flow files > currently being processed by NiFi, you

Re: Is there a configuration to limit the size of nifi's flowfile repository

No, he actually had mentioned “like content repository”. The answer is, there aren’t any properties that support this, AFAIK. Pierre’s response pretty much sums up why there aren’t any properties. Thanks, Sivaprasanna On Wed, 25 Apr 2018 at 7:10 PM, Mike Thomsen wrote:

Re: Custom Controller Service

Here is a proposal for how to modify the existing API to support both scenarios: https://issues.apache.org/jira/browse/NIFI-5121 The scope of that ticket would be to make the interface change, and then update all of NiFi's DB processors to pass in the attribute map. Then a separate effort to

Re: Custom Controller Service

Chiming in a bit late on this, but we faced this same issue and got around it by implementing a custom controller service which acts as a "router" to different dbcp services. It exposes a method which given a uuid, returns back the DBCPservice that corresponds with that uuid if it exists using

Re: Custom Controller Service

Charlie, That is a really nice solution, thanks for sharing. If we make the API changes in that JIRA I just sent, I could see having a new implementation of DBCPService that does something very similar. Basically there could be "DelegatingDBCPService" which still implemented the same

Re: Custom Controller Service

Yes, this was just one idea based on Charlie's solution. I'm not saying that approach solves the original request in this email, I was just saying its another nice idea that could be easily implemented once we make the changes in the JIRA. There can be as many "dynamic" DBCPService

Support for accessing sensitive values safely

Hi Since flowfile attributes and VariableRegistry is not suitable (not safe, to be specific), developers have to rely on manually configuring the sensitive values on the components (Processors & ControllerServices). And during CI/CD (using flow registry), the sensitive information are dropped and

Re: Custom Controller Service

Okay.. but two questions: 1. We are passing the attribute 'db.id' that means, we'll be using 'UpdateAttribute' to do add that attribute to flowfile? 2. If we are to use 'UpdateAttribute' to set the value for 'db.id', we need to know before hand, right? - Sivaprasanna On Wed, Apr

Re: [DISCUSS] Support for accessing sensitive values safely

Hi Sivaprasanna, This was a topic that was briefly considered earlier in the lifecycle of the project, but was sidelined due to other developments. With the NiFi Registry project, there has been renewed interest in securing sensitive values in the flow and allowing for easier

Re: Is there a configuration to limit the size of nifi's flowfile repository

All, This is something I think we shouldn't dismiss so easily. While the FlowFile repo is lighter than the content repo, allowing it to grow too large can cause major problems. Specifically, an "overgrown" FlowFile repo may prevent a NiFi instance from coming back up after a restart due to the

Re: [DISCUSS] Support for accessing sensitive values safely

The policy model would need more thought, but the point would be that a user can select variable references they have been given permission to. In order to configure the processor that is referencing the variable, they already need write permissions to that processor, or some parent in the

Re: [DISCUSS] Support for accessing sensitive values safely

There is definitely room for improvement here. Keep in mind that often the sensitive information is specific to a given environment. For example you build a flow in dev with your db.password. You don't actually want your dev db password to be propagated to the next environment, but you do want to

Re: [DISCUSS] Support for accessing sensitive values safely

"It would provide a list of variables that are readable to the current user and one can be selected, just like allowable values or controller services.” A person may have rights to configure nifi without knowing the “value” of the secure db password ( for example ), but that doesn’t mean they