Re: Custom processors/controller services without Maven

Hello, It would really be best to use Maven as NiFi's NAR Maven plugin will ensure that you are building the proper structure and will make things easy for you. Otherwise it would be up to you to somehow create the appropriate NAR structure (and if it ever changes you would have to update your

Re: [DISCUSS] Extension Registry

> > > > > > > > > > > > > Bryan > > > > > > > > > > > > > > Very exciting to see this under way!!! We desperately have to get > > > our > > > > > > > core nifi build size way down and make

[DISCUSS] Extension Registry

All, We've needed the elusive extension registry for quite some time now, and with NiFi Registry I think we are in a good place to make some progress in this area. I've started looking into adding "extension bundles" to NiFi Registry as the next type of versioned item, along side the existing

Re: Status of MetricsReportingTask?

You should be able to make your own NAR with two modules like: nifi-cloudwatch-metrics - contains your service implementation with a provided dependency on nifi-metrics-reporter-service-api nifi-cloudwatch-metrics-nar - packages your implementation and has a NAR dependency on

Re: Using record path API to remove field values

I'm not sure that removing a field from record path makes sense since technically removing the field all together is a different schema. You could easily make two versions of the schema, one with all fields, and a second with a subset, then just use any record processor with a reader using schema

Re: New Standard Pattern - Put Exception that caused failure in an attribute

I think processors should really have well defined relationships for the error scenarios that need to be handled. Having the exception message is ok for a human who wants to see it, but in order to do anything with it in the flow you will have to have a bunch of parsing/interpreting of the message

Re: Local development and testing w/ kerberos

ed the paths to krb5.conf and the keytab and they're both > pointing to /tmp/docker-kdc > > Any ideas? > > Thanks, > > Mike > > > On Wed, Oct 24, 2018 at 10:28 AM Mike Thomsen > wrote: > > > Awesome, thanks Bryan! I'm halfway through that (got klist view) and it's &

Re: Local development and testing w/ kerberos

There is a docker-kdc project that is easy to use: https://bryanbende.com/development/2016/08/31/apache-nifi-1.0.0-kerberos-authentication It was made before docker for mac was good/popular and it previously relied on boot2docker, but I made the following modification to not use boot2docker

Re: Load issues

You’ll probably need to provide some more information like hardware specs (cpus/cores, memory, disks), number of nodes in the cluster, JVM heap size, configuration of repositories, specific processors that are bottle necks, etc. On Tue, Oct 23, 2018 at 10:02 PM Phil H wrote: > Yep - although it

Re: [VOTE] Release Apache NiFi 1.8.0 (RC3)

+1 (binding) - Ran through release helper successfully - Setup a secure cluster with test flows - Verified the SSLContextService functionality On Tue, Oct 23, 2018 at 12:45 PM Richard St. John wrote: > > +1 (non-binding) > > Deployed to a three-node secure cluster on EC2 using Amazon Linux

Re: Need help with Controller Service implementation

Usually this is due to a missing NAR dependency somewhere, double-check the dependencies between the processors NAR and service API NAR, and between the service impl NAR and service API NAR.

Re: Compiling custom processor

and uploading my simple processor. > > Thanks for the assistance > John McGinn > > ---- > On Tue, 10/16/18, Bryan Bende wrote: > > Subject: Re: Compiling custom processor > To: figgie...@sbcglobal.net > Date: Tuesday, October 16, 2018, 4:45 P

Re: Compiling custom processor

----- > On Tue, 10/16/18, Bryan Bende wrote: > > Subject: Re: Compiling custom processor > To: dev@nifi.apache.org > Date: Tuesday, October 16, 2018, 3:34 PM > > There shouldn't be any issue > doing that, generally you just copy the &g

Re: Compiling custom processor

copied Wait over, I renamed it from Wait > to MyProcessorWait to be unique, and not copied. I'm just > attempting changes to Wait without modifying the actual one. > Also, thanks on the META-INF/services file information. > > Now I can debug and test. > > Sincerely, > John McG

Re: Compiling custom processor

In general, if your processor uses a controller service then the processors pom file needs a provided dependency on the API of the CS, and your NAR pom needs a NAR dependency on the NAR where the CS API is. Example is shown here in the section linking processors and controller services -

Re: Unable to List Queue

i Bryan > Thanks for your response. > The user have all access including view the data at root processor level. It > works when is.cluster is false. It doesn’t work when is.cluster is true. > > Thanks, > Milan Das > > > On 10/15/18, 2:56 PM, "Bryan Bende" wrot

Re: Unable to List Queue

The error message is saying your user does not have permission to view the data for the given processor. There is a specific policy for viewing data which is described in the admin guide component policies [1], the policy named "view the data". I think you should be able to create the "view the

Re: Issue with ValidateRecord processor and Avro map data type

} } ] } On Wed, Oct 10, 2018 at 10:00 AM Maxime Jattiot wrote: > > Sure, here is the schema : > { > "name" : "test", > "type" : "record", > "fields" : [ { > "name" : "field1"

Re: Issue with ValidateRecord processor and Avro map data type

Hello, The screenshots did not come through, maybe just paste the schema and JSON as text in the email. Thanks. On Wed, Oct 10, 2018 at 6:57 AM Maxime Jattiot wrote: > Hello everyone, > > I am having hard time to validate a simple JSON message with an Avro > schema. My schema works with

Re: Error log in ListenTCP

o value set > Client Auth NONE > > > Thank you very much: > > > Thanks and Regards, > Rajesh Biswas | +91 9886433461 | www.bridgera.com > > -Original Message- > From: Bryan Bende [mailto:bbe...@gmail.com] > Sent: Tuesday, October 9, 2018 1:20 AM >

Re: Error log in ListenTCP

ListenTCP uses new-line characters to determine logical message boundaries, so the data itself does not have to be characters, but there has to be new-lines indicating the end of a logical piece of data. Imagine streaming the content of several images over the same connection, the client would

Re: Building Ambari packages for newer NiFi

There isn't really an existing way to install any version of Apache NiFi via Ambari, there is only the HDF stack provided by a vendor which includes a version of NiFi based on an apache release. You would need to create a management pack for Ambari [1] and also have a yum repo which contained the

Re: RPG question

> Concurrent Tasks: 7 > Batch Settings >Count: 2000 >Size: 1000MB >Duration: not set > > On Tue, Oct 2, 2018 at 3:47 PM Bryan Bende wrote: > > > It looks like there may have been images attached that didn't make it > > through. > > > > I'm

Re: RPG question

processor on cluster 2 (where rpg nodes were), we don't see any back > pressure buildup on cluster 1 and the data flows between the two instances > very fast. > > > [image: image.png] > [image: image.png] > > On Tue, Oct 2, 2018 at 11:55 AM Bryan Bende wrote: > >

Re: RPG question

Hello, Can you describe the setup a little bit more? Is this site-to-site between two separate NiFi instances, or is it within the same cluster for purposes of redistribution? If it's separate instances then what type of instances, clustered or standalone? Have you made any changes to the

Re: Any down sides to putting a controller service in the same package as a processor?

That is fine for the CS implementation, the CS API should be in it's own NAR. On Tue, Sep 25, 2018 at 2:14 PM Mike Thomsen wrote: > > I picked up NIFI-5224, which is for creating a Solr client service, and was > wondering if there are any gotchas or down sides to putting the controller > service

Re: [VOTE] Release Apache NiFi Registry 0.3.0

+1 (binding) - Verified everything in release helper - Tested integration between NiF & NiFi Registry in unsecure and secure modes Thanks for RM'ing! On Mon, Sep 24, 2018 at 9:16 AM Mark Payne wrote: > > +1 (binding) > > Validated hashes, build with contrib-check. Started and ensured that

Re: Change Flowfile Name

PM Rajesh Biswas wrote: > Will be the UUID unique if we restart NiFi Server and create a new > instance of Nifi Server. > Would you please suggest a way to get the UUID and pass on as attribute > > Thank you very much for your support. > > Regards, > Rajesh Biswas > &g

Re: Change Flowfile Name

Filename is just a string attribute that could be anything, so it is not unique. Every flow file has an attribute called uuid which is unique. NiFi could have a GetFile processor that picks up the same file over and over and could have 100 flow files all with the same filename, but with different

Re: simple username+password authentication

Hi Luke, Just to elaborate a little bit more, the options for authenticating users are... - Client certificates - LDAP username/password - Kerberos username/password - Kerberos SPNEGO - Knox SSO - Open ID Connect - Custom login identity provider As Pierre mentioned, the management of the users

Re: Change Flowfile Name

You shouldn't need to use the "Delete Attributes Expression" since that is a regex indicating flow file attributes to remove, and you are not trying to remove anything here. When you add properties, the name of the property is the attribute name you want to add/modify, and the value is the

Re: Suggestion required HTTP and S3

as | +91 9886433461 | www.bridgera.com > > > -Original Message- > From: Bryan Bende [mailto:bbe...@gmail.com] > Sent: Thursday, September 20, 2018 7:08 PM > To: dev@nifi.apache.org > Subject: Re: Suggestion required HTTP and S3 > > Hello, > > I'm not sure

Re: Nifi -1.7 -Insufficient Permissions Untrusted proxy CN=host1, OU=NIFI error on cluster node

nested exception is org.apache.nifi.authorization.exception.AuthorizerCreationException: org.apache.nifi.authorization.exception.AuthorizerCreationException: Unable to locate node CN=ohlvnfiap004dd.oh.dev.dat.aws.vz-connect.net, OU=NIFI to seed policies. This means

Re: Suggestion required HTTP and S3

its making 2 > requests simultaneously, i.e 2 request for 1 image. > > Thanks and Regards, > Rajesh Biswas | +91 9886433461 | www.bridgera.com > > > -Original Message- > From: Bryan Bende [mailto:bbe...@gmail.com] > Sent: Tuesday, September 11, 2018 7:34 PM >

Re: [DISCUSS] Stale PRs

I'm in favor of voting to formalize the process. As part of voting we would need a draft of whatever our guidelines/rules are, even if it is just a modified version of Beam or Metron's, but basically I think it needs to be clear what we are voting on. On Tue, Sep 18, 2018 at 12:52 PM Andy

Re: Suggestion required HTTP and S3

I think the existing processors such as HandleHttpRequest can be used. The body of the POST will become the flow file content, and the headers will become flow file attributes. After HandleHttpRequest you can use RouteOnAttribute to make a decision based on one of the headers (flow file

Re: Variable / Nifi-EL for RPG Endpoint?

on, or specifying the flow.gz directly as an >> option...both seem to have some issues and inconveniences. >> >> Thanks! >> Jon >> >> On Thu, Sep 6, 2018 at 4:01 PM, Bryan Bende wrote: >> >> Jon, >> >> The RPG URL is treate

Re: Variable / Nifi-EL for RPG Endpoint?

em to have some issues and inconveniences. > > Thanks! > Jon > > On Thu, Sep 6, 2018 at 4:01 PM, Bryan Bende wrote: > > Jon, > > The RPG URL is treated similar to sensitive properties or parent > controller services, meaning that after deploying your flow you can

Re: Variable / Nifi-EL for RPG Endpoint?

Jon, The RPG URL is treated similar to sensitive properties or parent controller services, meaning that after deploying your flow you can issue an update to the RPG to set the URL to given environment's value and that change will not be considered a change as far as version control and will be

Re: Re: Pheonix client jar required for HBase conn

r then I'd be stumped. -Bryan On Thu, Aug 23, 2018 at 3:16 PM, Martini, Adam wrote: > Bryan, > > Yes, an HBase client upgrade makes sense for the Java 10 upgrade path. > However, the NoClassDefFoundError is more mysterious and does concern me. > > Thanks, > Adam > >

Re: Re: Pheonix client jar required for HBase conn

ed to Pheonix and I do not understand why > adding the client would help resolve the class. > > Would you advise reverting to Java 8 until Java 10 is fully supported? > > Thanks, > > Adam > > On 8/23/18, 11:27 AM, "Bryan Bende" wrote: > > Adam, > >

Re: Pheonix client jar required for HBase conn

Adam, The ClassSize class comes from hbase-common [1] so I'm not sure how that would related to the Phoenix client JAR. What version of HBase was this against? The only case I know of that needs the Phoenix client jar is when Phoenix has been installed which then modifies the HBase config files

Re: "Node Group" property of FileAccessPolicyProvider

This sounds like a good idea to me. Just to clarify how this would work, in the file-based policy provider we'd have something like: admin cluster-nodes During start up the "cluster-nodes" group gets granted permission to /proxy. Then a separate piece of work would be to implement a

Re: Build Failure for project nifi-toolkit-cli

Looks like all of the links go to the same test failure for TestCLICompleter, which seems to be failing because it can't find a file in src/test/resources [1]. The part of the CLI that is doing to file completion is from the JLine library, so we don't really have control over making that work on

Re: How to Add DFM user /group ?

Hello, In the 0.x versions of NiFi there were specific roles such as DFM, but in 1.x there are no specific roles and everything is based on fine-grained access policies. The initial admin is given the ability to get into the UI and to create policies, which then lets them grant themselves

Re: Stability of the Record API

I don't think there are any stability issues with the record API, it is definitely recommended to use the record approach where it makes sense. That comment was probably put there on the first release and never removed, and now it has been 4-5 releases later. As a general comment to APIs, the

Re: SSLPeerUnverifiedException Hostname "xyz" not verified

The authorizers.xml supports many different options, such user group providers for file-based, ldap, or composite, and policy provider for file-based and ranger. The concept of inheritance only really applies to the file-based cases because in the other scenarios like LDAP and Ranger, the users

Re: Registry H2 to PostgreSQL migration

Jagrut, I believe you should have access to edit the NiFi Registry wiki now. Let us know if it doesn't work. Thanks, Bryan On Sat, Jun 23, 2018 at 8:28 PM, Jagrut Sharma wrote: > Thanks! My confluence username is jagrutsharma. > > -- > Jagrut > > On Sat, Jun 23, 2018 at 4:

Re: Registry H2 to PostgreSQL migration

> Jagrut > > On Sat, Jun 23, 2018 at 2:52 PM, Bryan Bende wrote: > > > Hello, > > > > This is actually expected behavior... > > > > The migration is only setup to migrate the database from 0.1.0 which was > > named nifi-registry.mv.db. > &g

Re: Registry H2 to PostgreSQL migration

Hello, This is actually expected behavior... The migration is only setup to migrate the database from 0.1.0 which was named nifi-registry.mv.db. If you have an H2 DB named nifi-registry-primary.mv.db this is the name of the new H2 DB in 0.2.0, so it’s not looking for this because the idea was

Re: NIFI Multiple Kerberos configuration

Java assumes there is one krb5.conf file loaded by the JVM. It looks for the system property java.security.krb5.conf or falls back to looking in well-known locations, but still only expects one [1]. NiFi requires you to set the location in nifi.properties and uses that value to set the system

Re: NiFi Registry - bucket security

Hello, I believe this is expected behavior because the special privileges give read and write to all buckets and take precedence over the bucket privileges. Generally not many users/groups would have those special privileges. The common use case is to give a NiFi server read to all buckets so it

Re: how to get status of Process Instance

Hello, Processors are basically started, stopped, invalid, or disabled. There aren't really states like failed or completed because a processor doesn't complete, it just runs until it is told to stop. The closest thing to a failure is when an error occurs processing a particular flow file which

Re: [VOTE] Release Apache NiFi 1.7.0

+1 (binding) Verified everything in release helper and ran some test flows, thanks for RM'ing! On Thu, Jun 21, 2018 at 9:11 AM, Mark Payne wrote: > +1 (binding). > > Thanks for volunteering to handle the RM duties this time around, Andy! > > Was able to verify the checksums, build with

Re: How to configure LDAP with HandleHTTPRequest Processor for User authentication and authorization

Hello, Since processors are extensions to the NiFi framework, they don't have a way to utilize the framework's authentication and authorization. The only option for HandleHttpRequest is to use 2-way TLS by providing an SSL Context Service with Client Auth required. Thanks, Bryan On Thu, Jun

Re: Default NiFi registry port

Hello, Since the port is configurable and can easily be changed I don't think we would plan to change it. There are also lots of people who are not running NiFi Registry on the same server as Spark History Server, so I don't think changing it just for that makes sense. Thanks, Bryan On Wed,

Re: Apache NiFi 1.7.0 RC1 Release Helper Guide

on't > think DEPENDENCIES matters either unless I'm missing something) > > On Wed, Jun 20, 2018 at 10:11 AM Bryan Bende wrote: > >> Others may know a better way to do this, but the only way I know to >> truly verify the commit id is something like the following: >> >> git

Re: Apache NiFi 1.7.0 RC1 Release Helper Guide

Others may know a better way to do this, but the only way I know to truly verify the commit id is something like the following: git clone https://git-wip-us.apache.org/repos/asf/nifi.git git -C nifi checkout diff --brief -r For verifying the RC was branched off the correct git commit id, you

Re: SplitText - How to make each split unique?

Hello, In general you probably want to take a look at the "record" processors which will offer a more efficient way of performing this task without needing to split to 1 message per flow file. The flow with the record processors would probably be GetFile -> ConvertRecord (using CsvReader and

Re: [VOTE] Release Apache NiFi Registry 0.2.0

+1 (binding) Release this package as nifi-registry-0.2.0 - Ran through everything in the release helper and looked good, few minor things Andy mentioned - Tested upgrading an existing registry to 0.2.0 to test database migration - Tested basic event hook logging - Ran secure NiFi with secure

Re: Apache NiFi Registry 0.2.0 RC1 Release Helper Guide

I can't help you with the Docker part, but there shouldn't be any major issues setting up secure NiFi and registry. Andrew Lim put together some great videos that are linked to from the registry page of the website... Setting Up a Secure Apache NiFi Registry https://youtu.be/qD03ao3R-a4

Re: TCP Connection Limit Error

uggest upgrade the version may solve this issue? > > > > Thanks and Regards, > > *Rajesh Biswas* | +91 9886433461 | www.bridgera.com > > > > *From:* Bryan Bende [mailto:bbe...@gmail.com] > *Sent:* Friday, June 15, 2018 12:50 AM > *To:* us...@nifi.apache.org

Re: Should the elastic search client service impl get renamed before 1.7?

Lets make sure this is mentioned in the migration notes wiki for 1.7.0 since it will create a ghost component for existing flows that have the ES service with the original name. On Mon, Jun 11, 2018 at 9:16 AM, Mike Thomsen wrote: > Should mention that I did not check the L for the ES 6.X

Re: Primary Only Content Migration

Peter, There really shouldn't be any non-source processors scheduled for primary node only. We may even want to consider preventing that option when the processor has an incoming connection to avoid creating any confusion. As long as you set source processors to primary node only then everything

Re: [DISCUSS] Change of Cluster Flow Inheritance

Using the versioned flow logic seems like a good idea. Would the authorizer fingerprints still be checked as part of joining the cluster? Currently that is appended to the overall fingerprint to ensure each node has the same users/policies, or at least same config (i.e. LDAP). Would be nice if a

Re: API endpoint for /resources

l policies > available through the UI, and /resources does not appear in > authorizations.xml. > > Thanks, > -Mark > > On Thu, Jun 7, 2018 at 8:37 AM, Bryan Bende wrote: > >> Mark, >> >> The resources end-point returns all of possible resource

Re: [EXT] Re: URL configuration for the remote process group in Nifi 1.3

Paresh, Mark can correct me if I'm wrong, but I believe the information fetched in step 1 is persisted in-memory on each node where the RPG is running. This information is then periodically refreshed in a background thread. When data is flowing through it is distributing the data to the nodes in

Re: [ANNOUNCE] New Apache NiFi Committer Sivaprasanna Sethuraman

Congrats! and thank you for your contributions to the NiFi community. On Tue, Jun 5, 2018 at 10:16 AM, Kevin Doran wrote: > Congrats, Sivaprasanna! > > On 6/5/18, 10:09, "Tony Kurc" wrote: > > On behalf of the Apache NiFI PMC, I am very pleased to announce that > Sivaprasanna has

Re: oAuth for API stood up on nifi

Hello, Processors can't leverage NiFi's internal authentication and authorization mechanisms. For HandleHttpRequest it supports two-way TLS for client authentication. -Bryan On Mon, Jun 4, 2018 at 2:11 PM, Anil Rai wrote: > Team, for invoking the nifi API's on a secured cluster, we have to

Re: Publishing Kafka Topics via Apache Nifi cluster.

I was looking at EnforceOrder again and I'm not sure that will actually help here since I don't think it works across a cluster, but maybe others know more. I think you can only ever have 1 concurrent task for your PublishKafka processor. Even if you run everything on primary node, if you have 2

Re: Publishing Kafka Topics via Apache Nifi cluster.

Hello, If I'm understanding the situation correctly, you want ordering within a key, but not necessarily total ordering across all your data? I'm making this assumption since you said you have 9 partitions on your Kafka topic and you are partitioning by key, so the data for each key is in order

Re: Dependency versions in processors in 1.6.0

Jeff, What you described sounds correct. Can you share which dependency versions now need to be specified? We could at least update the processor bundle archetype to have these versions specified to make it easier for new bundles, and maybe add something to the migration notes for existing

Re: NiFi ListenTCP 1.4.0 "Rejecting Connections from xx.xx.xx.xx : xxxx, because max connection has been met" timeout

The max connection warning is simply based on the property in the processor called "Max Number of TCP Connections" which defaults to 2. So in the default case, if a third connection is made while two connections are open, then the third connection is rejected and you see this warning. If you are

Re: [VOTE] Change the name Apache NiFi Fluid Design System to Apache NiFi Flow Design System

+1 (binding) On Fri, May 18, 2018 at 10:18 AM, James Wing wrote: > +1 (binding) > >> On May 18, 2018, at 6:30 AM, Rob Moran wrote: >> >> Following positive response discussing the name change of nifi-fds [1], I'd >> like to call a vote to officially change

Re: User and Policies

s > there a way to hide the other process groups from the canvas that I do not > have permissions to? > > Thanks > Anil > > > > On Mon, May 14, 2018 at 3:10 PM, Anil Rai <anilrain...@gmail.com> wrote: > >> Thanks for the detailed explanation Bryan. >

Re: User and Policies

Hello, When a node joins the clusters, if the node has an empty flow.xml, no users, and no authorizations, then the node will inherit all of those from the cluster, but if any of those are populated then it won't be able to join. One common issue that prevents this from working, is if you have

Re: ReplaceText Flow File Processing Count

I don't know the history of this particular processor, but I think the purpose of the session.get() with batches is similar to the concept of @SupportsBatching. Basically both of them should have better performance because you are handling multiple flow files in a single session. The supports

Re: GetSplunk : How to pass parameters as input to GetSplunk processor

Hello, A ‘QuerySplunk’ processor that allowed incoming flow files probably makes sense. If you want to work on this feel free to create a JIRA. I don’t see any existing tickets for Splunk related processors. Thanks, Bryan > On May 2, 2018, at 3:56 AM, Brajendra Mishra >

Re: Nifi Development Question - Asynchronous Writes

Hello, I think to store off the flow files you would also need to store the session it came from, but I would probably question whether this is really the best idea... What type of data are you expecting to come into your processor? 1) If you can leverage the record reader concept in NiFi this

Re: RAT errors after pull of master

These modules were moved when you pulled from master so these are the leftover traces of things that aren't under version control, they are all things in target and .iml files so you just have to blow them away. On Mon, Apr 30, 2018 at 9:55 AM, Sivaprasanna wrote: >

Re: Updating / Adding LICENSE and NOTICE entries for new processors

Anthony, Each binary artifact that NiFi publishes must account for the binary artifacts that it includes. Each NAR is published to Maven central on a release, so each NAR potentially needs a LICENSE/NOTICE. The overall NiFi assembly includes all NARs, so the LICENSE/NOTICE in nifi-assembly is

Re: Custom Controller Service

In case it helps, Matt B was nice enough to quickly put up a PR for the API change discussed yesterday: https://github.com/apache/nifi/pull/2658 Once this is tested and merged it make it would make it easier to implement Charlie's approach without needing a new interface. On Thu, Apr 26, 2018

Re: [DISCUSS] Support for accessing sensitive values safely

a checkbox which > when checked, will consider that as an EL and evaluate from the variable > registry and when not checked, assume that the entered password is plain > password and no evaluation needs to happen. > > - > > Sivaprasanna > > On Thu, Apr 26, 2018 at 12:19 AM

Re: [DISCUSS] Support for accessing sensitive values safely

gt; > A person may have rights to configure nifi without knowing the “value” of > the secure db password ( for example ), but that doesn’t mean they > don’t have there rights to reference it. > > > > On April 25, 2018 at 14:15:16, Bryan Bende (bbe...@gmail.com) wrote: > > There

Re: [DISCUSS] Support for accessing sensitive values safely

There is definitely room for improvement here. Keep in mind that often the sensitive information is specific to a given environment. For example you build a flow in dev with your db.password. You don't actually want your dev db password to be propagated to the next environment, but you do want to

Re: Custom Controller Service

' that means, we'll be using >'UpdateAttribute' to do add that attribute to flowfile? >2. If we are to use 'UpdateAttribute' to set the value for 'db.id', we >need to know before hand, right? > > - > > Sivaprasanna > > On Wed, Apr 25, 2018 at 8:38 PM, Bryan Bende <bbe

Re: Custom Controller Service

ribute from incoming flow files, then send that to > the router which returns back the connection pool. > > On Wed, Apr 25, 2018 at 9:48 AM, Bryan Bende <bbe...@gmail.com> wrote: > >> Here is a proposal for how to modify the existing API to support both >> scenarios: >

Re: Custom Controller Service

to provide a new service implementation that used the attribute map to somehow manage multiple connection pools, or create connections on the fly, or whatever the desired behavior is. On Wed, Apr 25, 2018 at 9:34 AM, Bryan Bende <bbe...@gmail.com> wrote: > To Otto's question... > > For

Re: Custom Controller Service

ection it needs), we use the relevant > one. > > Thanks, > Sivaprasanna > > On Wed, 25 Apr 2018 at 6:07 PM, Bryan Bende <bbe...@gmail.com> wrote: > >> The issue here is more about the service API and not the implementations. >> >> The current API has no way to pas

Re: Custom Controller Service

Hello, Others who have worked on the DB related services and processors can correct me if I'm wrong here, but... In general the idea of a connection pool is that creating connections is somewhat expensive, and for a high-volume of operations you don't want to create a connection for each DB

Re: Pushing flows to Registry with Sensitive Information

Jorge, Currently variables are not meant to store sensitive information, the reason has to do with how users access variables... The way a user accesses a variable is via expression language, and since EL is just free from text entered into a property descriptor, it is impossible to restrict

Re: BUG: AbstractListProcessor stops to call performListing after primary node reelection to same node

Max, Thanks for reporting this. I've only glanced at the code quickly, but I see what you are saying about the boolean never getting set to false when it hits the context.yield(). I would recommend creating a JIRA with all of this info, and then you could submit your proposed fix as a pull

Re: How can I pass a flowfile attribute to a controller service?

I'm not sure if this helps, but you mentioned not being able to use the variable.registry because it requires a restart. That is true for the file-based variable registry, however it is not true for the UI-based variable registry [1]. Keep in mind that neither of the variable registries are

Re: Apache nifi isuues-while ingesting data to hdfs of aws instance

Hello, I have no idea how AWS works, but most likely what is happening is the Hadoop client in NiFi asks the name node to write a file, and the name node then responds with the data nodes to write to, but it is responding with the private IPs/hostnames of the data nodes which you can't reach from

Re: Calling getLogger() from @OnScheduled, @OnStopped, etc.

The example processor you showed won’t work because you are calling getLogger() inline as part of the variable declaration. The logger is given to the processor in an init method which hasn’t been called yet at that point, so that is assigning null to the variable. Generally you should just call

Re: Nifi Parallel Execution

Hello, DetectDuplicate uses a DistributedMapCacheClientService which would be connecting to a DistributedMapCacheServer on one of your nodes. So all nodes should be connecting to the same cache server which is where the information about previously seen data is stored. -Bryan On Tue, Apr 10,

Re: Programmatically building a nested schema

Add a RecordField to the RecordSchema where the DataType is a RecordDataType... a RecordDataType then has a child schema. May be helpful to look at the code that converts between RecordSceham and Avro schemas:

Re: [VOTE] Release Apache NiFi 1.6.0 (RC3)

+1 (binding) Release this package as nifi-1.6.0 - Ran through release helper - Ran sample flows and tested granular restricted components with versioned flows On Wed, Apr 4, 2018 at 4:40 PM, Matt Gilman wrote: > +1 (binding) Release this package as nifi-1.6.0 > > - Ran

Re: ListSFTP incoming relationship

o. > Maybe it would be best to output the results into a structured format, such > as AVRO? Or, maybe it would just be best to output one flowfile per remote > file found, and include updated time and fully qualified path as attributes? > > Scott > > > On 03/29/2018 04:32 AM,

Re: ListSFTP incoming relationship

..@apache.org <mailto:alopre...@apache.org> > > >> /alopresto.apa...@gmail.com <mailto:alopresto.apa...@gmail.com>/ > > >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > >> > > >>> On Mar 27, 2018, at 8:33 AM, Andrew Grande

<    1   2   3   4   5   6   7   8   >