Re: NiFi Registry -> NiFi

This is the Way, Team :) Congrats on this effort! Andrew On Fri, Jul 16, 2021, 8:29 AM Matt Burgess wrote: > All, > > Now that [1] is merged, NiFi Registry is now officially in the NiFi > codebase! Thanks to all who participated in discussions, reviews, and > testing. Here are some notes on

Re: Apache NiFi Registry

Isn't the proper state for this use case enabled/disabled? NiFi will start a PG and every schedulable component in it. If one needs to prevent this, disable a processor. Andrew On Thu, Jul 1, 2021, 2:17 PM Phillip Lord wrote: > Hello, > > My organization is considering utilizing the Registry.

Re: [discuss] we need to enable secure by default...

Mysql has been generating an admin password on default installs for, like, forever. This workflow should be familiar for many users. I'd suggest taking the automation tooling into account and how a production rollout (user-provided password) would fit into the workflow. Andrew On Tue, Feb 9,

Re: nifi deploy automation - variables pass through (incl controller services)

I think there is confusion here. Changing *properties* is considered a tracked change, the behavior is per design. There are *variables* on the PG level which are considered non-trackable environment settings and won't trigger a dirty flag. There is a dedicated command in the cli to list and set

Re: Nifi registry - extension-repository, extensions, bundles

Mike, These are upcoming features, not yet released. It's no surprise you don't see full reference docs where you'd expect them. Should probably ask if those will make it into NiFi 1.10.x or there's a different plan. Andrew On Tue, Jul 30, 2019, 1:43 PM HWANG, MICHAEL (MICHAEL) <

Re: RedisLabs license changes

Is there a separate license for a client driver? On Sat, Feb 23, 2019, 9:57 AM Mike Thomsen wrote: > > https://news.slashdot.org/story/19/02/22/2155223/redis-changes-its-open-source-licenseagain > > The reporting on it is sorta ambiguous as to whether the core Redis has > stopped being BSD

Re: [DISCUSS] Deprecate processors who have Record oriented counterpart?

I'm not sure deprecating is warranted. In my experience, record based processors are very powerful, but have a steep learning curve the way they are in NiFi today, and, frankly, simple things should be dead simple. Now, moving the record UX towards an easy extreme affects this equation, but e.g.

Re: NiFi, Docker Environment Variable for enabling debugging of NiFi inside a Docker container

Here's what I did previously, Erik. A user only had to specify an additional -debug flag for the main shell script which would in turn take care of any bootstrap rewriting/generation for allowing for the remote JVM debug session to connect. Maybe it could give a few ideas. Andrew On Wed, Feb 13,

Re: NiFi, Docker Environment Variable for enabling debugging of NiFi inside a Docker container

Erik, I was reading that email and was thinking of JVM debug options, with suspend y/n. I guess it just shows we meant very different things by debug mode. Maybe you could incorporate those into a PR too? Andrew On Fri, Feb 8, 2019, 2:15 PM Erik Anderson wrote: > I have a slight improvement

Re: Lowering the barrier of entry

(Yes, > I > > > > > > >> realise I could and possibly should submit PRs :) > > > > > > >> > > > > > > >> 1) I'm used to Java and Maven, so used the archetype. It > worked > > > fine, > > > > > > >> it woul

Re: Lowering the barrier of entry

alled when ^h^h^h - skimming back > > > >> over the docs, it looks pretty clear now > > > >> 4) Some example code for common operations e.g. getting/setting > > > >> attributes or reading/writing/modifying flowfile content would be > > > >>

Re: Lowering the barrier of entry

] https://start.spring.io/ > [2] https://github.com/spring-io/initializr > > On Fri, Jan 25, 2019 at 12:56 PM Andrew Grande wrote: > > > > We assume they create new projects from archetypes every day. They don't. > > > > We also assume they know how to deploy new NA

Re: Lowering the barrier of entry

jects+for+Extensions > > On Fri, Jan 25, 2019 at 12:23 PM Otto Fowler > wrote: > > > > I think this ties into my other discuss thread on refreshing the > archetypes > > > > > > On January 25, 2019 at 11:50:10, Andrew Grande (apere...@gmail.com) > wrot

Re: Lowering the barrier of entry

I consistently see my users struggling when they move up the nifi food chain and start looking at custom processors. The good content about prototyping processsors via scripting processors and finalizing with a full NAR bundle is everywhere but where it should be. A few simple changes could help

Re: [DISCUSS] Extension Registry

I would like to see a clear separation of blob and metadata storage. Most often you'd see some object storage being already distributed and replicated, let's think about an easy way to backup or migrate the metadata between registry instances. Andrew On Tue, Nov 13, 2018, 11:32 AM Michael Moser

Re: ExecuteGroovyScript processor unable to resolve IOUtils

Max, which NiFi version are you using. Can you try adding the @Grab annotation in your script declaring the commons-io dependency? IIRC, it was added recently. Andrew On Sun, Apr 22, 2018, 7:38 AM Max Viazovskyi wrote: > Recently I needed to write custom script to

Re: ListSFTP incoming relationship

The key here is that ListXXX processor maintains state. A directory is part of such state. Allowing arbitrary directories via an expression would create never ending stream of new entries in the state storage, effectively engineering a distributed DoS attack on the NiFi node or shared ZK quorum

Re: [EXT] Re: NiFi Versioned Process Group Status Icons

I would factor in any plans for a local version history, it may add new requirements to this status icon. Just saying there's more to that logic potentially. Andrew On Sun, Mar 25, 2018, 10:17 PM Peter Wicks (pwicks) wrote: > Matt, > > If you really want to enable awareness

Re: [DISCUSS] MiNiFi Flow Designer

My 2c. Using NiFi registry for both NiFi and MiNiFi flows is ideal. For the designer, IMO, adding another process/service to host it is not too great, as it adds to the complexity of overall ecosystem (the nifi/minifi java/cpp is already quite involved with c2 server, etc). I would like to see a

Re: Re: Re: 答复: Re: Is there a REST API to run a dataflow on demand?

One could write a script and call it in 1 step. I don't believe there is anything available OOTB. Andrew On Thu, Feb 22, 2018, 7:58 PM wrote: > Thanks a lot for your help. > > Yes. that is what I do to trigger a dataflow on demand. > But I want to know if there is an API

Re: Will you accept contributions in Scala?

Wasn't there a warning trigger about the NiFi distro size from Apache recently? IMO, before talking alternative languages, solve the modularity and NAR distribution problem. I think the implementation of a module won't matter much then, the point being not everything has to go in the core, base

Re: clear all flowfiles in all queues upon NiFi restart

Perhaps you could delete the repository directories when you need to restart with no data? On Thu, Jan 11, 2018, 9:16 PM 尹文才 wrote: > Hi Mark, forgot to ask about VolatileFlowFileRepository you mentioned, if I > switch to use VolatileFlowFileRepository, will NiFi swap out

Re: Question about Nifi

Check out the nifi-app.log file under $NIFI_HOME/logs. This is also customizable by editing the conf/logback.xml file. Andrew On Wed, Oct 18, 2017, 7:58 AM ☆★☆★☆ <1506877...@qq.com> wrote: > Hi, > > I am using nifi to develop some custom plugin(By Java), > can you please tell me after I use

Re: AvroSchemaRegistry doesn't enjoy copy and paste?

Definitely something is auto replacing quotes, I can confirm pasting worked fine before from a programmer's editor. Andrew On Thu, Jun 22, 2017, 9:06 AM Mark Payne wrote: > Andre, > > I've not seen this personally. I just clicked on the link you sent, copied > the schema,

Re: SplitText processor OOM larger input files

date the choice of > benchmarking flow and understand the performance on one box. > > Thanks a lot for all the people for helping me on this thread on my NiFi > evaluation journey. This is a really big plus for community support of > NiFi. > > M > > > > > &g

Re: SplitText processor OOM larger input files

It looks like your max bin size is 1000 and 10MB. Every time you hit those, it will write out a merged file. Update tge filename attribute to be unique before writing via PutHDFS. Andrew On Thu, Jun 1, 2017, 2:24 AM Martin Eden wrote: > Hi Joe, > > Thanks for the

Re: Not able to see the uploaded file in S3

Looks like you have checked the auto-terminate relationship and the upload fails silently. Make sure you expose the failure relationship from the PutS3 by sending it e.g. to a funnel. On Thu, May 25, 2017, 7:00 AM suman@cuddle.ai wrote: > Hi All, > > I have a simple

Re: Partitioning from actual Data (FlowFile) in NiFi

Yes to compress. The output of the merge step is a larger piece of data, no more/older than configured by the merge step. It can produce partial smaller buckets if it were configured with max age attribute. Andrew On Mon, May 15, 2017, 5:28 AM Anshuman Ghosh wrote:

Re: Zookeeper issues at initial Cluster startup

The delay on startup is due to the lack of entropy in your server, which is typical for cloud environments. I.e. SecureRandom doesn't yet have enough randomness to initialize and blocks waiting. Search the nifi entropy in archives, there were workarounds posted. Andrew On Tue, Feb 28, 2017, 3:34

Re: Central processor repository

Uwe, Are you aware of this one? https://issues.apache.org/jira/browse/NIFI-2168 Andrew On Sat, Feb 25, 2017, 7:53 AM u...@moosheimer.com wrote: > Pretty good idea! > Would appreciate a place where we all can upload sources (together with > some information) and everybody

Re: flow as code and minify scaling/isolation

Hi, I think all processors acting as clients do isolate Kerberos keytabs and client certificates. The Kafka situation is a current design limitation of Kafka, not NiFi. The good news is there's an effort underway to have Kafka not rely on global singleton config and specify those per connection

Re: [DISCUSS] Scale-out/Object Storage - taming the diversity of processors

. > > On 22 Feb 2017 10:21 AM, "Andrew Grande" <apere...@gmail.com> wrote: > > Andre, > > I came across multiple NiFi use cases where going through the HDFS layer > and the fs plugin may not be possible. I.e. when no HDFS layer present at > all, so no NN t

Re: [DISCUSS] Scale-out/Object Storage - taming the diversity of processors

Andre, I came across multiple NiFi use cases where going through the HDFS layer and the fs plugin may not be possible. I.e. when no HDFS layer present at all, so no NN to connect to. Another important aspect is operations. Current PutHDFS model with additional jar location, well, it kinda works,

Re: Add hidden show capability to sensitive text-area boxes

Hi, NiFi never pulls sensitive valued to the UI once set, it's a one-way operation only. Today, the UI has a hint 'sensitive value is set' to indicate it's not empty. Maybe you can provide more background on what you are trying to achieve in that context? Thanks, Andrew On Sun, Feb 12, 2017,

Re: Dealing with cluster errors

Joe, External ZK quorum would be my first move. And make sure those boxes have fast disks and no heavy load from other processes. Andrew On Fri, Feb 10, 2017, 7:23 AM Joe Gresock wrote: > I should add that the flows on the individual nodes appear to be processing > the

Re: Setting a node as primary?

Not anymore. The assumption is every node in a cluster can become a primary in the event of failover, and has to be configured to be the same across. Andrew On Thu, Feb 9, 2017, 5:58 AM Joe Gresock wrote: > Is there a way to request a node to become PRIMARY? I seem to

Re: [ANNOUNCE] New Apache NiFi Committer Joey Frazee

Great job, Joey, congrats :) On Tue, Jan 3, 2017 at 2:42 PM Aldrin Piri wrote: > On behalf of the Apache NiFI PMC, I am very pleased to announce that Joey > Frazee has accepted the PMC's invitation to become a committer on the > Apache NiFi project. We greatly appreciate all

Re: Adding JVM arguments when running NiFi...

Sounds like switching to the LinkedHashMap could address.the JVM startup flags ordering g? Andrew On Thu, Dec 15, 2016, 1:11 PM Russell Bateman wrote: > Joe, > > I have more detail and success to report... > > There is no problem in NiFi /per se/. It was not really

Re: Another holiday request...

; P.S. I hope my attempt at humor didn't offend. > > On 12/02/2016 07:08 AM, Andrew Grande wrote: > > I think the team was going for a toned down style anyway. One way to > draw a > > user's attention, which is available today already, is to use a standard > > log

Re: Another holiday request...

I think the team was going for a toned down style anyway. One way to draw a user's attention, which is available today already, is to use a standard log statement with proper level. This will issue a bulletin both on the processor (and visually indicate a note) as well as add it to a global list

Re: PostHttp processor: read timeout error (socketTimeoutException)

The influxdb rest endpoint can get overloaded and timeout. Also, see if you get better results with NiFi's InvokeHttp processor maybe. Andrew On Mon, Nov 14, 2016, 7:00 AM balacode63 wrote: > Hi all, > > I'm using /postHttp/ processor to post data to influxdb. I'm

Re: Uploading a csv file into an RDBMS table quickly like LOAD DATA

You could e.g. use NiFi to get the csv file, put it on a file system and invoke the native mysql loader command via ExecuteStreamCommand. Andrew On Sun, Nov 13, 2016, 6:57 PM raghav130593 wrote: > I receive tab delimited CSV files from an email and I want to insert them

Re: GetSFTP backpressure question

UpdateAttribute is a good example of a soft threshold behavior. By default, this processor operates in micro batches of 100. So, regardless if you set BP to e.g. 5, it will commit 100 events and then be throttled back by the connection. A user will see 100 events in the connection even though 5

Re: Processor running slow in production, not locally

Just a sanity check, number of open file handles increased as per quickstart document? Might need much more for your flow. Another tip, when your server experiences undesired hiccups like that try running 'nifi.sh dump save-in-this-file.txt' and investigate/share where NiFi threads are being held

Re: Custom expression language functions

I think there is a decent compromise available. Or maybe it's there already? Invoking a public static function via reflection. There are deployment and packaging limitations, sure, but at least it makes it easier for users to invoke utility functions (or custom ones). Andrew On Sun, Sep 11,

Re: ExecuteSql output formats.

Ate you thinking about some sort of a converter which can be referenced a la controller service? Andrew On Sat, Sep 3, 2016, 8:31 AM Toivo Adams wrote: > Hi, > > Currently Avro is supported, but often other formats are needed. > Workaround is to let ExecuteSql create

Re: Apache NiFi - 3 tier Architecture

Nishad, You can split NiFi clusters at will and connect them securely. Take a look at Remote Process Group and an underlying site-to-site protocol. In fact, linking data centers and layers is one of the mainstream NiFi use cases. Andrew On Thu, Sep 1, 2016, 1:05 PM Nishad Hameed

Re: Nifi Cross Account Download With A Profile Flag

Debug logging can be set in a processor itself in the UI, too. On Wed, Aug 31, 2016, 5:34 PM James Wing wrote: > Keren, > > Which version of NiFi are you using? > > One thing I noticed in your configuration of FetchS3Object is you are > setting both the Access Key and Secret

Re: PostHTTP Penalize file on HTTP 5xx response

Wasn't HTTP 400 Bad Request meant for that? 500 only means the server failed, not necessarily due to user input. Andrew On Wed, Aug 31, 2016, 10:16 AM Mark Payne wrote: > Hey Chris, > > I think it is reasonable to penalize when we receive a 500 response. 500 > means

Re: State sharing

cluster, so a > > given processor can access the state from any node because the processor > > will have the same UUID on each node. > > > > -Bryan > > > >> On Sunday, July 10, 2016, Andrew Grande <apere...@gmail.com> wrote: > >> > >> Sumo,

Re: State sharing

Sumo, IIRC there's a node one selects when setting state. If you invoke with a cluster mode, the state will be set to a ZK by default. Otherwise just local to this processor node. Andrew On Sun, Jul 10, 2016, 10:17 PM Sumanth Chinthagunta wrote: > If I set state from one

Re: Rollbacks

Devin, What you're asking for is a contradicting requirement. One trades individual message transactional control (and necessary overhead) for the higher throughput with micro-batching (but lesser control). In short, you can't expect to rollback a message and not affect the whole batch.

Re: Content Repo Large.. Archive in there?

Ryan, ./conf/archive is to create a snapshot of your entire flow, not the content repository data. See the attached screenshot (Settings menu on the right). Andrew On 10/23/15, 12:47 PM, "ryan.andrew.hendrick...@gmail.com on behalf of Ryan H"

Re: Content Repo Large.. Archive in there?

Attachments don't go through, view at imagebin: http://ibin.co/2K3SwR0z8yWX On 10/23/15, 12:52 PM, "Andrew Grande" <agra...@hortonworks.com> wrote: >Ryan, > >./conf/archive is to create a snapshot of your entire flow, not the content >repository data. See the a

Re: Newbie Dev Contribution : Questions

Venky, Thanks for putting yourself out to help. The best way is to employ an extensive testing framework in NiFi. E.g. take a look at

Re: Ingest Original data from External system by data's dependent condition

A typical pattern/workaround for this situation was to copy e.g. the json _in full_ into an attribute, leaving the payload in a binary format. But, as you can imagine, it's not ideal as FlowFile memory and disk pressure will be raised significantly and duplicate that of an existing content