Re: NiFi Queue Monitoring

Can't you leverage some of the recent nifi features and basically run sql queries over NiFi metrics directly as part of the flow? Then act on it with a full flexibility of the flow. Kinda like a push design. Andrew On Tue, Jul 20, 2021, 2:31 PM scott wrote: > Hi all, > I'm trying to setup some

Re: NiFi configuration files changes

The physical files will get synchronized to the reference state from a central config management source (CM). There's no point watching them on the file system. If you need a change log for config files, I'd look into CM api to fetch those instead. On Tue, Jun 29, 2021, 8:30 AM Tomislav Novosel <

Re: Stopping processor after MAX number of retries

I saw it several times and I have a strong conviction this is an anti-pattern. A dataflow must not mess with start/stop state of processors or process groups. Instead, a flow is always running and one puts a conditional check to either not get the data in or reroute/deny it. If you need to

Re: Tls-toolkit.sh?

NiFi toolkit link here https://nifi.apache.org/download.html Enjoy :) On Fri, Dec 11, 2020, 8:59 AM Darren Govoni wrote: > Hi > > I want to setup a secure local nifi and the online docs refer to this > script but i cant find it anywhere. > > Any clues? > > Darren > > Sent from my Verizon,

Re: Rehosting flow files from cluster nodes to primary node only

I think a better design is for every node to write to this network share with some form of a partition (node) id in the filename. In a 5 node cluster you will have 5 parts. Next, either query over this directory directly with your engine of choice. Reducing all traffic to 1 node will be a

Re: NIFI HandleHttpRequest API - Health Check when API or Node Down

You can always hit NiFi API status rest endpoint. It won't give you any idea about that specific http endpoint you exposed, though, as it is a general nifi rest api. Your LB would need to understand how to hit this URL too, especially if it's secured. Coming back to the easiest path, you'd rather

Re: Looking for Help! Custom Processor NAR autoloader

I confirmed previously that autoload applies to an initial load and won't hot-reload on NAR update (instead requires an instance restart). If the custom component doesn't come up, I'm pretty sure there was a problem with dependency resolution. Check the startup logs for errors and any WARN

Re: Connecting Controller Services Automatically

t; restarts of the NiFi service. I do not recommend this. >>> >>> >>> Andy LoPresto >>> alopre...@apache.org >>> *alopresto.apa...@gmail.com * >>> He/Him >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >>>

Re: Connecting Controller Services Automatically

Aren't those IDs generated? How can one enforce it? Andrew On Sat, May 23, 2020, 10:53 AM Andy LoPresto wrote: > If you want the process to be completely automated, you would have to > enforce the controller service IDs to be identical across environments. > Otherwise deployment would need a

Re: Can Nifi authenticate and authorize using 389DS?

It should work, as it is the LDAP implementation, but not sure it was tested explicitly with this one. Authorization is local, however, one doesn't put application-specific policies into LDAP as a best practice. Instead, NiFi manages its policies. On Wed, Jan 29, 2020, 12:32 PM Dan Stromberg

Re: Holiday scheduling

Maybe a good fit for a drools engine rule set? I remember there was a community processor. Andrew On Wed, Jan 15, 2020, 10:40 AM Dave Andrews wrote: > Hello community, > Has anyone come up with a clean internal to NiFi solution to changing > workflow based on date/time? For example, a

Re: Abstract Processor > Static final properties

Also, static fields are scoped to the classloader, they are not global in a sense you described. NiFi uses classloader isolation for its processors. Andrew On Mon, Aug 12, 2019, 12:24 AM Craig Knell wrote: > Thanks > > Best regards > > Craig Knell > > Mobile 61 402128615 > Skype craigknell > >

Re: Postgres table as Cached Lookup Service

: > There is, sorry I’m AFK ATM but there’s a SimpleDatabaseLookup and a > DatabaseRecordLookup (or something similarly named) :) > > Sent from my iPhone > > On Aug 10, 2019, at 1:31 PM, Andrew Grande wrote: > > Maybe this would help? > https://github.com/mrcsparker/nifi-sqllookup-

Re: Postgres table as Cached Lookup Service

Maybe this would help? https://github.com/mrcsparker/nifi-sqllookup-services-bundle/blob/master/README.md I wish there was a standard bundled jdbc lookup record implementation. Andrew On Fri, Aug 9, 2019, 11:56 PM Craig Knell wrote: > Hi Folks > > Cached Postgres Lookup Service > whats the

Re: DistributeLoad across a NiFi cluster

Jim, There's a better solution in NiFi. Right click on the connection between ListFile and FetchFile and select a cluster distribution strategy in options. That's it :) Andrew On Tue, Jul 2, 2019, 7:37 AM James McMahon wrote: > We would like to employ a DistributeLoad processor, restricted to

Re: Would a currency parser record path function be useful?

Honestly, I have seen so many unique ways to mangle the currency format, that I'm not even sure one can reliably parse one anymore. But more importantly, isn't it the responsibility of a representation layer to format and pretty print the value? Just have hard time seeing a widespread need for

Re: PutKafka use with large quantity of data?

What's the concurrency for these processors? What's a global NiFi thread pool size? I wonder if you might be running out of available threads while they are waiting for external system i/o under load. Andrew On Thu, Apr 4, 2019, 8:24 AM l vic wrote: > What's this particular processing group

Re: Weird ListFile Issue

Looks like the processor started listing CWD On Fri, Mar 22, 2019, 2:00 PM William Gosse wrote: > The real var name was aimuploaddir and the error was aimduploaddir. Think > I have a work around to prevent the tragedy was wiping out Nifi. I did this: > >

Re: Log Queries being executed by PutDatabaseRecord

Could ve a good idea to log values at TRACE level then. On Wed, Feb 27, 2019, 7:56 AM Matt Burgess wrote: > True, at a DEBUG level we could output the record values, although for > large flow files this will be quite verbose :) Also the point of the > ?s is not necessarily to not show the

Re: Using variables in SSLContextService

t; If you guys agree that this is a reasonable request, is it ok for me then > to raise such a feature request in > https://issues.apache.org/jira/projects/NIFI/issues/NIFI-4610?filter=allopenissues > ? > > In the meantime, is there another way to achieve what I’m after? > &g

Re: Using variables in SSLContextService

Mike, I think the ask here is for this CS to support variable registry values. IIRC, there are other cases in NiFi where EL is not supported, but VR is. A fair request, IMO. Supporting a full EL for the keystore/truststore path is a bad idea, no doubt. Do you agree? Andrew On Tue, Feb 19,

Re: Nifi provenance indexing throughput if it is being used as an event store

NiFi provenance searches are not a good integration pattern for external systems. I.e. using it to periodicaly fetch history burdens the cluster (those searches can be heavy) and disrupt normal processing SLAs. Pushing provenance events out to an external system (pitebtially even filtered down to

Re: Record-oriented DetectDuplicate?

Can I suggest a time-based option for specifying the window? I think we only mentioned the number of records. Andrew On Fri, Feb 8, 2019, 8:22 AM Mike Thomsen wrote: > Thanks. That answers it succinctly for me. I'll build out a > DetectDuplicateRecord processor to handle this. > > On Fri, Feb

Re: Preferred schema registry

Isn't AvroSchemaRegistry an embedded one to NiFi? I.e. it's not a comparable alternative to an external dedicated schema registry. Andrew On Wed, Jan 16, 2019 at 7:43 AM Mike Thomsen wrote: > I think I figured out what it was. We backed out a change and it was > fungible with an earlier

Re: Truststore/Trusted hostname

Walter, you could point to the default JRE truststore file, maybe. Andrew On Wed, Jan 9, 2019, 7:12 AM Kevin Doran wrote: > Hi Walter, > > I could be mistaken, but my interpretation of the Trusted Hostname > configuration option is that it is designed to work with/in-addition-to the >

Re: Variables handling in MiNiFi

Luie, Which version are you looking at? I think there was a recent discussion about supporting those in MiNiFi, not sure if it got imoleme ted already, though. Andrew On Wed, Dec 19, 2018, 9:26 AM luis_size wrote: > Hi > > I am a big fan of NiFi variables registry. This makes instantiating >

Re: NiFi JSON enrichment

James, The easiest would be to merge json in a custom processor. Not easy as in no work at all, but given your limitations with the NiFi version could be done sooner maybe. Andrew On Mon, Dec 17, 2018, 9:53 AM James Srinivasan wrote: > Hi all, > > I'm trying to enrich a data stream using

Re: NiFi Toolkit CLI issues with NiFi/Registry SSL handshake

Guessing - if it's a new container on the first invocation, maybe the JVM is generating the binary class cache? This operation is performed only once ever, but with a clean environment every time I can see it being invoked again and again. Other than that, you'd need to connect a profiler and see

Re: NiFi Toolkit CLI issues with NiFi/Registry SSL handshake

Are you running in some container environment? It should never take 15 secs and there's no caching performed by cli. I would review the container environment and see why it's taking forever to start. Andrew On Sun, Oct 28, 2018, 10:11 AM ara m. wrote: > The Keystore user had "view processor"

Re: how to organise processor or PG in the Nifi canvas ?

Try importing via thr NiFi cli tool. I've implemented the logic to auto layout newly imported PGs. But I agree, this should be something included in the core NiFi logic itself. Andrew On Fri, Sep 14, 2018, 4:17 PM Dominique De Vito wrote: > Hi, > > Is there a way to re-organize automatically

Re: AVRO is the only output format with ExecuteSQL

some refactoring of the parent > #onTrigger() to service methods. > > > > > > Andy LoPresto > > alopre...@apache.org > > alopresto.apa...@gmail.com > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > > On Aug 7, 2018

Re: AVRO is the only output format with ExecuteSQL

gt; >> [1] https://issues.apache.org/jira/browse/NIFI-4517 >> On Tue, Aug 7, 2018 at 9:20 AM Bryan Bende wrote: >> > >> > I would also add that the pattern of splitting to 1 record per flow >> > file was common before the record processors existed, and generally >

Re: Multiple registry instances sharing Git repo

There are also registry event hooks to help with the automation. Andrew On Tue, Aug 7, 2018, 12:46 PM Bryan Bende wrote: > Mike, > > It is not really made to have multiple instances pointing at the same git > repo. > > Each registry has a metadata DB and a flow storage component which can > be

Re: AVRO is the only output format with ExecuteSQL

Careful, that makes too much sense, Joe ;) On Tue, Aug 7, 2018, 8:45 AM Joe Witt wrote: > i think we just need to make an ExecuteSqlRecord processor. > > thanks > > On Tue, Aug 7, 2018, 8:41 AM Mike Thomsen wrote: > >> My guess is that it is due to the fact that Avro is the only record type >>

Re: docker nifi cluster not working

I think the idea is, if there is anything available today, it will have to be built from sources, including the new docker image. I.e. it's not a public image yet. Andrew On Fri, Jul 20, 2018, 7:20 AM Chris Herssens wrote: > Hello Mike, > > Where can I find nifi 1.8 image for docker ? > I get

Re: NiFi Registry with nested PGs

I would also check the poll period. E.g. give it 40 secs or more to detect new versions in the registry, it's not real time. Andrew On Thu, Jul 19, 2018, 6:35 AM Mike Thomsen wrote: > Thanks, Kevin. If I get a chance I'll try it out again because it appeared > to be the case that after the

Re: Only get file when a set exists.

Martijn, Here's an idea you could explore. Have the ListFile processor work as usual and create a custom component (start with a scripting one to prototype) grouping the filenames as needed. I don't know of the number of files in a set is different every time, so trying to be more robust. Once

Re: Allowing all users to connect

Juan, A cert implies one knows the identity of the cert holder. I'd imagine if you shared it with multiple users, you would have achieved this semi-anonymous requirement. I would take a really deep look into why you want to do it this way, though. Defeats the purpose of security. Is there a

Re: PDF generating processor

Hi Mike, Thanks for sharing this. May I also suggest you provide a binary NAR download on the releases page in this github? The NiFi build pupeline is non-trivial, it would help folks get started quickly. Andrew On Fri, May 11, 2018, 7:40 AM Mike Thomsen wrote: > A few

Re: NiFi cluster with DistributedMapCacheServer/Client

HBase-backed cache is a great choice here. Redis is nice and nimble, but when it comes to clustering and enterprise security, may not be the best fit. The original legacy cache server in NiFi is... well, it should be deprecated and removed, IMO :) Andrew On Fri, Apr 13, 2018, 3:45 PM James

Re: [Nifi 1.5.0] DistributedMapCacheClientService fails to open port

It's a client, it connects to a cache service. You need to start another service and point the client to it. Andrew On Tue, Apr 10, 2018, 6:29 AM françois lacombe wrote: > Hi all, > > Does anyone experience problems with DistributedMapCacheClientService > service?

Re: how to edit queue content

How is it going to work with e.g. 20GB of events in the queue? I'd be careful, as requirements blow up into a full db with indexes, search, and a UI on top. If one wanted to filter events, wouldn't a standard processor do the job better? Andrew On Tue, Mar 27, 2018, 12:11 AM Joe Witt

Re: 答复: put pictures from remote server into hdfs

I think the MOB expectation for HBase was around 10MB. I agree it will require some thought put in organizing the space and region server splits with column families, once this volume becomes significant. Andrew On Fri, Mar 23, 2018, 9:08 AM Mike Thomsen wrote: > Off

Re: setting processor concurrency based on the development/production environment

Hi Andrew, > > thanks for the idea. I've been playing with nipyapi recently so might give > this a try. > > Thanks > > On Thu, Mar 1, 2018 at 7:32 PM, Andrew Grande <apere...@gmail.com> wrote: > >> Boris, >> >> Here's an idea youncould explore _today

Re: setting processor concurrency based on the development/production environment

Boris, Here's an idea youncould explore _today_. Assume your dev and prod flows live in different bucket/registry instance. Given that you are trying out NiFi 1.6, you should be able to extract the versioned flow from DEV and process it to change the concurrency level for PROD before committing

Re: Registry, sdlc and promotion between environments

Yes, Georg, there's something coming up to address exactly that, please take a look at https://github.com/apache/nifi/pull/2477 Andrew On Wed, Feb 21, 2018, 2:45 AM Georg Heiler wrote: > Hi > > Can I use the new nifi registry to promote / move my nifi flows between >

Re: Routing Files Based on Dynamic Pattern Matching

lah.csv, I might have > a regular expression defined that says if the file name matches > .*blah_test_blah(?!_somethingelse).* then it should goto directory > /tmp/blah_test > > > > Thanks > > Shawn > > > > *From:* Andrew Grande <apere...@gmail.com> > *S

Re: Modularize nifi flows

A Processing Group would serve as a reusable unit, which then has version control applied to it. You would model your flows to abstract interactions with e.g. in/output port objects. They also mentioned that nested PG can be versioned separately, looks like git submodules behavior, but I haven't

Re: Routing Files Based on Dynamic Pattern Matching

Maybe take a look at one of the ScanContent/Attribute processors? It allows to map in a reloadable external file. Or, better yet, one of the LookupService variants, which is more generic. HTH, Andrew On Mon, Feb 19, 2018, 3:10 PM Shawn Weeks wrote: > Hi, I’m looking

Re: NiFi Registry Create Item in the UI

n > at the top-level which would create the flow and the first version as > one operation, I view this as what you are calling "New Flow", and > then there could be an "Import Version" action from the actions > drop-down of a specific flow. > > -Bryan &

NiFi Registry Create Item in the UI

Hi, I wanted to share some feedback and suggestions on the registry UI. It was a little unnatural to look for a New Bucket button in the Settings menu. Same sentiment with the Create Flow for when we expect to be able to import it in the UI in the near future. How about making the (+) or New

Re: Persistence mechanism in Nifi

I don't think you need to do anything. If your target is down, the data is stored in the NiFi connection. You can raise the backpressure limits as long as you have enough disk space (I don't recommend disabling it). Andrew On Thu, Jan 18, 2018, 12:31 AM Vikram KR wrote: >

Re: Polling Processors impact on Latency

Yes, polling increases latency in some cases. But no, NiFi is not just polling. It has all kinds of sources, and listening vs polling vs subscribing purely depends on the protocol of that given processor. Hope this helps, Andrew On Tue, Nov 7, 2017, 1:39 AM Chirag Dewan

Re: Back pressure deadlock

I wonder which jms broker you are using. The situation where a jms destination is full is absurd, the whole point was to decouple publishers and consumers. I would additionally look into what jms broker settings are available to address the situation. Andrew On Mon, Oct 23, 2017, 10:32 AM Arne

Re: Dynamically adding avro schema to AvroSchemaRegistry at runtime

Hi, Using an external schema registry is the way to go. The embedded one is meant for ease of use, but once you grow beyond an initial phase, the best practice is to have a full service and potentially use standard InvomeHTTP to perform operations beyond just lookups. Does it help? Andrew On

Re: org.apache.nifi.spark.NiFiReceiver and spark spark.streaming.receiver.maxRate

A typical production setup is to use Kafka in the middle. Andrew On Mon, Sep 18, 2017, 3:02 AM Margus Roo wrote: > Hi > > I need to take flow with Spark streaming from Nifi port. As we know > Spark supports spark.streaming.receiver.maxRate and >

Re: Re: QueryDatabaseTable - Deleted Records

As an interesting architectural approach we took eons ago, before NiFi, was to take daily snapshots of a full table. Every row would then be hashed/digested or in any other way uniquely identified and 2 datasets would be crossed and compared to find inserts/deletes/updates. It was involved, but

Re: Missing nifi-app.log files

Typically this is better handled by using the -F switch instead of -f, it has more robust file handling and manages files disappearing correctly. Unfortunately, some OS don't have that switch in their toolchain. Andrew On Thu, Aug 17, 2017, 5:07 PM James McMahon wrote: >

Re: Parameterizing the nifi flow

Hi, Read up on the variable registry in the docs, that sounds like a good fit. I don't remember if it were available in 1.1 though. Andrew On Fri, Aug 11, 2017, 5:12 PM More, Vikram (CONT) < vikram.m...@capitalone.com> wrote: > Hi, > > > > I have a nifi flow which pulls/extracts from source

Re: ExecuteSQL with MSSQL Server Express

Try with forward slashes, please. On Wed, Jul 19, 2017, 12:17 PM Praveen Reddy wrote: > Hi, > > I am using ExecuteSQL processor to connect with MSSql server express to > get some data. But I am getting the below exception: > > [image: Inline image 1] > > My Connection

Re: Processor classpath

It's not a classpath, but rather a user configurable location, exposed as a processor property. Does it help? Andrew On Tue, Jul 4, 2017, 4:32 PM James Srinivasan wrote: > Hi, > > I'm developing a processor which needs to read some of its config from > the

Re: Quickly find queues with data...

Pro tip - this connection backlog info is available in the json returned from the status API. This is a good way to integrating with monitoring system via polling. Andrew On Fri, May 5, 2017, 5:41 PM Russell Bateman wrote: > Ah, I see. Thanks very much. (I spend too much

Re: Publish with persistence using PublishAMQP?

James, you make too much sense :) Mind filing a jira for that enhancement? Thanks, Andrew On Thu, May 4, 2017, 9:56 AM James McMahon wrote: > amqp$deliveryMode > > from the documentation: > Attributes extracted from the FlowFile are considered candidates for AMQP >

Re: Input flowfile for GetHTTP

Try with InvokeHttp. Andrew On Tue, May 2, 2017, 3:42 AM Buntu Dev wrote: > I'm trying to connect ExtractText to GetHTTP processor but that doesn't > seem to be allowed. Wanted to check if that is true that GetHTTP processor > doesn't accept input flowfiles and if so, how

Re: Apache nifi 1.0.0 consuming high CPU utilization

What is your flow doing? It very much depends on this. While 6 cores is not a great hardware to run a full NiFi system on, I'd guess ReplaceText and EvaluateJsonPath are the hotspots. See how much data queues up in front​ of these processors. Andrew On Tue, May 2, 2017, 5:09 AM kunal

Re: csv output processor?

There is something interesting coming out in 1.2.0 potentially, the new pairs of RecordReaders/RecordSetWriters. I did see CSV format support in there. Take another look maybe. Andrew On Fri, Apr 28, 2017, 4:55 PM Frank Maritato wrote: > Is there a nifi processor

Re: Back Pressure Object threshold not honored

I would also suggest the ControlRate processor in front of a sensitive step. There's a chance data may accumulate and sent in burst after downtime, so it ensures a predictable outflow. Andrew On Thu, Apr 27, 2017, 8:03 PM Joe Percivall wrote: > Hello Kevin, > > I

Re: NPE in ListenSyslog processor

I wonder if the cause of zero length messages is the health check from the f5 balancer. Worth verifying with your team. Andrew On Tue, Apr 25, 2017, 3:15 PM Andy LoPresto wrote: > PR 1694 [1] is available for this issue. > > [1] https://github.com/apache/nifi/pull/1694 >

Re: NiFi best practices to manage big flowfiles

ed recipient, please contact immediately the sender by > telephone, fax or e-mail and delete the information in this message > that has been received in error. The sender does not give any warranty > or accept liability as the content, accuracy or completeness of sent > messages and accept

Re: NiFi best practices to manage big flowfiles

Hi, First, there won't be multiple copies of a file within NiFi. If you pass around the content and don't change it (only attributes), it will merely point a reference to it, no more. You need to decide if you want to delete processed files, this is what GetFile does. Might want to look into

Re: Clustering Best Practices?

BTW, your NiFi instance is not single-threaded, it's a single node. It still runs multiple worker threads in the flow. Andrew On Thu, Apr 20, 2017, 7:01 AM James McMahon wrote: > Good morning. I have established an initial single-threaded NiFi server > instance for my

Re: Change data capture processor

Yes, it's in 1.2.0. However, it's unreleased yet. Should be soon, you can track progress by following the 1.2.0 release email thread on tbe dev list. Andrew On Wed, Apr 12, 2017, 1:55 AM Buntu Dev wrote: > Is the new CaptureChangeMySQL processor available in 1.2 and is

Re: Logging changes to workflow

John, the Flow Configuration History menu item already captures it. Does it look like a fit? Andrew On Sun, Apr 9, 2017, 10:50 PM HARRIOTT, John wrote: > Is it possible to log changes to a workflow so they can be captured by a > central logging/auditing capability

Re: Re: new Nifi Processors

Basically the GPL license puts restrictions on how one can distribute in practical terms. Meaning your work may live under GPL license as long as it's not part of the official package. End users will have to download your NAR themselves. Andrew On Wed, Mar 1, 2017, 8:43 AM Matt Burgess

Wait/Notify State in a Cluster

Hi Guys, I've noticed the upcoming version has wait/notify set of processors (great write ups by Koji, avain). Does it support clustered state, though? Planned maybe? For high throughput scenarios, is it possible to switch it out to algernative implementations if e.g. default relies on a ZK?

Re: Nifi taking forever to start

I'm not sure piggy-backing on the host entropy will work reliably. I have seen this issue in ec2, openstack boxes, etc. A newly spun up box will exhibit this issue often. Andrew On Wed, Feb 15, 2017, 10:09 AM Bryan Rosander wrote: > Hey Arnaud, > > Andy's solution is

Re: nifi at AWS

Isn't it more advisable to use the HTTP mode instead, i.e. no additional ports to open? Make sure to change the client RPG mode to http from RAW (in the UI). Andrew On Sun, Jan 22, 2017, 10:47 AM Bryan Bende wrote: > Hello, > > I'm assuming you are using site-to-site since

Re: DetectDuplicate

Juan, no change from how you remember this processor yet. I personally would love to have a more pluggable backend for it, too. Andrew On Mon, Dec 19, 2016, 2:35 PM Juan Sequeiros wrote: > Hello, > > I am wondering if DetectDuplicate still has single dependency on >

Re: How to integrate a custom protocol with Apache Nifi

Kant, Look into a custom processor. You have a choice of either implementing e.g. a parser for your data, or, if the protocol is more involved, implement a receiver in the processor as well which would spit out meaningful data messages next. Andrew On Tue, Dec 6, 2016, 7:47 PM kant kodali

Re: Data Provenance is not available

Yes. It's a combination of a generic view data permission in the global menu and specific access in the processing group. Andrew On Mon, Nov 21, 2016, 12:28 PM Pablo Lopez wrote: > Hi, > > Any ideas as of why Data Provenance option is grayed out (not available) >

Re: NiFi 1.0.0 canvas background

Hi Russ, Alignment or any snapping to a grid wasn't there before, but that would be a very welcome feature, I agree. Regarding the background, I personally didn't notice a change, isn't it the same? Andrew On Tue, Nov 15, 2016, 3:14 PM Russell Bateman < russell.bate...@perfectsearchcorp.com>

Re: Delay Processor

Joe, It's good to know some thinking went into this feature before. Basically, I'm trying to put a spotlight on these 2 areas: 1. Making it, potentially, more generic than a retry loop. E.g. enhance the ControlRate processor. 2. Making these policies more explicit, so a user wouldn't

Re: Delay Processor

Oleg, I'll break my response in 2 threads. I understand your use case of 'delay until X', but frankly would design it differently if we're talking about long-term transactions or schedules. It may involve systems external to NiFi. Anyway, I'd like to keep this use case out of scope for the delay

Delay Processor

Hi, I'd lime to check where discussions are on this, ir propose the new component otherwise. Use case: make delay strategies explicit, easier to use. E.g. think of a failure retry loop. Currently, ControlRate is somewhat related, but can be improved. E.g. introduce delay strategies a la

Re: stream one large file, only once

Neither GetFile nor FetchFile read the file into memory, they only deal with the file handle and pass the contents via a handle to the content repository (NiFi streams data into and reads as a stream). What you will face, however, is an issue with a SplitText when you try to split it in 1

Re: unable to empty the connection queue between 2 processors in NIFI secure cluster

Hi, There are 2 levels basically. One is the global policies in the top right menu. Another is in the operator menu on the left and is specific to every processing group. Sometimes you need a combination of both to allow for an action. E.g. try data provenance and modify data permissions to

Re: Enable Compression on Remote Port?

Disable transmission on RPG, go into the ports view again. Now, you should be able to modify settings like compression, concurrent threads and security on ports. Andrew On Thu, Nov 10, 2016, 2:26 PM Peter Wicks (pwicks) wrote: > When I have a Remote Process Group and I view

Re: Trouble with PublishKafka10

2 things to check: 1. Are you connecting to Kafka 0.10 broker? 2. Which port are you using? Recent Kafka clients must point to the kafka broker port directly. Older clients were connecting through a zookeeper and used different host/port. Andrew On Sun, Oct 30, 2016, 1:11 PM Daniel Einspanjer <

Re: How I put the cluster down.

Hi, I'd suggest couple things. Have you configured backpressure controls on connections? NiFi 1.0.0 adds 1evt/1GB by default IIRC. This can help avoid overwhelming components in a flow. Next, the 2 core CPU is really inadequate for high throughput system, see if you can get something better.

Re: SelectHiveQL Error

t;mattyb...@gmail.com>] > *Sent:* Thursday, October 06, 2016 5:17 PM > *To:* users@nifi.apache.org > *Subject:* Re: SelectHiveQL Error > > > > Andrew is correct. Although the HiveServer > > 1 driver is included with the NAR, the HiveConnectionPool is hardcoded to > use the

Re: nifi Rest API to get full details of the flow.

This isn't an ideal approach, IMO. There is a standard API to get a summary of the flow and,status of every processor, check what the Summary tab is invoking for a URL. You can then drill into any specific component by ID. Andrew On Wed, Sep 28, 2016, 6:42 AM Sandeep Khurana

Re: read in values from nifi.properties in a Groovy ExecuteScript processor

Which NiFi version? With 1.0 there are some bits for variable registry available, basically one can reference values from external config files via regular EL expressions. Andrew On Fri, Sep 23, 2016, 6:00 PM Tom Gullo wrote: > I want to read in values from nifi.properties

Re: UI: flow status and counters feedback

Alright guys, do we have enough consensus to start filing jira work items? :) Andrew On Tue, Sep 20, 2016, 2:01 PM Andrew Grande <apere...@gmail.com> wrote: > Let's fade the connection slowly to an inverted if backpressure engages? > > On Tue, Sep 20, 2016, 1:17 PM Rob Moran &l

Re: UI: feedback on the processor 'color' in NiFi 1.0

we could explore other ways to call these things > out so users do not come up with their own methods. Perhaps there are layer > options, like on a map (e.g., "show terrain" or "show traffic"). > > Rob > > On Tue, Sep 20, 2016 at 11:23 AM, Andrew Grande <apere..

Re: UI: flow status and counters feedback

s see what others think and then I can work on filing a jira to > capture thoughts. > > Rob > > On Mon, Sep 19, 2016 at 6:22 PM, Andrew Grande <apere...@gmail.com> wrote: > >> Hi All, >> >> I'd like to provide some feedback on the NiFi 1.0 UI now that I had

Re: UI: feedback on the processor 'color' in NiFi 1.0

ssor, that the color of the drop shadow could be changed as well. >>> That would provide more contrast, but preserve readability, in my opinion. >>> >>> On Mon, Sep 19, 2016 at 6:39 PM Andrew Grande <apere...@gmail.com> >>> wrote: >>> >>&

UI: flow status and counters feedback

Hi All, I'd like to provide some feedback on the NiFi 1.0 UI now that I had a chance to use it for a while, as well as pass along what I heard directly from other end users. Attached is a screenshot of a status bar right above the main flow canvas. The biggest difference from the 0.x UI is how

Re: Configure Multiple NCM

Hi Tijo, Take a look at clustering docs for NiFi 1.0. Zero-master clustering changed a few things, any node can be elected to be a primary or coordinator now. The 0.x concept of NCM is gone. >From the UI access standpoint, one can hit any node in a cluster to get the same experience. Andrew On

Re: Processor to send flowfile to two different destinations?

It's actually very simple - connect a processor output to 2 or more other processors or ports, use the 'success' relationship if prompted to choose from multiple. Andrew On Tue, Sep 13, 2016, 6:14 PM Russell Bateman < russell.bate...@perfectsearchcorp.com> wrote: > *DuplicateFlowFile* sends

Re: ExecuteProcess (fetch output)

Sven, take a look at ExtractText component, it will allow you to promote a result into an attribute. Andrew On Sat, Jul 2, 2016, 7:41 PM Sven Davison wrote: > I’ve been trying to run a script and fetch said output from the script as > a variable work with. The process

  1   2   >