Re: MergeContent/SplitText - Performance against large CSV files

Is RouteText available on any branch where I could build and play around with it before 0.4.0? I see it in the master branch on github, it's there. Andrew

Re: Custom Logic in Nifi

o write business logic, in processor or somethinng else? On Friday, October 30, 2015, Andrew Grande <agra...@hortonworks.com<mailto:agra...@hortonworks.com>> wrote: Naveen, Take a look at https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensi

Re: Start flow automatically on Nifi start

- I would like to start some of the flows whenever the nifi server is restarted. I think I can use 'nifi.flowcontroller.autoResumeState' to achieve that. Or would that start all processors? Yes, that's the purpose of this property and it's enabled by default. Did you try it and not see

Re: Mirroring nifi lists on gmane

In general these can be mirrored without the source list even knowing, Over years I found (personally) that gmane wasn't a very usable mirror (but exposed to many other indexers in general). In my searches Nabble and Markmail come up much more often and more pleasant to use. Nabble can be

Standard SSL context always requires keystore/truststore?

Hi, In my flow I need to hit an external URL over HTTPS (using PostHTTP). The https endpoint is using standard certificates which have root CA already bundled in browsers and JVMs. I'd expect to be able to simply interact with it and rely on bundled JVM root certs, but it doesn't appear to be

Re: PutKafka won't Put! "Failed while waiting for acks from Kafka"

Pat, which NiFi version is that? Which Kafka broker version on the other end? Andrew On Sat, Jun 11, 2016 at 12:15 PM Pat Trainor wrote: > Users, > > Has anyone come across PutKafka simply not wanting to send to Kafka? > > In this troubleshooting, I have replicated

Re: Scheduling using CRON driven on Windows OS

Keith, Was your processor running at all times? It has to be started and enabled. I guess sharing the cron expression and maybe a quick screenshot will help next. Andrew On Wed, Jun 15, 2016, 6:58 PM Keith Lim wrote: > I tried setting Scheduling Strategy property to CRON

Re: MergeContent: Correlation Attribute Name syntax for matching syslog events

Hi, Are you doing anything special between Listen and Parse? Trying to understand the reasoning for why you split those. E.g. in default ListenSyslog mode I can see syslog.hostname correctly set. Could it be MergeContent settings? Maybe worth sharing its config. Andrew From: Andre

Re: NiFi logging

Take a look at ReportingTask, this sounds like a good fit. Much higher level than explicitly putting log steps in the flow or reconfiguring the logging system to forward a category elsewhere. HTH, Andrew [1] https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#reporting-tasks

Re: processor stats and count

I was reviewing the ListenSyslog processor and noticed how it exposes message counters. These are then available in the admin UI, and I suppose, would be returned in that API call result as well. I haven't found any mention of these counters in user or dev guides, did I miss it? I would like

Re: Dataflow architecture for multiple sources

Aurélien, The choice of a multiplexing channel or multiple dedicated ones is really up to any constraints your environment may (not) have. E.g. if you are able to expose every port required for a socket-based protocol or no. On the NiFi side, take a close look at Backpressure here, it will

Re: Nifi JSON event storage in HDFS

Sumo, True, MapR FS implementation may have compatibility issues. Additionally, things are complicated by a need to bundle some of their proprietary jars which can't be redistributed with NiFi. We at Hortonworks, have enabled some of our customers to have NiFi and MapR working together

Re: Disabling Data Provenance

While not about provenance, but related. One can change log levels for NiFi on the fly - edits to the logback.xml file will be reloaded after ~30 seconds without the need to restart the instance. Andrew On 3/1/16, 3:30 AM, "Shekhar Bansal" wrote: >Thanks Joe. >In

Re: Refer to original FlowFile after AttributesToJSON processor

Igor, this is a common pattern for such systems, you need to 'save' old content in advance. E.g. as a property of the message, then you can transform the content and still have access to the original value (via an attribute value on the message). Keep in mind, though, that you don't want to

Re: NiFi Clustering Issue

Mark, There are some improvements which can be made in a clustering setup. Maybe this one is related? https://issues.apache.org/jira/browse/NIFI-1190?filter=-2 Andrew From: Mark Payne > Reply-To:

Re: NiFi Clustering Issue

Chris, We often see real-life NiFi deployments on servers with up to 3 network interfaces. Really, as much as I'd like NiFi to be smart about it, as a deployer, one must guide it and tell which interface to bind to, no guessing. Andrew From: Christopher Wilson

Re: NiFi - Configuration option

Manish, Maybe not available as a variable necessarily (this will come later), but you can apply properties when you deploy your template. Basically, set them directly as part of the deployment. Take a look https://github.com/aperepel/nifi-api-deploy Andrew From: Manish Gupta 8

Re: Logstash/ Filebeat/ Lumberjack -> Nifi

Conrad, Set up a site-to-site connection between nifi edge nodes and your main processing cluster running a bigger nifi instance. This is the 'application' level protocol native to NiFi. MiNiFi, in turn, uses it under the hood as well, which will ease migration for you in the _near_ future ;)

Re: Json Split

Try SplitText with a header line count of 1. It should skip it and give the 2nd line as a result. Andrew From: Madhukar Thota > Reply-To: "users@nifi.apache.org"

Re: Sensitive values

Ravi, As Simon mentioned, the current behavior is an intentional design choice. Typically one would set properties like these in different environment, but not retrieve them. Take a look at https://github.com/aperepel/nifi-api-deploy for an example of how applying environment-specific

Re: ExecuteProcess (fetch output)

Sven, take a look at ExtractText component, it will allow you to promote a result into an attribute. Andrew On Sat, Jul 2, 2016, 7:41 PM Sven Davison wrote: > I’ve been trying to run a script and fetch said output from the script as > a variable work with. The process

Re: nifi at AWS

Isn't it more advisable to use the HTTP mode instead, i.e. no additional ports to open? Make sure to change the client RPG mode to http from RAW (in the UI). Andrew On Sun, Jan 22, 2017, 10:47 AM Bryan Bende wrote: > Hello, > > I'm assuming you are using site-to-site since

Wait/Notify State in a Cluster

Hi Guys, I've noticed the upcoming version has wait/notify set of processors (great write ups by Koji, avain). Does it support clustered state, though? Planned maybe? For high throughput scenarios, is it possible to switch it out to algernative implementations if e.g. default relies on a ZK?

Re: Nifi taking forever to start

I'm not sure piggy-backing on the host entropy will work reliably. I have seen this issue in ec2, openstack boxes, etc. A newly spun up box will exhibit this issue often. Andrew On Wed, Feb 15, 2017, 10:09 AM Bryan Rosander wrote: > Hey Arnaud, > > Andy's solution is

Re: Re: new Nifi Processors

Basically the GPL license puts restrictions on how one can distribute in practical terms. Meaning your work may live under GPL license as long as it's not part of the official package. End users will have to download your NAR themselves. Andrew On Wed, Mar 1, 2017, 8:43 AM Matt Burgess

Re: Processor to send flowfile to two different destinations?

It's actually very simple - connect a processor output to 2 or more other processors or ports, use the 'success' relationship if prompted to choose from multiple. Andrew On Tue, Sep 13, 2016, 6:14 PM Russell Bateman < russell.bate...@perfectsearchcorp.com> wrote: > *DuplicateFlowFile* sends

Re: read in values from nifi.properties in a Groovy ExecuteScript processor

Which NiFi version? With 1.0 there are some bits for variable registry available, basically one can reference values from external config files via regular EL expressions. Andrew On Fri, Sep 23, 2016, 6:00 PM Tom Gullo wrote: > I want to read in values from nifi.properties

Re: SelectHiveQL Error

t;mattyb...@gmail.com>] > *Sent:* Thursday, October 06, 2016 5:17 PM > *To:* users@nifi.apache.org > *Subject:* Re: SelectHiveQL Error > > > > Andrew is correct. Although the HiveServer > > 1 driver is included with the NAR, the HiveConnectionPool is hardcoded to > use the

Re: UI: flow status and counters feedback

s see what others think and then I can work on filing a jira to > capture thoughts. > > Rob > > On Mon, Sep 19, 2016 at 6:22 PM, Andrew Grande <apere...@gmail.com> wrote: > >> Hi All, >> >> I'd like to provide some feedback on the NiFi 1.0 UI now that I had

Re: UI: feedback on the processor 'color' in NiFi 1.0

ssor, that the color of the drop shadow could be changed as well. >>> That would provide more contrast, but preserve readability, in my opinion. >>> >>> On Mon, Sep 19, 2016 at 6:39 PM Andrew Grande <apere...@gmail.com> >>> wrote: >>> >>&

Re: UI: feedback on the processor 'color' in NiFi 1.0

we could explore other ways to call these things > out so users do not come up with their own methods. Perhaps there are layer > options, like on a map (e.g., "show terrain" or "show traffic"). > > Rob > > On Tue, Sep 20, 2016 at 11:23 AM, Andrew Grande <apere..

Re: UI: flow status and counters feedback

Alright guys, do we have enough consensus to start filing jira work items? :) Andrew On Tue, Sep 20, 2016, 2:01 PM Andrew Grande <apere...@gmail.com> wrote: > Let's fade the connection slowly to an inverted if backpressure engages? > > On Tue, Sep 20, 2016, 1:17 PM Rob Moran &l

Re: Configure Multiple NCM

Hi Tijo, Take a look at clustering docs for NiFi 1.0. Zero-master clustering changed a few things, any node can be elected to be a primary or coordinator now. The 0.x concept of NCM is gone. >From the UI access standpoint, one can hit any node in a cluster to get the same experience. Andrew On

UI: flow status and counters feedback

Hi All, I'd like to provide some feedback on the NiFi 1.0 UI now that I had a chance to use it for a while, as well as pass along what I heard directly from other end users. Attached is a screenshot of a status bar right above the main flow canvas. The biggest difference from the 0.x UI is how

Re: nifi Rest API to get full details of the flow.

This isn't an ideal approach, IMO. There is a standard API to get a summary of the flow and,status of every processor, check what the Summary tab is invoking for a URL. You can then drill into any specific component by ID. Andrew On Wed, Sep 28, 2016, 6:42 AM Sandeep Khurana

Re: Enable Compression on Remote Port?

Disable transmission on RPG, go into the ports view again. Now, you should be able to modify settings like compression, concurrent threads and security on ports. Andrew On Thu, Nov 10, 2016, 2:26 PM Peter Wicks (pwicks) wrote: > When I have a Remote Process Group and I view

Re: stream one large file, only once

Neither GetFile nor FetchFile read the file into memory, they only deal with the file handle and pass the contents via a handle to the content repository (NiFi streams data into and reads as a stream). What you will face, however, is an issue with a SplitText when you try to split it in 1

Re: Data Provenance is not available

Yes. It's a combination of a generic view data permission in the global menu and specific access in the processing group. Andrew On Mon, Nov 21, 2016, 12:28 PM Pablo Lopez wrote: > Hi, > > Any ideas as of why Data Provenance option is grayed out (not available) >

Re: Delay Processor

Oleg, I'll break my response in 2 threads. I understand your use case of 'delay until X', but frankly would design it differently if we're talking about long-term transactions or schedules. It may involve systems external to NiFi. Anyway, I'd like to keep this use case out of scope for the delay

Re: Delay Processor

Joe, It's good to know some thinking went into this feature before. Basically, I'm trying to put a spotlight on these 2 areas: 1. Making it, potentially, more generic than a retry loop. E.g. enhance the ControlRate processor. 2. Making these policies more explicit, so a user wouldn't

Re: unable to empty the connection queue between 2 processors in NIFI secure cluster

Hi, There are 2 levels basically. One is the global policies in the top right menu. Another is in the operator menu on the left and is specific to every processing group. Sometimes you need a combination of both to allow for an action. E.g. try data provenance and modify data permissions to

Re: NiFi 1.0.0 canvas background

Hi Russ, Alignment or any snapping to a grid wasn't there before, but that would be a very welcome feature, I agree. Regarding the background, I personally didn't notice a change, isn't it the same? Andrew On Tue, Nov 15, 2016, 3:14 PM Russell Bateman < russell.bate...@perfectsearchcorp.com>

Delay Processor

Hi, I'd lime to check where discussions are on this, ir propose the new component otherwise. Use case: make delay strategies explicit, easier to use. E.g. think of a failure retry loop. Currently, ControlRate is somewhat related, but can be improved. E.g. introduce delay strategies a la

Re: How I put the cluster down.

Hi, I'd suggest couple things. Have you configured backpressure controls on connections? NiFi 1.0.0 adds 1evt/1GB by default IIRC. This can help avoid overwhelming components in a flow. Next, the 2 core CPU is really inadequate for high throughput system, see if you can get something better.

Re: Trouble with PublishKafka10

2 things to check: 1. Are you connecting to Kafka 0.10 broker? 2. Which port are you using? Recent Kafka clients must point to the kafka broker port directly. Older clients were connecting through a zookeeper and used different host/port. Andrew On Sun, Oct 30, 2016, 1:11 PM Daniel Einspanjer <

Re: How to integrate a custom protocol with Apache Nifi

Kant, Look into a custom processor. You have a choice of either implementing e.g. a parser for your data, or, if the protocol is more involved, implement a receiver in the processor as well which would spit out meaningful data messages next. Andrew On Tue, Dec 6, 2016, 7:47 PM kant kodali

Re: DetectDuplicate

Juan, no change from how you remember this processor yet. I personally would love to have a more pluggable backend for it, too. Andrew On Mon, Dec 19, 2016, 2:35 PM Juan Sequeiros wrote: > Hello, > > I am wondering if DetectDuplicate still has single dependency on >

Re: Change data capture processor

Yes, it's in 1.2.0. However, it's unreleased yet. Should be soon, you can track progress by following the 1.2.0 release email thread on tbe dev list. Andrew On Wed, Apr 12, 2017, 1:55 AM Buntu Dev wrote: > Is the new CaptureChangeMySQL processor available in 1.2 and is

Re: Logging changes to workflow

John, the Flow Configuration History menu item already captures it. Does it look like a fit? Andrew On Sun, Apr 9, 2017, 10:50 PM HARRIOTT, John wrote: > Is it possible to log changes to a workflow so they can be captured by a > central logging/auditing capability

Re: Clustering Best Practices?

BTW, your NiFi instance is not single-threaded, it's a single node. It still runs multiple worker threads in the flow. Andrew On Thu, Apr 20, 2017, 7:01 AM James McMahon wrote: > Good morning. I have established an initial single-threaded NiFi server > instance for my

Re: NiFi best practices to manage big flowfiles

ed recipient, please contact immediately the sender by > telephone, fax or e-mail and delete the information in this message > that has been received in error. The sender does not give any warranty > or accept liability as the content, accuracy or completeness of sent > messages and accept

Re: NiFi best practices to manage big flowfiles

Hi, First, there won't be multiple copies of a file within NiFi. If you pass around the content and don't change it (only attributes), it will merely point a reference to it, no more. You need to decide if you want to delete processed files, this is what GetFile does. Might want to look into

Re: Parameterizing the nifi flow

Hi, Read up on the variable registry in the docs, that sounds like a good fit. I don't remember if it were available in 1.1 though. Andrew On Fri, Aug 11, 2017, 5:12 PM More, Vikram (CONT) < vikram.m...@capitalone.com> wrote: > Hi, > > > > I have a nifi flow which pulls/extracts from source

Re: ExecuteSQL with MSSQL Server Express

Try with forward slashes, please. On Wed, Jul 19, 2017, 12:17 PM Praveen Reddy wrote: > Hi, > > I am using ExecuteSQL processor to connect with MSSql server express to > get some data. But I am getting the below exception: > > [image: Inline image 1] > > My Connection

Re: Processor classpath

It's not a classpath, but rather a user configurable location, exposed as a processor property. Does it help? Andrew On Tue, Jul 4, 2017, 4:32 PM James Srinivasan wrote: > Hi, > > I'm developing a processor which needs to read some of its config from > the

Re: NPE in ListenSyslog processor

I wonder if the cause of zero length messages is the health check from the f5 balancer. Worth verifying with your team. Andrew On Tue, Apr 25, 2017, 3:15 PM Andy LoPresto wrote: > PR 1694 [1] is available for this issue. > > [1] https://github.com/apache/nifi/pull/1694 >

Re: Back Pressure Object threshold not honored

I would also suggest the ControlRate processor in front of a sensitive step. There's a chance data may accumulate and sent in burst after downtime, so it ensures a predictable outflow. Andrew On Thu, Apr 27, 2017, 8:03 PM Joe Percivall wrote: > Hello Kevin, > > I

Re: Missing nifi-app.log files

Typically this is better handled by using the -F switch instead of -f, it has more robust file handling and manages files disappearing correctly. Unfortunately, some OS don't have that switch in their toolchain. Andrew On Thu, Aug 17, 2017, 5:07 PM James McMahon wrote: >

Re: csv output processor?

There is something interesting coming out in 1.2.0 potentially, the new pairs of RecordReaders/RecordSetWriters. I did see CSV format support in there. Take another look maybe. Andrew On Fri, Apr 28, 2017, 4:55 PM Frank Maritato wrote: > Is there a nifi processor

Re: Apache nifi 1.0.0 consuming high CPU utilization

What is your flow doing? It very much depends on this. While 6 cores is not a great hardware to run a full NiFi system on, I'd guess ReplaceText and EvaluateJsonPath are the hotspots. See how much data queues up in front​ of these processors. Andrew On Tue, May 2, 2017, 5:09 AM kunal

Re: Input flowfile for GetHTTP

Try with InvokeHttp. Andrew On Tue, May 2, 2017, 3:42 AM Buntu Dev wrote: > I'm trying to connect ExtractText to GetHTTP processor but that doesn't > seem to be allowed. Wanted to check if that is true that GetHTTP processor > doesn't accept input flowfiles and if so, how

Re: Publish with persistence using PublishAMQP?

James, you make too much sense :) Mind filing a jira for that enhancement? Thanks, Andrew On Thu, May 4, 2017, 9:56 AM James McMahon wrote: > amqp$deliveryMode > > from the documentation: > Attributes extracted from the FlowFile are considered candidates for AMQP >

Re: Quickly find queues with data...

Pro tip - this connection backlog info is available in the json returned from the status API. This is a good way to integrating with monitoring system via polling. Andrew On Fri, May 5, 2017, 5:41 PM Russell Bateman wrote: > Ah, I see. Thanks very much. (I spend too much

Re: Re: QueryDatabaseTable - Deleted Records

As an interesting architectural approach we took eons ago, before NiFi, was to take daily snapshots of a full table. Every row would then be hashed/digested or in any other way uniquely identified and 2 datasets would be crossed and compared to find inserts/deletes/updates. It was involved, but

Re: org.apache.nifi.spark.NiFiReceiver and spark spark.streaming.receiver.maxRate

A typical production setup is to use Kafka in the middle. Andrew On Mon, Sep 18, 2017, 3:02 AM Margus Roo wrote: > Hi > > I need to take flow with Spark streaming from Nifi port. As we know > Spark supports spark.streaming.receiver.maxRate and >

Re: Polling Processors impact on Latency

Yes, polling increases latency in some cases. But no, NiFi is not just polling. It has all kinds of sources, and listening vs polling vs subscribing purely depends on the protocol of that given processor. Hope this helps, Andrew On Tue, Nov 7, 2017, 1:39 AM Chirag Dewan

Re: PDF generating processor

Hi Mike, Thanks for sharing this. May I also suggest you provide a binary NAR download on the releases page in this github? The NiFi build pupeline is non-trivial, it would help folks get started quickly. Andrew On Fri, May 11, 2018, 7:40 AM Mike Thomsen wrote: > A few

Re: Allowing all users to connect

Juan, A cert implies one knows the identity of the cert holder. I'd imagine if you shared it with multiple users, you would have achieved this semi-anonymous requirement. I would take a really deep look into why you want to do it this way, though. Defeats the purpose of security. Is there a

Re: Only get file when a set exists.

Martijn, Here's an idea you could explore. Have the ListFile processor work as usual and create a custom component (start with a scripting one to prototype) grouping the filenames as needed. I don't know of the number of files in a set is different every time, so trying to be more robust. Once

Re: Dynamically adding avro schema to AvroSchemaRegistry at runtime

Hi, Using an external schema registry is the way to go. The embedded one is meant for ease of use, but once you grow beyond an initial phase, the best practice is to have a full service and potentially use standard InvomeHTTP to perform operations beyond just lookups. Does it help? Andrew On

Re: Back pressure deadlock

I wonder which jms broker you are using. The situation where a jms destination is full is absurd, the whole point was to decouple publishers and consumers. I would additionally look into what jms broker settings are available to address the situation. Andrew On Mon, Oct 23, 2017, 10:32 AM Arne

Re: Persistence mechanism in Nifi

I don't think you need to do anything. If your target is down, the data is stored in the NiFi connection. You can raise the backpressure limits as long as you have enough disk space (I don't recommend disabling it). Andrew On Thu, Jan 18, 2018, 12:31 AM Vikram KR wrote: >

Re: NiFi Registry Create Item in the UI

n > at the top-level which would create the flow and the first version as > one operation, I view this as what you are calling "New Flow", and > then there could be an "Import Version" action from the actions > drop-down of a specific flow. > > -Bryan &

NiFi Registry Create Item in the UI

Hi, I wanted to share some feedback and suggestions on the registry UI. It was a little unnatural to look for a New Bucket button in the Settings menu. Same sentiment with the Create Flow for when we expect to be able to import it in the UI in the near future. How about making the (+) or New

Re: Routing Files Based on Dynamic Pattern Matching

lah.csv, I might have > a regular expression defined that says if the file name matches > .*blah_test_blah(?!_somethingelse).* then it should goto directory > /tmp/blah_test > > > > Thanks > > Shawn > > > > *From:* Andrew Grande <apere...@gmail.com> > *S

Re: Registry, sdlc and promotion between environments

Yes, Georg, there's something coming up to address exactly that, please take a look at https://github.com/apache/nifi/pull/2477 Andrew On Wed, Feb 21, 2018, 2:45 AM Georg Heiler wrote: > Hi > > Can I use the new nifi registry to promote / move my nifi flows between >

Re: Routing Files Based on Dynamic Pattern Matching

Maybe take a look at one of the ScanContent/Attribute processors? It allows to map in a reloadable external file. Or, better yet, one of the LookupService variants, which is more generic. HTH, Andrew On Mon, Feb 19, 2018, 3:10 PM Shawn Weeks wrote: > Hi, I’m looking

Re: Modularize nifi flows

A Processing Group would serve as a reusable unit, which then has version control applied to it. You would model your flows to abstract interactions with e.g. in/output port objects. They also mentioned that nested PG can be versioned separately, looks like git submodules behavior, but I haven't

Re: AVRO is the only output format with ExecuteSQL

gt; >> [1] https://issues.apache.org/jira/browse/NIFI-4517 >> On Tue, Aug 7, 2018 at 9:20 AM Bryan Bende wrote: >> > >> > I would also add that the pattern of splitting to 1 record per flow >> > file was common before the record processors existed, and generally >

Re: AVRO is the only output format with ExecuteSQL

some refactoring of the parent > #onTrigger() to service methods. > > > > > > Andy LoPresto > > alopre...@apache.org > > alopresto.apa...@gmail.com > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > > On Aug 7, 2018

Re: AVRO is the only output format with ExecuteSQL

Careful, that makes too much sense, Joe ;) On Tue, Aug 7, 2018, 8:45 AM Joe Witt wrote: > i think we just need to make an ExecuteSqlRecord processor. > > thanks > > On Tue, Aug 7, 2018, 8:41 AM Mike Thomsen wrote: > >> My guess is that it is due to the fact that Avro is the only record type >>

Re: Multiple registry instances sharing Git repo

There are also registry event hooks to help with the automation. Andrew On Tue, Aug 7, 2018, 12:46 PM Bryan Bende wrote: > Mike, > > It is not really made to have multiple instances pointing at the same git > repo. > > Each registry has a metadata DB and a flow storage component which can > be

Re: docker nifi cluster not working

I think the idea is, if there is anything available today, it will have to be built from sources, including the new docker image. I.e. it's not a public image yet. Andrew On Fri, Jul 20, 2018, 7:20 AM Chris Herssens wrote: > Hello Mike, > > Where can I find nifi 1.8 image for docker ? > I get

Re: NiFi Registry with nested PGs

I would also check the poll period. E.g. give it 40 secs or more to detect new versions in the registry, it's not real time. Andrew On Thu, Jul 19, 2018, 6:35 AM Mike Thomsen wrote: > Thanks, Kevin. If I get a chance I'll try it out again because it appeared > to be the case that after the

Re: [Nifi 1.5.0] DistributedMapCacheClientService fails to open port

It's a client, it connects to a cache service. You need to start another service and point the client to it. Andrew On Tue, Apr 10, 2018, 6:29 AM françois lacombe wrote: > Hi all, > > Does anyone experience problems with DistributedMapCacheClientService > service?

Re: NiFi cluster with DistributedMapCacheServer/Client

HBase-backed cache is a great choice here. Redis is nice and nimble, but when it comes to clustering and enterprise security, may not be the best fit. The original legacy cache server in NiFi is... well, it should be deprecated and removed, IMO :) Andrew On Fri, Apr 13, 2018, 3:45 PM James

Re: how to edit queue content

How is it going to work with e.g. 20GB of events in the queue? I'd be careful, as requirements blow up into a full db with indexes, search, and a UI on top. If one wanted to filter events, wouldn't a standard processor do the job better? Andrew On Tue, Mar 27, 2018, 12:11 AM Joe Witt

Re: 答复: put pictures from remote server into hdfs

I think the MOB expectation for HBase was around 10MB. I agree it will require some thought put in organizing the space and region server splits with column families, once this volume becomes significant. Andrew On Fri, Mar 23, 2018, 9:08 AM Mike Thomsen wrote: > Off

Re: setting processor concurrency based on the development/production environment

Hi Andrew, > > thanks for the idea. I've been playing with nipyapi recently so might give > this a try. > > Thanks > > On Thu, Mar 1, 2018 at 7:32 PM, Andrew Grande <apere...@gmail.com> wrote: > >> Boris, >> >> Here's an idea youncould explore _today

Re: setting processor concurrency based on the development/production environment

Boris, Here's an idea youncould explore _today_. Assume your dev and prod flows live in different bucket/registry instance. Given that you are trying out NiFi 1.6, you should be able to extract the versioned flow from DEV and process it to change the concurrency level for PROD before committing

Re: NiFi Toolkit CLI issues with NiFi/Registry SSL handshake

Are you running in some container environment? It should never take 15 secs and there's no caching performed by cli. I would review the container environment and see why it's taking forever to start. Andrew On Sun, Oct 28, 2018, 10:11 AM ara m. wrote: > The Keystore user had "view processor"

Re: NiFi Toolkit CLI issues with NiFi/Registry SSL handshake

Guessing - if it's a new container on the first invocation, maybe the JVM is generating the binary class cache? This operation is performed only once ever, but with a clean environment every time I can see it being invoked again and again. Other than that, you'd need to connect a profiler and see

Re: Preferred schema registry

Isn't AvroSchemaRegistry an embedded one to NiFi? I.e. it's not a comparable alternative to an external dedicated schema registry. Andrew On Wed, Jan 16, 2019 at 7:43 AM Mike Thomsen wrote: > I think I figured out what it was. We backed out a change and it was > fungible with an earlier

Re: NiFi JSON enrichment

James, The easiest would be to merge json in a custom processor. Not easy as in no work at all, but given your limitations with the NiFi version could be done sooner maybe. Andrew On Mon, Dec 17, 2018, 9:53 AM James Srinivasan wrote: > Hi all, > > I'm trying to enrich a data stream using

Re: Truststore/Trusted hostname

Walter, you could point to the default JRE truststore file, maybe. Andrew On Wed, Jan 9, 2019, 7:12 AM Kevin Doran wrote: > Hi Walter, > > I could be mistaken, but my interpretation of the Trusted Hostname > configuration option is that it is designed to work with/in-addition-to the >

Re: how to organise processor or PG in the Nifi canvas ?

Try importing via thr NiFi cli tool. I've implemented the logic to auto layout newly imported PGs. But I agree, this should be something included in the core NiFi logic itself. Andrew On Fri, Sep 14, 2018, 4:17 PM Dominique De Vito wrote: > Hi, > > Is there a way to re-organize automatically

Re: Variables handling in MiNiFi

Luie, Which version are you looking at? I think there was a recent discussion about supporting those in MiNiFi, not sure if it got imoleme ted already, though. Andrew On Wed, Dec 19, 2018, 9:26 AM luis_size wrote: > Hi > > I am a big fan of NiFi variables registry. This makes instantiating >

Re: Weird ListFile Issue

Looks like the processor started listing CWD On Fri, Mar 22, 2019, 2:00 PM William Gosse wrote: > The real var name was aimuploaddir and the error was aimduploaddir. Think > I have a work around to prevent the tragedy was wiping out Nifi. I did this: > >

Re: PutKafka use with large quantity of data?

What's the concurrency for these processors? What's a global NiFi thread pool size? I wonder if you might be running out of available threads while they are waiting for external system i/o under load. Andrew On Thu, Apr 4, 2019, 8:24 AM l vic wrote: > What's this particular processing group

Re: Log Queries being executed by PutDatabaseRecord

Could ve a good idea to log values at TRACE level then. On Wed, Feb 27, 2019, 7:56 AM Matt Burgess wrote: > True, at a DEBUG level we could output the record values, although for > large flow files this will be quite verbose :) Also the point of the > ?s is not necessarily to not show the

  1   2   >