Re: nifi toolkit

> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Sep 20, 2018, at 3:41 PM, Juan Sequeiros wrote: > > Hello all, > > Using nifitoolkit bin/tls-toolkit.sh > > While using client / server mode ( CA issuer ) > Is there any way to pass it par

Re: Nifi toolkit 1.7 additional question

All, Disregard the previous email. I was running the toolkit for my user cert in same directory that already had previous server certs in there. Once I removed them I was able to generate my "user" cert with -D On Thu, Sep 20, 2018 at 7:29 PM Juan Sequeiros wrote: > Hello, > &

Nifi toolkit 1.7 additional question

Hello, When using nifi toolkit 1.7.1 to get user cert I run this: ./tls-toolkit.sh client -c CAhost -t AlongTokenString -p AportNumber -D "CN=Juan Carlos Sequeiros,OU=NIFI" -T pksc12 And it fails with this return: 2018/09/20 23:23:57 INFO [main]

nifi toolkit

Hello all, Using nifitoolkit bin/tls-toolkit.sh While using client / server mode ( CA issuer ) Is there any way to pass it parameters like the ones using standalone mode? Want to be able to set the password through the command line Similar to how I can on standalone mode EX: Do this equivalent

Re: JoltTransformJson issues with date enrichment

xpression? > > Regards, > Matt > > On Tue, Aug 7, 2018 at 11:03 AM Juan Sequeiros > wrote: > > > > Hello all, > > > > I am trying to "enrich" my json. > > My problem happens when I try to give it a timestamp. > > My requirement is to give

JoltTransformJson issues with date enrichment

Hello all, I am trying to "enrich" my json. My problem happens when I try to give it a timestamp. My requirement is to give it time in this format: ${now():format("-MM-dd HH:mm",'Z')} So at the top level and every element of my json should have an entry called EX: "createTime":"2018-08-07

ElasticSearch 6

Hello all, I am getting unexpected behavior while using QueryElasticSearchHttp into an ES 6 instance. Does NIFI support ES 6? I can send it documents but when I query it I ingest my flowfile instead of getting a result back. At this point, I am not sure if I am just using the processor wrong or

Re: Allowing all users to connect

t gives > the group access to the root canvas. > > > On Thu, May 17, 2018 at 10:06 AM, Juan Sequeiros <helloj...@gmail.com> > wrote: > > Thanks, Andrew / Bryan. > > Yeah this right now is a prototype with a team of about 10 people. > > > > We all have c

Re: Allowing all users to connect

ake a really deep look into why you want to do it this way, > though. Defeats the purpose of security. Is there a problem issuing client > certificates? > > > Andrew > > On Wed, May 16, 2018, 9:50 PM Juan Sequeiros <helloj...@gmail.com> wrote: > >> Hello all, >> >

Allowing all users to connect

Hello all, Is there a way I can set up authorization to secured UI to anyone who has a valid cert with out knowing who they will be?

nifi on openshift

Good morning all, Curious if anyone using openshift to host nifi service has figured out how to make your "route" map to {NIFI_URL:PORT/nifi} As of now, we have the route hit {NIFI_URL} since so far can't figure out how to configure it to anything other than a hostname it doesn't like url:port

Re: Cluster Crashing When Nodes Unresponsive?

Hi what version of NIFI are you on? In my experience it’s usually resource issue and most of the time it’s disk I/O. On Tue, May 1, 2018 at 4:43 PM Jon Logan wrote: > We are running into issues where if a node is unresponsive for whatever > reason, the entire cluster seems

Am I doing this right? with regarding to records

Hello all, I have one file on local disk with thousands of lines each representing valid JSON object. My flow is like this: GetFile > SplitText > PartitionRecord ( based on a key ) > MergeRecord > PutElasticSearchRecord This works well, however, I seem to bottleneck at PartitionRecord So I

Re: MergeRecord, queue & backpressure

Good afternoon, Another thing to help you out maybe ... You can also tweak the nifi.properties setting: nifi.queue.swap.threshold=2 This setting will control the value of the max flowfile count on a connection if exceeded it will flush those flowfiles to disk. I am not sure however there

Re: Routing Files Based on Dynamic Pattern Matching

You could also RouteOnAttribute [1] doing the same regular expression checks using NIFI expression language [2]. For instance, I can create a route for "DATA A" if my attribute "filename" matches a regular expression. ${filename:find'SomeJavaRegEx')} If you can control your workflow maybe bring

Re: How to preserve original file name

Hi forgot to mention if posting to ListenHTTP You must set “HTTP Headers to receive as Attributes (Regex)” to .* Or something more restrictive to include “filename” Hope that helps. On Wed, Nov 29, 2017 at 7:14 PM Juan Sequeiros <helloj...@gmail.com> wrote: > Hi, > > My expe

Re: How to preserve original file name

Hi, My experience has been that I need to pass it as a header with filename “attribute” So — header “filename: AfileName” Sorry on my mobile. On Wed, Nov 29, 2017 at 6:43 PM tj5527 wrote: > I use the latest the nifi version (1.4.0) transforming xml to json. The > flow

Re: NiFi stops processing further flowfiles

and see what happens  > > > Does it also make sense to increase the "Maximum Event Driven Thread > Count"? > > > Finally, is there a way to monitor thread usage within NiFi? > > > Many thanks, > > Panos > > > > -- >

Re: NiFi stops processing further flowfiles

Good afternoon, My quick guess is you are running out of threads. If your system can handle it increase the default "max thread" count found: Left side controller settings > Maximum timer driven thread count On Fri, Nov 24, 2017 at 10:47 AM Panos Geo wrote: > Hello

Re: minifi release and mapping that to the nifi release

t are not present in any of the Java > variations of NiFi efforts. > > [1] https://issues.apache.org/jira/browse/MINIFI-408 > > On Tue, Nov 21, 2017 at 3:47 PM, Juan Sequeiros <helloj...@gmail.com> > wrote: > >> Hi all, >> >> Based on previous comment "

minifi release and mapping that to the nifi release

Hi all, Based on previous comment "I think it may need to line up with the version of NiFi that MiNiFi 0.2.1 is based on, which is NiFi 1.2.0" What is a quick way of knowing what version of NIFI that MINIFI was built on? And with knowing that, is it a good assumption that the processors found

HandleHttpRequest

Hello all, Is it considered a best practice to leave the thread count on HanddleHttpRequest to 1? Assuming the JETTY server is handling requests? are we making things worst by increasing that? We are moving towards using HanddleHttpRequest versus ListenHttp but we are struggling with

Global setting for backpressure

Good morning, Is there a way to globally change back pressure settings? To the point that new processors added to canvas would default to our set numbers? I was hoping for a NIFI properties approach. EX: I don't want to default to: Back Pressure Object: 10,000 Back Pressure Data Size: 1 GB

Re: ConsumerJMS throughput is degraded after running some hours

Hello, With out knowing more about your system specs and what else is going on with your flow, I suggest you look at increasing the max threads the nifi instance can run. I'm on mobile but it's under controller settings too right side of the UI. Thanks, Juan On Tue, Jul 25, 2017 at 1:57 PM

Re: Detect whether a flowfile has a particular attribute

Jim, This might be related and coincidentally today we were talking with a coworker about the "advanced" button of UpdateAttribute and its ability to set attributes based on conditions. It's pretty powerful. [1] It might come in useful for your efforts. [1]

Re: Execute Process with Command Arguments

Yan, To work around that we just call a shell script with the full command, from ExecuteStream. On Mon, May 22, 2017 at 8:36 AM Juan Sequeiros <helloj...@gmail.com> wrote: > Yan, > > There is a jira opened for the issue you describe.[1] > > > [1] https://issues.apache.

Re: Execute Process with Command Arguments

Yan, There is a jira opened for the issue you describe.[1] [1] https://issues.apache.org/jira/plugins/servlet/mobile#issue/NIFI-3221 On Mon, May 22, 2017 at 5:32 AM 刘岩 wrote: > > Hi All > > > Currently i try to use the 1.2.0 Nifi‘s execute process and below > the

Re: Archive selected "feeds"

uch hints about > retention be placed on the processors that create/modify/touch the > data because when you consider that most flows are graphs and > non-linear the decisions about retention become more complex. > > Thanks > > On Wed, May 17, 2017 at 12:40 PM, Juan

Archive selected "feeds"

All, On latest NIFI can we archive only specific FlowFiles based on an attribute? I can put in a feature request if not. For instance: archive.max.retention.period=12 hours archive.max.retention.attributes.period=5 days

Re: Fully virutalized Nifi cluster with shared storage

Ali, Yes that is correct. Reference section "Apache NIFI in depth" of the NIFI docs. [1] And I still use this article for general Apche NIFI best practices when handling high amounts of data. [2] It is tailored for Apache NIFI pre 1 release but still applies. [1]

Re: How to lock getfile upto putfile write into same file?

Prabhu, PutFile should pre-pend files with a "." while it is writing to it and then in the end will copy the "." file to its "filename" value. On GetFile side make sure that "ignore hidden files" is set to true. On Tue, May 16, 2017 at 1:29 AM prabhu Mahendran wrote:

Re: Back Pressure Object threshold not honored

Joe, Thanks for clarifying since I had seen the same thing. And chalked it up to research further if object meant same thing as FlowFile I just mention it because it's first instance that I notice when object is implied as FlowFile and it is confusing. Thx On Thu, Apr 27, 2017 at 8:16 PM

Re: List processors

al.WriteAheadLocalStateProvider >./state/local >false >16 >2 mins > > > Basically defining another instance of the local state provider as the > cluster provider. > > Not totally sure if this works, but theoretically it should. > > -Bryan > > &g

Re: List processors

s is a bug or working as expected Feels like the dataflow manager should configure how to handle state and be able to use ListS3 in cluster even though I do not have zookeeper? On Fri, Apr 21, 2017 at 10:57 AM Juan Sequeiros <helloj...@gmail.com> wrote: > Hello all, > > My preliminar

List processors

Hello all, My preliminary testing shows that if I run ListS3 ( maybe all list processors? ) processor on a cluster and that cluster is not running or configured to talk to zookeeper that he does not maintain state at all even though I would expect him to maintain state locally. EX:

Re: Provenance Event performance

Simon, I feel that " provenance event is emitted for each flowfile for each processor." is accurate understanding "each processor" means the unique processors the flowFile goes through. The provenance database is a lucene database and 1 million provenance events is not unreasonable. It would

Re: Provenance Event Use Cases?

Simon, We use NIFI's data provenance capabilities, to track the like cycle of a "flowFile" / data object as it goes through its system lifecycle. ( LINEAGE ) We also use it for troubleshooting as we can see the nifi attributes ( metadata ) and its content ( if configured ) You can also use

Re: UpdateAttribute Advanced tab

what you want. > > Regards, > Joe > > On Wed, Apr 19, 2017 at 11:37 AM, Juan Sequeiros <helloj...@gmail.com> > wrote: > >> Hello all, >> >> Using Advanced feature of UpdateAttribute >> I do not believe that there is a way to delete an attribute ( ac

UpdateAttribute Advanced tab

Hello all, Using Advanced feature of UpdateAttribute I do not believe that there is a way to delete an attribute ( action ) ? Example if attribute1 = A ( condition ) Delete attributeY But we need attributeY for many other flowfiles that dont have attribute1 and we "enrich" through this

Re: Some question

(step3)? > > > > Thanks, > > Ravi Papisetti > > > > *From: *Juan Sequeiros <helloj...@gmail.com> > *Reply-To: *"users@nifi.apache.org" <users@nifi.apache.org> > *Date: *Monday, April 17, 2017 at 11:14 AM > *To: *Richard Hanson <rhan...

Re: Some question

get server > > > Another minor question, is it enough just to replace flow.xml.gz with my > customized flow graph under conf directory (cos checking nifi.properties, > it shows nifi.flow.configuration.file points to conf/flow.xml.gz)? > > > Thank you again for the advice! > &

Re: Flow not loading in 1.2.0

Uwe, Could not be that your previous flow had the custom processors and with this upgrade you've not put them in lib dir? On Fri, Apr 14, 2017 at 12:29 PM Uwe Geercken wrote: > Hi, > > I just downloaded 1.2.0 and unpacked it to the usual place. Then I simply > copied my

Re: Avoiding Forced Refreshes that Result in Lost Changes

We currently have a solid dev / prod environment and when ready we import the template from dev in to prod. We also have updated central processors that can't be replaced by a template with rest calls. But in general NIFI 1 is a massive upgrade to U/I, and this feature and user policies from a

Re: Logging changes to workflow

Hi Joe / all, " we make it available via reporting tasks " What reporting task is that? I've looked at NIFI 1 available reporting tasks and I don't see anything that would let me export "flow configuration history". Coincidentally we are having to meet similar requirement. Our solution was to

Re: Options for increasing performance?

If you have wireshark you could use: tshark -f "port 8446 or port 9448" On Wed, Apr 5, 2017 at 3:45 PM James McMahon wrote: > Thank you Bryan. I will explore these things. I suspect we are not > receiving from the source optimally. Reason I say that is this: I am doing

Re: Help with SFTP processor

Good afternoon, I would try this command from command line: ssh -vvv -i user@server Example: ssh -vvv -i /some/path/.ssh/id_rsa nifi@10.10.10.10 If that works then I would double check the "private key path" property of your GetSFTP it should point to the fully qualified file to the private

nifi 1.1.1 force SSL

Hello all, I am trying to force SSLv3 on a ListenHTTP. I've changed that on my StandardSSLContextService but when I test it fails when I specify sslv3 and works when I specify tlsv1 curl example. Similar thing if NIFI * postHTTP inside same canvas to myself. Works fine if

tag processors with deprecated

Hi all, Lately I've been discussing interacting with messaging queues using JMS and I've had to stop others from trying to use GET and PUT JMS. I know that ideally if starting from new you should use Consume and Publish JMS with the JMS controller service connection factory. Initial impression

Re: RouteText / Replcace Text to remove first line of file

ine. ESC has an option to put > the > >> output of the command directly into a new attribute, and pass the > "original > >> contents" to the next processor. The value of the new attribute > contains > >> the first line, while the entire file remains

RouteText / Replcace Text to remove first line of file

Good afternoon all,, I am trying to remove the first line of a file if it has a certain word in it "ERROR" I know it will exist only in the first line ( I can not fix the reason why it gets put there ) These files are big and lots of them. and I can not find a "fast" fix to pop the first line

Minifi question

Good afternoon all, Is there a current way to look at a minifi to check on flowfile count or stats? And how do I view its provenance? Assume it does not have a rest API? Thx!

Re: Executestream command - stream closed

ExecuteStreamCommand will "Executes an external command on the contents of a flow file" Looks like you want to format the output of your ExecuteSQL, instead I would suggest. ListDatabaseTables > ExecuteSQL > ReplaceText > {do something else with the flowfile} > PutHDFS / MergeContent / PutFile

Re: ExecuteStreamcommand

quot;test\" I get: *\test\* On Fri, Dec 16, 2016 at 2:31 PM Juan Sequeiros <helloj...@gmail.com> wrote: > That did not work it could be my command. > The command takes an argument in double quotes. "HOUSECOLOR" and outputs > BLUE > > On Fri, Dec 16, 2016 at 2:10 P

Re: ExecuteStreamcommand

e quote it would be \\" as the property value. Joe - - - - - - Joseph Percivall linkedin.com/in/Percivall e: joeperciv...@yahoo.com On Friday, December 16, 2016 2:05 PM, Juan Sequeiros <helloj...@gmail.com> wrote: Hey Joe, That did not work. As example if I use echo "test"

ExecuteStreamcommand

Hi all, Escape character "\" does not seem to work on ExecuteStreamCommand "command arguments" section, any suggestions? I want to escape double quotes but the command is taking the backslash as literal. thanks

Apache Minifi questions

Hello all, Will latest MINIFI work with Apache NIFI 0.7? In the sense that I build my dataflow and then deploy the dataflow to Apache MINIFI edge node to run? How does the provenance of Apache MINIFI work how can I view it, we will not have Apache NIFI 1 at the "lake" Thank you

Re: PutKafka won't Put! "Failed while waiting for acks from Kafka"

Hi Pat, Looks to me that on your PutKafka you are pointing it to your zookeeper instance port 2181 you should point it to your kafka broker. Port 9092 ( I think ) Not home to verify ... On Sat, Jun 11, 2016 at 3:31 PM, Pat Trainor wrote: > *OK... Prep'ing kafka:* > >

Re: GetHBase and "Connection Refused"

4) 296-7644 mobile > www.centricconsulting.com | @Centric <https://twitter.com/centric> > > On Wed, Mar 23, 2016 at 8:07 PM, Juan Sequeiros <helloj...@gmail.com> > wrote: > >> Hi you also need the controller service DistributedMapCacheServer >> configured.

Re: Search provenance data based on an attribute

Sudeep, Add it to your nifi.properties. This line: EX: nifi.provenance.repository.indexed.attributes=locarion And yes they will be constant. On Fri, Mar 11, 2016 at 1:40 PM, sudeep mishra wrote: > Hi, > > How can I search provenance data based on specific

Re: ExecuteSQL and NiFi 0.5.1 - Error org.apache.avro.SchemaParseException: Empty name

I wonder if on the controller service DBCPConnectionPool associated to your ExecuteSQL processor you have something that can't be found since it's stored on your older release. On Fri, Mar 4, 2016 at 11:12 AM, Marcelo Valle Ávila wrote: > Hello community, > > I'm starting my

Re: Nifi 0.50 and GetKafka Issues

Excuse me my message got cut short. I point you to the state management section since you mention you are using zookeeper and you are running nifi in cluster mode. The default cluster state provider is ZookeeperStateProvider Thanks On Sat, Feb 20, 2016 at 7:29 PM, Juan Sequeiros <hel

Re: Add date in the flow file attribute

Sudeep, You can pass the attributes by expressing them like this: ${key} Or AttributeToJson: " *Destination* flowfile-attribute - flowfile-attribute - flowfile-content Control if JSON value is written as a new flowfile attribute 'JSONAttributes' or written in the flowfile content.

Re: polling nifi status -- nifi reports not running every 10 to 20 min

Good afternoon, We monitor our instance through a cron and checking its command output using service nifi status. I can only assume that you might have external factors causing the result to not meet true. I suggest that you also send the return result to a file to inspect it closer. So if you

Re: Nifi Kite-dataset URI issue

You can also make sure there is no "white space" on your value. Copy and paste can do this . On Mon, Dec 14, 2015 at 2:04 PM, Alan Jackoway wrote: > Try adding the path to hive-site.xml. > > Did it give you any error message beyond just "cannot validate url"? Is > there

Re: Wait from multiple inputs before ending the flow

Good afternoon, All processors have a scheduling tab or run every X time. I would set it at your last invokeHttp. If time is important look at the prioritizer settings too. So that you can send through as example a more important flowFile. On Dec 12, 2015 13:48, "Louis-Étienne Dorval"

Re: queued files

You can also check the NiFi logs for a searchable id or for what the previous processor ID produced to help search provenance. On Nov 19, 2015 21:22, "Bryan Bende" wrote: > Charlie, > > The behavior you described usually means that the processor encountered an > unexpected