Re: Unit test is getting failed for custom ,Awscredentialprovidercontrolerservice

2019-10-22 Thread Bryan Bende
You shouldn't be modifying the service code to make the test pass. You need to set whatever properties are needed to make it valid by using Runner.setProperty(service, property name, value) On Tue, Oct 22, 2019 at 6:54 AM Otto Fowler wrote: > >

Re: Sudo service nifi command not working

2019-10-17 Thread Bryan Bende
It is best to email only one list, so dropping dev... If you are starting the service with sudo then its the root user starting it, so you need to check if echo "$JAVA_HOME" works as root. On Thu, Oct 17, 2019 at 10:19 AM Praveen Anguralia wrote: > > Hi team, > > I need your inputs on an issue

Re: High CPU consumption

2019-10-16 Thread Bryan Bende
Hi Luis, Can you describe the part of the flow that turned out to be a problem a little more? Was it a port on the root canvas used for s2s that was then connected into a process group where everything inside was disabled? And what did you do to solve the problem, did you stop the port?

Re: Apache Nifi with IBM Event Streams

2019-10-10 Thread Bryan Bende
You can't create a TLS connection to something if you don't have access to the public key of the CA. This is how all TLS/https works, your browser has the public keys of all the major CAs in order to trust the websites you visit over https. If IBM Cloud is generating certificates and signing them

Re: Apache Nifi with IBM Event Streams

2019-10-10 Thread Bryan Bende
You need to set the truststore filename and password. The truststore needs to have public key of the CA that signed the certs used by the Kafka broker (i.e. it needs to trust the certs used by the Kafka broker). On Thu, Oct 10, 2019 at 10:00 AM dkommineni wrote: > > I think I have messed up

Re: Apache Nifi with IBM Event Streams

2019-10-10 Thread Bryan Bende
https://issues.apache.org/jira/browse/NIFI-4820 On Thu, Oct 10, 2019 at 9:56 AM Joe Witt wrote: > > You can use SASL/Plain today with the kafka 2 procs in NiFi but as noted it > is unpleasant to configure. A much easier/clear configuration is being > worked on right now. Not positive what

Re: UpdateRecord Unexpected EOF

2019-09-26 Thread Bryan Bende
we might have a bug there. Turns out the EOF error is > because I was Record Path mode instead of Literal Value. Something I missed > from the documentation. > > Thanks > Shawn Weeks > ________ > From: Bryan Bende > Sent: Wednesday, Septem

Re: How to create custom processor which needs 2 or more NiFi bundles?

2019-09-25 Thread Bryan Bende
1) You can only have one parent NAR, and it creates a hierarchy, so if you need hadoop libraries you would want your parent NAR to be nifi-hadoop-libraries-nar, which already has a parent of nifi-standard-services-api-nar, so then you have both in your parent hierarchy. 2) This is really up to

Re: UpdateRecord Unexpected EOF

2019-09-25 Thread Bryan Bende
Not sure if this is reason but I believe record path would be /log_date instead of //log_date On Wed, Sep 25, 2019 at 1:14 PM Shawn Weeks wrote: > > Trying to figure out what's causing this issue. In a simple test of > UpdateRecord I'm getting an exception. This is on NiFi 1.9.2 > > 2019-09-25

Re: node died unexpectedly

2019-09-25 Thread Bryan Bende
Maybe something interesting nifi-bootstrap.log when this happens? On Wed, Sep 25, 2019 at 11:51 AM Jean-Sebastien Vachon < jsvac...@brizodata.com> wrote: > I could if the Nifi was up and running but when I loose the node, the Nifi > process itself dies so I can hardly take a heap dump of

Re: implementing policies through REST interface

2019-09-24 Thread Bryan Bende
t; Le 24/09/2019 à 16:25, Bryan Bende a écrit : > > The best way to figure out the REST calls would be to use the UI while > > you have Chrome Dev Tools open and go through the process of creating > > the policies you are interested in and then you'll see the requests > >

Re: implementing policies through REST interface

2019-09-24 Thread Bryan Bende
The best way to figure out the REST calls would be to use the UI while you have Chrome Dev Tools open and go through the process of creating the policies you are interested in and then you'll see the requests that are made. In terms of the a REST client, there isn't really an official client, but

Re: Unsubscribe

2019-09-20 Thread Bryan Bende
You will need to email the appropriate unsubscribe address [1] For the users list it is: users-unsubscr...@nifi.apache.org [1] https://nifi.apache.org/mailing_lists.html On Fri, Sep 20, 2019 at 4:55 PM Paul Martin wrote: > > > > On Friday, 20 September 2019, 20:20:47 BST, Christopher J.

Re: Nifi: Replicate or Put file on all nodes of cluster

2019-09-20 Thread Bryan Bende
Currently there isn't really a standard approach for this, but you might be able to make part of the flow get the file from somewhere. For example, if you had an SFTP server or HDFS, you could put it there and have processors like ListSFTP -> FetchSFTP -> PutFile. On Fri, Sep 20, 2019 at 3:20 PM

Re: Stateful Dataflow Moved to New Cluster

2019-09-20 Thread Bryan Bende
You could copy the flow.xml.gz, as long as the sensitive properties keys are the same in each nifi.properties. Otherwise the new environment won't be able to decrypt the encrypted sensitive properties in flow.xml.gz. On Thu, Sep 19, 2019 at 10:41 AM Noe Detore wrote: > > I may have gotten a

Re: NiFi active thread count is no more than 10 ?

2019-09-18 Thread Bryan Bende
The overall timer-driven thread pool defaults to 10 (configured from the controller settings in top right menu). So even if a processor has 100 concurrent tasks, there are still only 10 threads available. On Wed, Sep 18, 2019 at 8:20 AM Joe Witt wrote: > > Hello > > The 100 threads for the

Re: Use Array Type On Avro Schema

2019-09-17 Thread Bryan Bende
What is the error you are getting? It looks like you don't have a type specified for any of the array fields. Example: {"type": "array", "items": "string"} Also, all the arrays are empty in the example json. On Tue, Sep 17, 2019 at 7:40 AM Wesley C. Dias de Oliveira wrote: > > Hello,

Re: Use of NiFi Registry’s Bundles / Extensions

2019-09-12 Thread Bryan Bende
Hello, The APIs are usable, but we really need NiFi 1.10.0 to be released first because in order to generate a NAR that can be version controlled in registry, the NAR must be built against nifi-api 1.10.0. If you are feeling ambitious you could build NiFi's master branch and build your NAR's

Re: Nifi Cluster Untrusted Proxy Error

2019-09-05 Thread Bryan Bende
016c-1000-8396-950d03ad5e07" > action="W"> > > > resource="/operation/processors/b3c8228a-016c-1000-8e36-f4315d3da34c" > action="W"> > > > resource="/operation/p

Re: Nifi Cluster Untrusted Proxy Error

2019-09-03 Thread Bryan Bende
Please show authorizations.xml, thank you. Also, you shouldn't really be using wildcard certs - https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#wildcard_certificates On Tue, Sep 3, 2019 at 5:32 AM Dweep Sharma wrote: > > Can someone take a peek at this - what could be wrong?

Re: FreeFormTextWriter and flowfiles attributes

2019-09-03 Thread Bryan Bende
FreeFormTextWriter allows you to use expression language to reference fields in the record, so ${field1} ${field2} where field1 and field2 are part of the record. If you are trying to access attributes then I think you need to use UpdateRecord first. On Tue, Sep 3, 2019 at 6:06 AM DEHAY Aurelien

Re: Oversized queue between process groups

2019-08-30 Thread Bryan Bende
Can you show what is happening inside the first process group? Is there a SplitText processor with line count of 1? On Fri, Aug 30, 2019 at 4:21 AM Jeremy Pemberton-Pigott < fuzzych...@gmail.com> wrote: > Hi Pierre, > > I'm using Nifi version 1.6.0. > > 04/03/2018 08:16:22 UTC > > Tagged

Re: Process group name in S2SStatusReporterTask

2019-08-29 Thread Bryan Bende
I think the applicationName may be left over from the provenance s2s reporting task since I think provenance events had a field called applicationName ? But I think each status event should still have the process group name in the componentName field:

Re: Nifi Role Based Access Controls

2019-08-27 Thread Bryan Bende
Yes, NiFi provides resource based access control. Every component such as a processor, controller service, process group, etc, is a resource which can have policies created for it to control view/modify privileges. On Tue, Aug 27, 2019 at 2:52 PM Christopher J. Amatulli wrote: > > Within Nifi,

Re: authenticated nifi agent wih unauthenticated registry

2019-08-26 Thread Bryan Bende
There is a video on the NiFi Registry page of the site that shows how to setup NiFi talking to secure NiFi Registry... https://nifi.apache.org/registry.html https://www.youtube.com/watch?v=DSO12fhnZ90=youtu.be On Mon, Aug 26, 2019 at 4:26 AM Nicolas Delsaux wrote: > > Hi all > > I have

Re: AttributeRollingWindow 1.9.2 Value to Track UUID possible ?

2019-08-23 Thread Bryan Bende
I think AttributeRollingWindow was made to aggregate a numeric attribute. So for example, if all flows files had an attribute called "amount" that was an integer, then you could say Value to Track = ${amount} and it would aggregate those values over window. In your case the attribute you have is

Re: Installing nifi-registry as service

2019-08-23 Thread Bryan Bende
mmand, it shows service nifi-registry installed, but > actually I cannot find nifi-registry under /etc/init.d. The OS I'm using is > Ubuntu 18.04-LTS. > > Thanks, > Qiong > > -Original Message- > From: Bryan Bende > Sent: Friday, August 23, 2019 9:57 AM >

Re: Installing nifi-registry as service

2019-08-23 Thread Bryan Bende
Hello, Does it look like the service is correctly installed in /etc/init.d/nifi-registry ? Also, what operating system? -Bryan On Fri, Aug 23, 2019 at 12:26 PM Qiong Wei wrote: > > Hi All, > > > > I’m new to NiFi-Registry, now trying to install NiFi-Registry as a service, > here are the

Re: Unable to connect to Confluent Schema Registry with credentials

2019-08-20 Thread Bryan Bende
I believe it would require a code change to the ConfluentSchemaRegistry service to have properties for basic auth username and password, and then pass them down to the client calls. Feel free to create a JIRA to capture this improvement. On Mon, Aug 19, 2019 at 8:17 PM Ali Nazemian wrote: > > I

Re: Custom Processor Upgrade

2019-08-15 Thread Bryan Bende
processor while migrating it as kylo processor was not working as is in our > environment.. Will check that and have it packaged in the processors module. > > On Thu, Aug 15, 2019 at 1:50 AM Bryan Bende wrote: >> >> Where is application-context.xml in your NAR?

Re: Custom Processor Upgrade

2019-08-14 Thread Bryan Bende
nitions(XmlBeanDefinitionReader.java:508) > at > org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:392) > ... 66 common frames omitted > > On Wed, Aug 14, 2019 at 4:44 PM Bryan Bende wrote: > >> You have to a

Re: Custom Processor Upgrade

2019-08-14 Thread Bryan Bende
teScan, but this is not a > valid Processor type] > 2019-08-14 20:16:18,946 DEBUG [Timer-Driven Process Thread-6] > o.a.n.c.r.m.SecondPrecisionEventContainer Updated bin 39. Did NOT replace. > > On Wed, Aug 14, 2019 at 1:42 PM Bryan Bende wrote: >> >>

Re: Custom Processor Upgrade

2019-08-14 Thread Bryan Bende
ent.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108) > at > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) > at org.eclipse.jetty.server.Server.doStart(Server.java:386) > at > org.eclipse.jetty.util.component.AbstractLifeCyc

Re: Custom Processor Upgrade

2019-08-14 Thread Bryan Bende
nd gives the error message I stated above) and properties are > missing as well. > > > On Tue, Aug 13, 2019 at 10:47 PM Bryan Bende wrote: > >> I don’t remember all the reasoning behind the change, but it had to do >> with an issue when we upgraded Jetty... >>

Re: Custom Processor Upgrade

2019-08-13 Thread Bryan Bende
inside META_INF when it was built. > However I cant see that file anymore in the unpacked nar. > > On Tue, Aug 13, 2019 at 8:57 PM Bryan Bende wrote: > >> Does that custom processor type show as an option if you try to add a new >> processor to the canvas? >> >> On T

Re: Custom Processor Upgrade

2019-08-13 Thread Bryan Bende
sure if CDF supports 1.9.2 yet or not. I will reach out to Cloudera >> and see if we can get it upgraded. >> >> >> >> On Thu, Aug 8, 2019, 8:51 AM Bryan Bende wrote: >> >>> What is in the template for the bundle coordinates of your processor? >>>

Re: unable to post updates api with user certificate.

2019-08-13 Thread Bryan Bende
r.java:195) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > On Mon, Aug 5, 2019 at 10:38 PM Bryan Bende wrote: > >> Hello, >> >> Issue #1 - You should be able to specify an LDAP user as your initial >> admin, what is t

Re: Abstract Processor > Static final properties

2019-08-11 Thread Bryan Bende
Hello, The final static variables are usually the descriptors which are just the definition of the properties. The actual values of the properties are storied in a separate map per instance of the processor. -Bryan On Mon, Aug 12, 2019 at 12:03 AM Craig Knell wrote: > Hi Folks, > > I would

Re: Custom Processor Upgrade

2019-08-08 Thread Bryan Bende
loudera >> distributed version of NiFi 1.9. >> Any idea why is this happening? >> >> Thanks >> >> >> >> On Wed, Aug 7, 2019 at 3:46 PM Bryan Bende wrote: >>> >>> Hello, >>> >>> Most likely your processor built

Re: Custom Processor Upgrade

2019-08-07 Thread Bryan Bende
Hello, Most likely your processor built against 1.6 would run fine in 1.9, but to make sure you just need to update any nifi dependencies in your poms to 1.9.2. If you created your project from the archetype and didn't change anything, then this should just be changing the parent in the root pom

Re: Communication Error Between NiFi and Registry: Error retrieving all buckets

2019-08-07 Thread Bryan Bende
tes. I am getting the same error. Do I need > to be restarting NiFi or using a clean browser session each time I make an > update? My intuition says no since the authentication should be happening > server side but wondering if I am missing something. > > On Wed, Aug 7, 2019 at

Re: Communication Error Between NiFi and Registry: Error retrieving all buckets

2019-08-07 Thread Bryan Bende
the organizational unit, OU in the example above, >>> is NIFI. I have created a number of other users with slight variations on >>> the CN and OU values but any attempt to connect the two services fails. >>> Still getting the error, "Unable to obtain listing of buc

Re: Optimizing Performance of Apache NiFi's Network Listening Processors

2019-08-06 Thread Bryan Bende
ta data and push the records to > various consumers. > > thanks > Clay > > On Tue, Aug 6, 2019 at 6:26 AM Bryan Bende wrote: >> >> Can you describe what you want to do with each message? >> >> Right now I’m not following why you need to parse them. &g

Re: Optimizing Performance of Apache NiFi's Network Listening Processors

2019-08-06 Thread Bryan Bende
of syslog messages, if you don't have the > batching option? I suppose we could have had the option of parsing each > syslog record in a batch and then writing the syslog message along with the > syslog headers to the flowfile content. > thanks > Clay > > On Mon, Aug 5, 2019

Re: Communication Error Between NiFi and Registry: Error retrieving all buckets

2019-08-05 Thread Bryan Bende
cess. When I set the NiFi > Identity 1 for a Kerberos scheme should it follow the u...@domain.com format? > If it does, would the NiFi Identity 1 for localhost be nifi@LOCALHOST? > > On Mon, Aug 5, 2019 at 1:47 PM Bryan Bende wrote: >> >> Hello, >> >> I believe

Re: Communication Error Between NiFi and Registry: Error retrieving all buckets

2019-08-05 Thread Bryan Bende
Hello, I believe the video should cover this, but did you add a user representing your NiFi instance and grant it the permissions for proxy and read all buckets? That is what "NiFi Identity 1" would have done, but that only gets used on initial setup, so you would do it from the UI now. -Bryan

Re: Optimizing Performance of Apache NiFi's Network Listening Processors

2019-08-05 Thread Bryan Bende
the selector to run on the pool. So in short >> unless a single Nifi server gets 4,000 syslog messages in a very short space >> time (< 1 micro-second) I can't see it being an issue. >> >> Edward >> >> On Fri, Aug 2, 2019 at 2:06 PM Bryan Bende wrote: >>

Re: unable to post updates api with user certificate.

2019-08-05 Thread Bryan Bende
Hello, Issue #1 - You should be able to specify an LDAP user as your initial admin, what is the error you get? Keep in mind it is case and white-space sensitive, and also depends on whether you are returning full DN or short name, it must match exactly. Issue #2 - Since you are able to query

Re: CaptureChangeMySQL processor takes more than 100% cpu usage after cathing up the lastet binlog position

2019-08-03 Thread Bryan Bende
I only looked at this quickly, but I think the queue.poll in the onTrigger method should use a small blocking time of about 10ms so that when no events are available it won’t crush the CPU. Right now you could change the scheduling of the processor from 0 seconds to 10ms but then that slows it

Re: Setting zookeeper.sasl.client=false

2019-08-02 Thread Bryan Bende
Chad, I was looking into something related to this recently and I think your description is accurate. Unfortunately ZooKeeper client relies heavily on system properties which isn't great for talking to a bunch of different systems like NiFi. One thing I would be curious about, what value do you

Re: Optimizing Performance of Apache NiFi's Network Listening Processors

2019-08-02 Thread Bryan Bende
The actual connections themselves are managed with a selector, so if all the connections are idle there should only be one thread for the socket. As soon as a connection has something available to read then a thread is spawned to start reading the connection until either no matter is available,

Re: Default receive data via site-to-site policy

2019-07-29 Thread Bryan Bende
, Jul 29, 2019 at 2:45 AM Vos, Walter wrote: > > Hi Bryan, > > Thanks, that's good to hear. Any idea about when 1.10 will be out? 1 month, 3 > months, 6 months? > > -Walter > > -Oorspronkelijk bericht- > Van: Bryan Bende [mailto:bbe...@gmail.com] > Verz

Re: Default receive data via site-to-site policy

2019-07-26 Thread Bryan Bende
Hello, I know this doesn't solve your problem right now, but in an upcoming release there will be the ability to create remote ports anywhere in the flow [1], so your devs won't need any access to the root canvas. -Bryan [1] https://issues.apache.org/jira/browse/NIFI-2933 On Fri, Jul 26, 2019

Re: Installing and running Nifi first time on Mac

2019-07-25 Thread Bryan Bende
I thought jEnv set JAVA_HOME automatically... did you run "source ~/.bash_profile" to load the changes you made? Technically that should be fine, it doesn't really matter if JAVA_HOME is set, as long as "java -version" shows Java 8 from the terminal where you launch NiFi, which your output does

Re: Installing and running Nifi first time on Mac

2019-07-25 Thread Bryan Bende
We should definitely update that statement in the System Requirements to say "Requires Java 8" and remove the "or newer part", at least until the Java 11 work is complete. I'm fairly certain what happened was that statement was written when we moved from pre-Java 8 to requiring Java 8, and at the

Re: Installing and running Nifi first time on Mac

2019-07-25 Thread Bryan Bende
The output of your "java -version" shows you have openjdk version "12.0.2" installed, which is Java 12, and currently NiFi only supports Java 8. So you'll likely want to install Java 8 and set JAVA_HOME to point to that. A useful tool for managing multiple Java version is jEnv [1]. [1]

Re: Continuing my LDAP auth adventures

2019-07-19 Thread Bryan Bende
The FileAccessPolicyProvider is making a call to the user group provider using the value you entered for initial admin: final User initialAdmin = userGroupProvider.getUserByIdentity(initialAdminIdentity); It has something to do with the value you entered for the initial admin not lining up with

Re: Run duration warning?

2019-07-11 Thread Bryan Bende
Hi Joe, The message is shown based on this logic: if (ui.value > 0 && (processor.inputRequirement === 'INPUT_FORBIDDEN' || processor.inputRequirement === 'INPUT_ALLOWED')) The reasoning is because run duration > 0 means calls to the session are being batched, so a source processor may read data

Re: Kafka to parquet to s3

2019-07-06 Thread Bryan Bende
Currently put and fetch parquet are tied to the Hadoop API so they need the config files. As mentioned you can create a core-site with a local file system, and then you could use another part of the flow to pick up the file using ListFile -> FetchFile -> PutS3Object. There is a way to write to S3

Re: NiFi Wait / Notify not releasing on signal

2019-06-17 Thread Bryan Bende
Not sure I follow, would probably have to see another template that shows what is not working. On Mon, Jun 17, 2019 at 3:52 PM ara m. wrote: > > Hey Bryan - Thank you ! you're absolutely right. Putting in FIFO totally > makes it work. > > My original issue was actually with prioritizer set to >

Re: NiFi Wait / Notify not releasing on signal

2019-06-17 Thread Bryan Bende
Hello, I think the issue is that wait relationship looped back to itself has no prioritizer configured on the queue. The flow file with ${signal} = aa is in the queue first, and each time it checks it and transfers back to wait, it happens to be put back in front of the one with bb. If you

Re: ElasticSearchClientServiceImpl

2019-06-13 Thread Bryan Bende
Unfortunately this looks like a bug in the code: (sslService != null && sslService.isKeyStoreConfigured() || sslService.isTrustStoreConfigured()) Should be: (sslService != null && (sslService.isKeyStoreConfigured() || sslService.isTrustStoreConfigured())) Currently it will hit the last

Re: Connect NIFI with Impala by DBCPConnectionPool

2019-06-07 Thread Bryan Bende
NiFi has log4j-over-slf4j-1.7.26.jar in the root lib directory, so it is on the classpath of all components. I'm guessing the Impala JDBC driver is a shaded JAR that includes slf4j-log4j12.jar? Not really sure of a solution, just theorizing on what the problem is. On Fri, Jun 7, 2019 at 10:45

Re: ListenUDP: internal queue at maximum capacity, could not queue event

2019-06-05 Thread Bryan Bende
;> >> Op wo 5 jun. 2019 18:07 schreef James Srinivasan >> : >>> >>> In our case the stream is UDP broadcast, so available to all nodes anyway. >>> I've been meaning to test UDP multicast but not got round to it yet. >>> >>> >>> On

Re: ListenUDP: internal queue at maximum capacity, could not queue event

2019-06-05 Thread Bryan Bende
That is probably a valid point, but how about putting a load balancer in front to handle that? On Wed, Jun 5, 2019 at 11:30 AM James Srinivasan wrote: > > Presumably you'd want to mirror the stream to all nodes for when the primary > node changes? > > On Wed, 5 Jun 2019, 13

Re: ListenUDP: internal queue at maximum capacity, could not queue event

2019-06-05 Thread Bryan Bende
there must be something in the >> configuration of the cluster thats causing this. I will also try a vanilla >> Nifi install on one of the nodes without clustering to see if my >> configuration and cluster setup is somehow the cause. >> >> Op di 4 jun. 2019 om 16:14

Re: Load balancer queues stuck on 1.9.2?

2019-06-04 Thread Bryan Bende
Joe, There are two known issues that possibly seem related... The first was already addressed in 1.9.0, but the reason I mention it is because it was specific to a connection between two ports: https://issues.apache.org/jira/browse/NIFI-5919 The second is not in a release yet, but is addressed

Re: ListenUDP: internal queue at maximum capacity, could not queue event

2019-06-04 Thread Bryan Bende
Hi Erik, It sounds like you have tried most of the common tuning options that can be done. I would have expected batching + increasing concurrent tasks from 1 to 3-5 to be the biggest improvement. Have you increased the number of threads in your overall thread pool according to your hardware?

Re: InvokeHTTP with SSL

2019-06-03 Thread Bryan Bende
Hello, You should be specifying an SSL Context Service in the processor which points to a truststore that trusts the certificate of the service you are calling. Alternatively, if the CA certs system truststore trusts the service cert then it should also work. Thanks, Bryan On Mon, Jun 3, 2019

Re: ScanHBase processor time range.

2019-05-22 Thread Bryan Bende
Hello, It looks like it would require a code change to the processor to make it work as your described. -Bryan On Wed, May 22, 2019 at 5:34 AM Dnyaneshwar Pawar < dnyaneshwar_pa...@persistent.com> wrote: > Hi All, > > > > We are facing difficulty in ScanHBase processor where min and max time >

Re: flowfile through invokehttp put

2019-05-16 Thread Bryan Bende
of UnpackContent. On Thu, May 16, 2019 at 11:26 AM Michael Di Domenico wrote: > > On Thu, May 16, 2019 at 11:20 AM Bryan Bende wrote: > > > > Well MergeContent in general is meant to take many flow files and > > merge them together, so typically if you were using the flow file &g

Re: flowfile through invokehttp put

2019-05-16 Thread Bryan Bende
Well MergeContent in general is meant to take many flow files and merge them together, so typically if you were using the flow file format, the idea would be to create a single flow file where the content contained (flow1 attrs, content)(flow 2 attrs, content) etc, but what I was suggesting was to

Re: flowfile through invokehttp put

2019-05-16 Thread Bryan Bende
Normally this would all be handled for you with site-to-site, but not sure that can be made to work with your custom proxy. Before site-to-site there was PostHttp and ListenHttp, and PostHttp has an options for "Send As Flow File". PostHTTP is marked as deprecated because generally you should use

Re: What will happen if we lost a node in cluster?

2019-05-15 Thread Bryan Bende
Hello, Each node of the cluster is running a copy of the dataflow that you see in the UI, and the data that is being operated on is what is divided across the nodes (depending on how the data enters the flow). If a node goes down, the other nodes continue processing, and the data that is on the

Re: Getting the value of a variable defined in a processor group

2019-05-14 Thread Bryan Bende
to provide an > accessor method on the context to return a variable map, or at least the > entry set. > > I’d like to hear from Matt Gilman, Bryan Bende, and/or Mark Payne with their > thoughts, since much of this code was written by them. > > [1] > https://github.com/apach

Re: Penalization on retry, on failure and session commit clarificatoion

2019-05-07 Thread Bryan Bende
Hi Michal, Regarding the session.commit... most processors extend from AbstractProcessor which takes care of the commit [1]. If you extend from AbstractSessionFactoryProcessor, or directly implement the Processor interface with no parent class, then it is up to you to call commit. For the

Re: Recovering From Bad Flow Configs

2019-05-02 Thread Bryan Bende
You can set nifi.flowcontroller.autoResumeState=false to start NiFi without running everything. On Thu, May 2, 2019 at 12:23 PM Ryan H wrote: > > Hi All, > > We spin up multiple instances of NiFi for multiple users in a containerized > environment. A common issue that we run into is users

Re: Design question - code smell?

2019-04-24 Thread Bryan Bende
I think having a lot of error paths converge to a standard error handling part of the flow is a common pattern. It might help to converge different parts using funnels along the way so that it doesn't end up with a hundred connections into the one processor at the end, but its really just a

Re: Implementing Gates with the Wait and Notify Processors

2019-04-24 Thread Bryan Bende
I think the Wait/Notify processors may have changed a bit since I last used them, but the original use case was for when a processor produces two flow files, and you want to send one of them down one part of the flow, and have the other wait until that part is done. The easiest example to think of

Re: Merge identical JSON records to single JSON with subarray

2019-04-12 Thread Bryan Bende
Hello, I think you will likely need to use a JOLT transform to perform this operation. I don't know JOLT well enough to suggest the correct operation, so maybe others can help there. ConvertRecord is more about converting between formats like using a JSON reader and a CSV writer where the schema

Re: NIFI- 5340

2019-04-12 Thread Bryan Bende
to 5min then the update > completes successfully. > > On Thu, Apr 11, 2019 at 7:25 PM Bryan Bende wrote: >> >> The underlying issue is that there are components in your versioned >> flow that can't be stopped for some reason. >> >> Do you have any custom processors? I

Re: Registry and db connection services

2019-04-11 Thread Bryan Bende
ocess groups and only > version the parent? > > Sorry for the basic questions here, I'm still trying to figure out a good > pattern how to work with NiFi and deploy the result. :-) > > On Wed, 10 Apr 2019 at 21:47, Bryan Bende wrote: >> >> Hello, >> >> When

Re: NIFI- 5340

2019-04-11 Thread Bryan Bende
The underlying issue is that there are components in your versioned flow that can't be stopped for some reason. Do you have any custom processors? If you try to stop everything manually do all processors stop cleanly? On Thu, Apr 11, 2019 at 7:22 AM saloni udani wrote: > > Ref:

Re: Registry and db connection services

2019-04-10 Thread Bryan Bende
Hello, When you version control a process group, it captures all components in that group and child groups. If a component references a controller service from a parent group, then when you import this process group to the next environment you will have to re-link that component to the

Re: HiveQL to load data in hive after generating file on HDFS

2019-04-09 Thread Bryan Bende
I meant to say that in ReplaceText you would need use "Always Replace" with mode of "Entire Text". On Tue, Apr 9, 2019 at 10:38 AM Matt Burgess wrote: > Since you know you just want to overwrite the contents with HiveQL, you > could use ExecuteScript with the following Groovy script: > > def

Re: HiveQL to load data in hive after generating file on HDFS

2019-04-09 Thread Bryan Bende
ReplaceText after step 3 should be fine, it won't need to load the entire data into memory, it just writes new flow file content and updates the flow file to point to the new content claim, leaving the old one behind to be garbage collected. On Tue, Apr 9, 2019 at 9:49 AM DEHAY Aurelien wrote:

Re: threads not terminating correctly

2019-04-09 Thread Bryan Bende
In a lot of cases the stuck thread is waiting on an I/O operation, typically reading/writing over a network connection. The only way to really figure out what is happening is to take a series of thread dumps when this happens: ./bin/nifi.sh dump Run that a couple of times in a row while a

Re: Best way to wait until everything is finished

2019-04-05 Thread Bryan Bende
Since you are using SplitJson, it should be adding the standard "fragment" attributes to each flow file. You can then use MergeContent in Defragment mode, which uses those attributes to merge all the fragments back together. On Fri, Apr 5, 2019 at 2:39 PM Jean-Sebastien Vachon wrote: > > Hi

Re: PutKafka use with large quantity of data?

2019-04-04 Thread Bryan Bende
Each queue has back-pressure settings on it which default to 10k flow files or 1GB size of flow files. When one of these thresholds is exceeded, the preceeding processor will not execute until the queue goes back below the threshold. Most likely if GenerateFlowFile has a Run Schedule of 0

Re: PutKafka use with large quantity of data?

2019-04-04 Thread Bryan Bende
We need to define what "NiFi stops responding" means... Are there tons of flow files queued up before before PublishKafka? Are there back-pressure indicators on any of the queues? Do the kafka related processors show active threads in the top right corners of the processors? Does NiFI crash? On

Re: Reusing same flow for different database connections

2019-04-04 Thread Bryan Bende
Hello, Take a look at the DBCP lookup service, it allows you to register one or more connection pool services and then select one at runtime based on an incoming flow file having an attribute called database.name. Thanks, Bryan On Thu, Apr 4, 2019 at 8:47 AM Max wrote: > Hello! > > We are

Re: set penalty duration on a flowfile

2019-04-03 Thread Bryan Bende
Hello, Even if there is a way to make this work, I would not recommend doing this. Processors should not have access to internals of NiFI's framework (i.e. nifi-framework-nar), they should only have access to what is provided through the nifi-api which is the session and context. If you start

Re: NiFi 1.9.1 release contains a bug causing content repos to fill...

2019-04-02 Thread Bryan Bende
The associated JIRA is NIFI-6150 [1] and it leads to the content repo not properly cleaning up and eventually filling up disks. There will be a 1.9.2 release to address this. Of course you are free to take the patch for the fix [2], apply it to the 1.9.1 release tag [3], and build your own fix,

Re: Load balancing strategy/thoughts

2019-04-01 Thread Bryan Bende
Hello, Is there a reason why it has to be brought back to the same original node? As long as MergeContent is scheduled on all nodes, then you can choose "Single node" strategy for the queue leading into MergeContent, and one of the nodes will get all the pieces and can do the merge/defragment.

Re: Empty "nifi users" page.

2019-03-26 Thread Bryan Bende
What browser and browser version are you using? Have you tried clearing your browser cache just to make sure the page is loading properly? On Tue, Mar 26, 2019 at 10:21 AM DEHAY Aurelien wrote: > > Hello. > > I expect to see at least the local users with associated rights (e.g. content > of

Re: Apache NiFi registry and Apache Taverna?

2019-03-26 Thread Bryan Bende
Hello, NiFi Registry is very specific to NiFi so I'm not sure that another project would integrate with it unless they were using the same data model. Most likely the data model for a workflow in Taverna would not be the same as a NiFi versioned flow. -Bryan On Tue, Mar 26, 2019 at 6:58 AM

Re: No load on second node

2019-03-22 Thread Bryan Bende
Hello, The host and port used for load balancing are defined in nifi.properties of each node with the following properties: # cluster load balancing properties # nifi.cluster.load.balance.host= nifi.cluster.load.balance.port=6342 nifi.cluster.load.balance.connections.per.node=4

Re: sensitive variable values ?

2019-03-21 Thread Bryan Bende
Sensitive variables is something we'd like to support, but we currently don't have that capability yet. If you are using a script to create components, or to instantiate templates, you should be able to set the property values of those components using the REST API. You would be making the same

Re: Automate the Enablement of a StandardHttpContextMap

2019-03-21 Thread Bryan Bende
Also meant to say that it can only be enabled if it is valid, meaning there are no validation messages on the component. So it might be worth checking in the UI to see if there are any validation issues on the service. On Thu, Mar 21, 2019 at 3:02 PM Bryan Bende wrote: > > If y

Re: Automate the Enablement of a StandardHttpContextMap

2019-03-21 Thread Bryan Bende
When a template, or a versioned flow, is instantiated, all the processors are initially stopped and the services are disabled. This is done on purpose since starting and enabling components may cause interactions with external systems, and it may not be clear or desired that just by instantiating

  1   2   3   4   5   6   7   >