Re: Merge content Defrag with high activity

2019-10-22 Thread Joe Witt
Hello You should only have 1 or a few tasks at most for this processor. Scheduling can be frequent but choosing different options and seeing for your case is best. This processor is relatively difficult to configure correctly as it is a complex case and has powerful options. What you will need

Re: Clarification Regarding custom controller service & AWS-CREDENTIAL-CONTROLER-SERVICE

2019-10-20 Thread Joe Witt
Sanjeet Your e-mail was sent 10 hours ago and is during the weekend. Please be patient. I would do two nars. The controller service nar and the processor nar which depends on it. This way you can have other processor nars that also depend on that controller service nar if necessary. Thanks

Re: ElasticSearchClientServiceImpl not working for secured ElasticSearch

2019-10-18 Thread Joe Witt
is a daily effort at this point. i am close to pushing first rc. have been watching for stability on bug fixes. On Fri, Oct 18, 2019 at 1:10 PM Juan Pablo Gardella < gardellajuanpa...@gmail.com> wrote: > Any ETA for Nifi 1.10 release? > > On Fri, 18 Oct 2019 at 13:39, Mike Thomsen wrote: > >>

Re: Can balance by attribute guarantee the order of the FlowFile?

2019-10-16 Thread Joe Witt
Lei The order won't necessarily be the same. You'd want EnforceOrder to follow the load balanced connection most likely. It is important to keep in mind the queues are basically insertion order and the system is inherently multi-threaded so the data can be shuffled in such cases. You can

Re: Re: NiFi backpressure not work

2019-10-16 Thread Joe Witt
Lei Please do not post to both users and dev list. Dropping dev. Yes these scenarios are based on limitations of a given processor implementation or the nature of a given protocol/mechanism. NiFi enforces back pressure by slowing/stopping scheduling a component and making fact of back pressure

Re: NAR has test level dependencies

2019-10-14 Thread Joe Witt
Chandra Something is making them compile scope likely. Use maven tools like help:effective-pom to see what the actual pom is when building the nar. You can then zero in by doing the same on the processor bundle. thanks On Mon, Oct 14, 2019 at 7:51 AM Chandrashekhar Kotekar <

Re: access token (secured NiFi) in InvokeHTTP, PostHTTP

2019-10-10 Thread Joe Witt
Tomas It just need someone to implement various standards. Right now I believe it is purely TLS one way or mutual auth and also supports basic and digest. Thanks On Thu, Oct 10, 2019 at 11:00 AM Tomas Hudik wrote: > Hi Erik > thank you very much for the mail. > > My try is not about coding

Re: Apache Nifi with IBM Event Streams

2019-10-10 Thread Joe Witt
You can use SASL/Plain today with the kafka 2 procs in NiFi but as noted it is unpleasant to configure. A much easier/clear configuration is being worked on right now. Not positive what the JIRA for it is though. Should be available quite soon. Thanks On Thu, Oct 10, 2019 at 9:31 AM

Re: Weird behaviour

2019-10-02 Thread Joe Witt
Jean Id recommend switching to the new provenance repo called WriteAheadProvenaceRepository. Look at a new nifi downloads nifi.properties as it has been the default for a while. This will help the prov stuff. You may also want to stop using g1gc if on java 8. I cant explain the status history

Re: Nifi: Replicate or Put file on all nodes of cluster

2019-09-20 Thread Joe Witt
Yeah to double down on Bryan's comments I know this is frequently done. For instance we often would pull data from a website that we'd use for lookups/enrichments in the flow. We'd use GetHTTP or related processors to grab the contents of the given URL constantly and honor things like

Re: V1.10 Release Date

2019-09-19 Thread Joe Witt
Craig I plan to RM the release and am awaiting a set of jiras tagged as 1.10 and with a lot of review traction to merge. I also think it makes sense to attempt to scan and find lingering prs as there is a lot of good work there that needs review attention. But I am hopeful 1.10 rc processes

Re: NiFi active thread count is no more than 10 ?

2019-09-18 Thread Joe Witt
Hello The 100 threads for the controller overall is the maximum number of threads that could run concurrently. On a 16 core system and a flow which is very I/O bound this is definitely achievable. Generally you want to look at some multiple of the number of physical cores such as 2,4,8, etc..

Re: Stateful Dataflow Moved to New Cluster

2019-09-17 Thread Joe Witt
quick reply: There is a zookeeper state migrator utility in the toolkit I believe. That should be quite helpful. http://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#zookeeper_migrator Thanks On Tue, Sep 17, 2019 at 11:35 AM Noe Detore wrote: > Hello, > > I am currently using a

Re: Mailing List Question

2019-09-04 Thread Joe Witt
Just to follow up Adam sent me some more detail offline. No sign of the emails so recommend subscribing and trying again. Thanks On Wed, Sep 4, 2019 at 8:06 PM Joe Witt wrote: > adam > > can you confirm which list and ask them to forward. A couple of us > moderate but nothing

Re: Mailing List Question

2019-09-04 Thread Joe Witt
adam can you confirm which list and ask them to forward. A couple of us moderate but nothing has lingered that i know of. if you know time and subject that helps too. thanks On Wed, Sep 4, 2019 at 6:56 PM Adam Taft wrote: > I am trying to help a colleague get a message through to the user

Re: "Deadlock" data provenance after few days

2019-08-19 Thread Joe Witt
< giulia.scalabe...@genomedics.it> wrote: > Hi all, > > As requested by Joe Witt, here I attach the dump log after the lock. > > My Nifi after nearly 30 days of scheduled activity has locked as usual… > > > > BG, > > Giulia > > > > *Da:* Joe Witt [mailt

Re: [EXT] Re: FlowFile Repository can't checkpoint, out of heap space.

2019-08-15 Thread Joe Witt
Peter All the details you can share on this would be good. First, we should be resilient to any sort of repo corruption in the event of heap issues. While obviously the flow isn't in a good state at that point the saved state should be reliable/recoverable. Second, how the repo/journals got

Re: Nifi Registry best practices

2019-08-15 Thread Joe Witt
Muazma, It is strongly recommended to have a single shared registry across the environments if your policies allow. This will give the best (by design) experience of porting flows from one environment to another. The remaining challenges you see with this is that you will have to enter things

Re: Anti-Virus Scanning

2019-08-13 Thread Joe Witt
Jason The work dir gets created at startup and possible as new nars are loaded. I think you'd be ok to scan this. The flowfile and content repository and provenance directories as configured should be skipped. The logs dir should be skipped. The state directory should be skipped. All else I

Re: "Deadlock" data provenance after few days

2019-08-01 Thread Joe Witt
Giulia, When you're experiencing this condition can you capture a thread dump and share the logs (bootstrap/app)? To create this you can run /bin/nifi.sh dump threaddump-locked-prov.log Thanks On Thu, Aug 1, 2019 at 3:57 AM Giulia Scalaberni < giulia.scalabe...@genomedics.it> wrote: > Hi, > >

Re: Certificates in Truststore

2019-07-25 Thread Joe Witt
at 11:58 AM Joe Witt wrote: > Joseph > > You are absolutely right that it would be terrible to have to edit the > truststore on the nifi server(s) each time you wanted to add a client > cert. You're also right that there is a way to never do this. I'll poke > around for some link

Re: Certificates in Truststore

2019-07-25 Thread Joe Witt
Joseph You are absolutely right that it would be terrible to have to edit the truststore on the nifi server(s) each time you wanted to add a client cert. You're also right that there is a way to never do this. I'll poke around for some links to help send you in the right direction. Thanks On

Re: QueryRecord processor where clause does not work with equals operator and decimal numbers.

2019-07-17 Thread Joe Witt
thanks for reporting. please file a jira showing your steps and example to reproduce. thanks On Wed, Jul 17, 2019 at 8:43 AM Dnyaneshwar Pawar < dnyaneshwar_pa...@persistent.com> wrote: > Hi, > > We are using QueryRecord processor to read and parse the the CSV files > using CSV Reader as

Re: Site to Site Compression

2019-07-15 Thread Joe Witt
Noe Just activate compression on the s2s port and the client will honor it if able. I dont believe the protocol has changed in quite a while so you should be fine with the versions noted. Thanks Joe On Mon, Jul 15, 2019 at 9:08 AM Noe Detore wrote: > Hello, > > What is the best way to

Re: 1.9.2 Does not show provenance events

2019-07-11 Thread Joe Witt
Hello I suspect you have to add the new policy. Please see in the migration guide from old version you had until now. Thanks On Thu, Jul 11, 2019, 2:56 PM Mikhail Rolshud (BLOOMBERG/ 120 PARK) < mrols...@bloomberg.net> wrote: > Hi, > > We noticed that after some time 1.9.2 stops showing

Re: Remote Process Group fails to distribute any flowfiles to primary node

2019-07-10 Thread Joe Witt
planation correctly: behavior exhibited through the first 4000 flowfiles > as past performance may not represent future results. It will do what it > does, and I may find that node1 does get loaded as I work through flowfiles > in steady state. > Again, thanks. > > On Wed, Jul 1

Re: Remote Process Group fails to distribute any flowfiles to primary node

2019-07-10 Thread Joe Witt
James For distributing work across the cluster the load balanced connection capability in NiFi 1.8 and beyond is the right answer - purpose built for the job. I'd strongly recommend upgrading to avoid use of s2s for this scenario and instead use load balanced connections. When using load

Re: Content repository data filling up disk...

2019-06-19 Thread Joe Witt
Russell If data remains in the content repository beyond the specified archive values then it suggests there is content remaining in the flow that is not yet eligible to be removed/deleted. This is not always a direct "500 MB of content waiting for delivery results in 500 MB of content in the

Re: NiFi cluster goes 100% CPU in no time

2019-06-10 Thread Joe Witt
as ~60% > > Joe, I think it has something to do with what Wookcock suggested. Clearing > up content & FlowFiles seem to have CPU manageable. > Allow me 1-2 days and I shall report back if it solves the problem. > > On Mon, Jun 10, 2019 at 6:23 PM Joe Witt wrote: > >> h

Re: Keeping NiFi 1.9.2 console available

2019-06-10 Thread Joe Witt
...also how does the heap and garbage collection look during this? On Sun, Jun 9, 2019, 5:58 PM Joe Witt wrote: > Joe > > When you view top or other tools what is dominating the cpu? > > thanks > joe > > On Sun, Jun 9, 2019, 5:35 PM Joe Gresock wrote: > >>

Re: NiFi cluster goes 100% CPU in no time

2019-06-10 Thread Joe Witt
des and throttled at >1600%. > > > Meanwhile, I am trying to clear up all FlowFiles from disk and start the > flows afresh. > > > On Mon, Jun 10, 2019 at 5:42 PM Joe Witt wrote: > >> Sneh >> >> It was stable for months but now is high... >> >

Re: NiFi cluster goes 100% CPU in no time

2019-06-10 Thread Joe Witt
the pipeline is *just > too less* to throttle my CPU ideally. > > The machine config and NiFi config remains untouched - this has left me > confused where the problem might be. Something which had been running > smoothly since months, has become a challenge now. > > On Fri

Re: Keeping NiFi 1.9.2 console available

2019-06-09 Thread Joe Witt
Joe When you view top or other tools what is dominating the cpu? thanks joe On Sun, Jun 9, 2019, 5:35 PM Joe Gresock wrote: > I posted about this a while back on 1.6.0, but as far as I can tell it has > only gotten worse in 1.9.2. > > I have a cluster of 7 nifi nodes running on CentOS 6 VMs.

Re: NiFi cluster goes 100% CPU in no time

2019-06-07 Thread Joe Witt
You can also identify where top performance hitters are and ensure that a ControlRate or otherwise throttled amount of data and/or threads are leveraged at once. This allows you to effectively control how much effort to put on any single point of the flow at once. This is necessary when you want

Re: NiFi cluster goes 100% CPU in no time

2019-06-07 Thread Joe Witt
Shanker It sounds like you've gone through some changes in general and have worked through those. Now you have a flow running with a high volume of data (history load) and want to know which parts of the flow are most expensive/consuming the CPU. You should be able to look at the statistics

Re: ListenUDP: internal queue at maximum capacity, could not queue event

2019-06-05 Thread Joe Witt
...this feels like a bug to me. I think Erik-Jan's expectation that nothing would have begun for ListenUDP given primary node only config is fair. I also think our current position of 'just not calling onTrigger' is fair too but less intuitive for users. What do ya'll think? On Wed, Jun 5,

Re: Distribute Load processor not working as expected?

2019-05-23 Thread Joe Witt
Jon Just want to make sureare you sure each relationship is unique (1..7) and not copied? Could you share a screenshot perhaps? Thanks On Thu, May 23, 2019 at 11:48 AM Jon Belanger < jon.belan...@fidelissecurity.com> wrote: > I’ve got a Distribute Load processor with a single incoming

Re: Open / Close Gate examples?

2019-05-17 Thread Joe Witt
Dave Using Wait/Notify would ensure you only have one message in flight at a time (or it can/should). But the message will be ack'd before processed. For Kafka and some of these message queue mechanisms if we want to offer a 'do not ack until the whole flow is done' behavior we should update

Re: Apache Nifi issues

2019-05-09 Thread Joe Witt
ons in version 1.8 to resolve these? If not then I > will plan to upgrade soon. If there are any quicker solutions which can be > applied in 1.8 without any data loss then I ll upgrade it later. > > On Thursday, May 9, 2019, Joe Witt wrote: > >> Suman >> >> Yea

Re: Apache Nifi issues

2019-05-09 Thread Joe Witt
Suman Yeah it looks related to the queue/load balance fixes. Latest release should be much better for you. Thanks On Thu, May 9, 2019, 5:30 AM Suman B N wrote: > Team, > We are running a 3 node nifi cluster in docker. Version is 1.8. > Everything has been running smoothly from the last 2-3

Re: About Nar Classloader

2019-05-08 Thread Joe Witt
Jianan We have not done the work to fully isolate the concept of a NiFi Archive (Nar) such that it could be used outside of NiFi as a general classloader isolation pattern. There was one other person interested in helping make this happen in the past but I'm not sure where it has gone. With

Re: How to handle processors hanging due to Error

2019-04-26 Thread Joe Witt
Dave, Generally such a case where a processor combined with a flowfile can result in an error of some kind should have a failure relationship (or similarly named) and the flowfile should go there. However, some processors in certain cases will just rollback/fail and the data will sit in the

Re: Too many files open on CentOS 7

2019-04-12 Thread Joe Witt
Got to about 6500-6800 before hitting >>> the ceiling. >>> >>> On Fri, Apr 12, 2019 at 7:30 AM Joe Witt wrote: >>> >>>> mike >>>> >>>> lsof -p >>>> >>>> with the pid of the actual nifi process is p

Re: Too many files open on CentOS 7

2019-04-12 Thread Joe Witt
mike lsof -p with the pid of the actual nifi process is probably better to look at for nifi resource handling observation. what is that count. yes the jars and such will all be loaded. you can expect a few thousand off that. then there are sockets and content and prov and flowfilewhich

[ANNOUNCE] Apache NiFi 1.9.2 release.

2019-04-10 Thread Joe Witt
Hello The Apache NiFi team would like to announce the release of Apache NiFi 1.9.2. Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. Apache NiFi was made for dataflow. It supports highly configurable directed graphs of data routing, transformation,

[ANNOUNCE] Apache NiFi 1.9.2 release

2019-04-09 Thread Joe Witt
Hello The Apache NiFi team would like to announce the release of Apache NiFi 1.9.2. Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. Apache NiFi was made for dataflow. It supports highly configurable directed graphs of data routing, transformation,

Re: threads not terminating correctly

2019-04-09 Thread Joe Witt
Hello You could see hung threads like this because the processor is simply taking a long time to do its task (possible in certain listing cases but probably not ListS3), or because it is truly stuck such as a live-lock or timeout condition it has hit. These are almost always bugs and avoidable.

Re: insufficient content written

2019-04-05 Thread Joe Witt
Hello Can you share logs or screenshots or anymore details to illustrate what you're seeing? Thanks On Fri, Apr 5, 2019 at 2:14 PM Jean-Sebastien Vachon wrote: > Hi all, > > I've added an UpdateAttribute processor to change the filename attribute > and since then I am not able to either view

Re: PutKafka use with large quantity of data?

2019-04-04 Thread Joe Witt
; rate contol) > > > On Thu, Apr 4, 2019 at 10:42 AM Joe Witt wrote: > >> Hello >> >> There isn't really a feedback mechanism based on load on the Kafka >> topic. When you say overrunning the topic do you mean that you don't want >> there to be a larg

Re: PutKafka use with large quantity of data?

2019-04-04 Thread Joe Witt
Hello There isn't really a feedback mechanism based on load on the Kafka topic. When you say overrunning the topic do you mean that you don't want there to be a large lag between consumers and their current offset and if that grows you want NiFi to slow down? I dont believe there is anything

Re: NiFi 1.9.1 release contains a bug causing content repos to fill...

2019-04-03 Thread Joe Witt
S8%26s%3DVP_sGtxIOZIh_pm1wipeONZ6l1i3G5epzYUT_J_K2dk%26edata=02%7C01%7Cwilliam.gosse%40aifoundry.com%7C65f8067fa3cf41f6eb6e08d6b78b241d%7Cd29b7a9b6edb472099a83c5c6c3eeeb0%7C0%7C0%7C636898205580491866sdata=9B2wS%2FXeHsS9U02A72CKE4MWcr9mQXXY2LIGnFD8ME0%3Dreserved=0= > > > > > > On Tue, Apr 2,

Re: NiFi 1.9.1 release contains a bug causing content repos to fill...

2019-04-02 Thread Joe Witt
through this week. Thanks On Tue, Apr 2, 2019 at 12:17 PM William Gosse wrote: > Also can someone explain more what this issue is and how serious it is. > > Can a patch be made available to fix this issue for existing 1.9.1 > installations? > > > > *From:* Joe Witt >

Re: NiFi 1.9.1 release contains a bug causing content repos to fill...

2019-03-30 Thread Joe Witt
.2 RC? > > > > Cheers, > > Mathieu. > > > > *From:* Joe Witt [mailto:joe.w...@gmail.com] > *Sent:* Wednesday, March 27, 2019 11:00 AM > *To:* users@nifi.apache.org > *Subject:* NiFi 1.9.1 release contains a bug causing content repos to > fill... > > &

NiFi 1.9.1 release contains a bug causing content repos to fill...

2019-03-27 Thread Joe Witt
All, We will work to promptly produce an Apache NiFi 1.9.2 release which will correct a regression introduced in the 1.9.1 release. The issue, NIFI-6150 has been identified and fixed on master and once further testing confirms we're good we'll get the RC going. I've updated the release guide to

Re: Weird ListFile Issue

2019-03-22 Thread Joe Witt
William What was the real dir name vs the erroneous name? What might happens depends on many factors such as os behaviors. thanks On Fri, Mar 22, 2019, 12:47 PM William Gosse wrote: > I ran into kind of a weird issue with the ListFile processor. I was > referencing a variable for my input

Re: sensitive variable values ?

2019-03-21 Thread Joe Witt
Hello The variables of a pg are not, at this time, for sensitive values. You can set the sens values programatically to ensure they are never shown. We will likely add support for secrets (ie sensitive variables) but eta there depends on progress in the community. thanks On Thu, Mar 21, 2019,

Re: [ANNOUNCE] Apache NiFi 1.9.1 release.

2019-03-18 Thread Joe Witt
Docker build is fixed and out too. Thanks Aldrin On Mon, Mar 18, 2019 at 11:48 AM Joe Witt wrote: > Thanks Otto. > > To others: Please avoid replying all (which includes announce). we have > to reject that. > > But I hear you on the docker thing...I screwed up that part o

Re: [ANNOUNCE] Apache NiFi 1.9.1 release.

2019-03-18 Thread Joe Witt
: > Congratulations everyone! > > > On March 18, 2019 at 09:07:25, Joe Witt (joew...@apache.org) wrote: > > Hello > > The Apache NiFi team would like to announce the release of Apache NiFi > 1.9.1. > > Apache NiFi is an easy to use, powerful, and reliable system t

[ANNOUNCE] Apache NiFi 1.9.1 release.

2019-03-18 Thread Joe Witt
Hello The Apache NiFi team would like to announce the release of Apache NiFi 1.9.1. Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. Apache NiFi was made for dataflow. It supports highly configurable directed graphs of data routing, transformation,

Re: Using NiFi Expression Language outside of NiFi

2019-03-15 Thread Joe Witt
Tim, It is certainly feasible/interesting but it would require a lot of legwork to extract and make it into its own thing that then is used by NiFi and then other things. We could spawn it into its own subproject to facilitate perhaps.. Such ideas are often tough to pursue though as it might be

Re: Export/Import flow files

2019-03-15 Thread Joe Witt
Eric, You could download the flowfile content using the provenance click-to-content feature. You can see the attributes there too if you need to recreate those. You can then ingest that file using GetFile or something on the other system. There are additional built-in mechanisms that allow you

Re: NIFI ListenBeats Processor Issue

2019-03-15 Thread Joe Witt
Hello I am not very familiar with beats or this processor specifically but it does appear to have some beats specific framing that it does by looking at the processor code. Json based framing appears supported as well but it isn't clear that it would be or is intended to be valid JSON in a

Re: Registry 1.8.0 flows fail to import into 1.9.1 node

2019-03-14 Thread Joe Witt
glad you found how to get through it - we should make the user experience better. What led you to find the solution? On Thu, Mar 14, 2019 at 4:33 PM Mikhail Rolshud (BLOOMBERG/ 120 PARK) < mrols...@bloomberg.net> wrote: > Ignore this - it had nothing to do with registry or versions. > > Looks

Re: Most efficient means to search for a character in flowFiles

2019-03-13 Thread Joe Witt
James For the problem as you described it the processor you definitely want is https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.ScanContent/index.html It is a rather impressively fast implementation of a string search

Re: Processor(s) to monitor for new subdirectories?

2019-03-11 Thread Joe Witt
Denes You might want to ensure FetchFile is able to understand such flowfiles as well and to skip them so it doesn't try to Fetch a directory and/or it handles it gracefully. Thanks On Mon, Mar 11, 2019 at 12:51 PM Denes Arvay wrote: > Hi Jim, > > I suppose you want to monitor the newly

Re: Hive processors to make it compatible with Hive 2.1.1

2019-03-08 Thread Joe Witt
Ravi Correct. We've created integrations that we know work with Hive 1.1, Hive 1.2, and Hive 3.x. There has not been work toward Hive 2.x. The above correlate to CDH, HDP, and HDP respectively. You can see how there have been Hive bundles for different versions and could model off that. In

Re: Different NiFi Node sizes within same cluster

2019-03-06 Thread Joe Witt
This is generally workable (heterogenous node capabilities) in NiFi clustering. But you do want to leverage back-pressure and load balanced connections so that faster nodes will have an opportunity to take on the workload for slower nodes. Thanks On Wed, Mar 6, 2019 at 3:48 PM James Srinivasan

Re: Nifi 1.8.0 Very Slow NAR Unpacking

2019-03-06 Thread Joe Witt
John it is too hard to guess from just the shared images. A full stack trace and set of logs may be more enlightening. If unzipping/opening such a thing is so incredibly slow there are two culprits I've heard of/seen in the past. 1) Anti-virus/security related software that is overly

Re: When do I download the older releases of Apache NiFi

2019-02-27 Thread Joe Witt
Vijay, The downloads page is meant to link to the current and previous release and otherwise point to the archives. This is generally something the ASF requests because the current links are served by mirrors whereas older ones come from ASF infra/archives and have stricter bandwidth/rate

Re: Getting exception with multiple DBCPConnectionPools: sqljdbc_auth.dll already loaded in another classloader

2019-02-26 Thread Joe Witt
Brad What you’re running into is that the same native lib, by name, cannot be loaded more than once in the entire jvm. Since the component you’re using is loading the missal driver in two diff processors we are hitting this. We can work around this limitation of jvm classloader isolation by

Re: join two datasets

2019-02-22 Thread Joe Witt
I should add you can use NiFi to update the reference dataset in a database/backing store in one flow. And have another flow that handles the live stream/lookup,etc. MarkPayne/Others: I think there are blogs that describe this pattern. Anyone have links? On Fri, Feb 22, 2019 at 12:27 PM Joe

Re: join two datasets

2019-02-22 Thread Joe Witt
Boris, Great. So have a process to load the periodic dataset into a lookup service. COuld be backed by a simple file, a database, Hive, whatever. Then have the live flow run against that. This reminds me - we should make a Kudu based lookup service i think. I'll chat with some of our new Kudu

Re: join two datasets

2019-02-22 Thread Joe Witt
Right I agree with Bryan so let me expand a bit. There are some key primitives that stream processing systems address as it relates to joining two live streams that those systems are designed to solve well. NiFi offers nothing special/unique in that space. Now, as Bryan pointed out a really

[ANNOUNCE] Apache NiFi 1.9.0 release

2019-02-21 Thread Joe Witt
Hello The Apache NiFi team would like to announce the release of Apache NiFi 1.9.0. In addition to more than 100 improvements and bug fixes, this release makes integration with Apache Kudu and Impala a breeze. Provides stronger integration with Google Big Query and Amazon Web Services. New

Re: merging flowfiles?

2019-02-20 Thread Joe Witt
Hello You could put a funnel for your two jsonpath processors to send to then the funnel to mergecontent. That at least addresses the multi input paths comment. Whether your data can simply be merged like this or not is possibly another matter but I presume you have that in hand. Thanks On

Re: Proposal for ElasticSearch support

2019-02-20 Thread Joe Witt
...probably for dev thread but I am starting to think we should just start removing certain nars from the convenience build/assembly and documenting it in the migration guide for users that need those. We can then show them how to use the hot loading/etc.. Thanks On Wed, Feb 20, 2019 at 10:04

Re: Using variables in SSLContextService

2019-02-19 Thread Joe Witt
I agree that there is value in having EL enabled properties for some of the SSLContext properties. I dont understand the security concern raised but am open to what I might be missing. It would need variable and env var access. Thanks Joe On Tue, Feb 19, 2019 at 9:16 PM Beutel, Maximilian <

Re: Nifi provenance indexing throughput if it is being used as an event store

2019-02-17 Thread Joe Witt
and > set them to be indexed for the provenance, the mentioned rate should be > alright? > > Cheers, > Ali > > On Sat, Feb 16, 2019 at 2:56 PM Joe Witt wrote: > >> Ali >> >> You certainly can and at the rates you mention you should be able to keep >> it for

Re: 1.9 release date?

2019-02-16 Thread Joe Witt
dan we did rc1 this week and will have rc2 up today or tomorrow ideally. thanks On Sat, Feb 16, 2019, 10:42 AM dan young Heya folks, > > Any insight on 1.9 release date? Looks like a lot of goodies and fixes > included... > > Regards, > > Dano >

Re: Nifi provenance indexing throughput if it is being used as an event store

2019-02-15 Thread Joe Witt
Ali You certainly can and at the rates you mention you should be able to keep it for a good while. Just set the properties you need for your system and measure the rate at which prov storage fills. Thanks On Fri, Feb 15, 2019 at 10:29 PM Ali Nazemian wrote: > I didn't mean to use Nifi

Re: Asymmetric push/pull throughput with S2S, possibly related to openConnectionForReceive compression?

2019-02-14 Thread Joe Witt
...interesting. I dont have an answer but will initiate some research. Hopefully someone else replies if they know off-hand. Thanks On Thu, Feb 14, 2019 at 11:43 AM Pat White wrote: > Hi Folks, > > Could someone point me at the correct way to modify Nifi's embedded jetty > configuration

Re: Failed to read TOC File

2019-02-13 Thread Joe Witt
Chad, In your conf/nifi.properties please see what the implementation is for your provenance repository. This specied on nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository Is that what you have? The above error I believe could occur if the

Re: NiFi Repo's on Shared Storage

2019-02-09 Thread Joe Witt
Rich I haven't experimented with NFS in a long time but early results with NFS were very uneven. Performance in general was fine but stability as it related to locking and other behaviors was less desirable. NiFi is no longer as aggressive with file locks as it used to be so it is possible

Re: Provenance missing after nifi 1.6 -> 1.8 upgrade

2019-02-07 Thread Joe Witt
Hello If this is a secured instance please ensure your user has the proper permissions to access prov. thanks On Thu, Feb 7, 2019, 8:31 PM Kon Soulianidis Hi, > > After upgrading from 1.6 to 1.8 recently, I’ve noticed Provenance events > aren’t being generated (or if they are they aren’t

Re: Question on NiFi upgrade.

2019-01-25 Thread Joe Witt
Hello The Flow Registry is designed to deal with challenge #1 as far as versioned flows go. For the extensions themselves you'll really want to take advantage of the nar versioning capability since versioned flows can use specific versioned extensions. Once we make an extension registry happen

Re: Migrate NiFi 1.5 to 1.8 Error - A Blank Sensitive Properties Key Was Provided

2019-01-24 Thread Joe Witt
to initial post) (and yes, a smoking gun). We can > mark this issue as resolved as it has nothing to do with the initial error > I thought it was. Thanks for the quick response and help! > > Cheers, > > Ryan H > > On Thu, Jan 24, 2019 at 12:41 PM Joe Witt wrote: > >&g

Re: Migrate NiFi 1.5 to 1.8 Error - A Blank Sensitive Properties Key Was Provided

2019-01-24 Thread Joe Witt
gt; org.apache.nifi.StdErr Failed to start web server: Unable to start Flow > Controller. > 2019-01-24 17:33:08,479 ERROR [NiFi logging handler] > org.apache.nifi.StdErr Shutting down... > > > -Ryan H > > On Thu, Jan 24, 2019 at 11:48 AM Joe Witt wrote: > >>

Re: Migrate NiFi 1.5 to 1.8 Error - A Blank Sensitive Properties Key Was Provided

2019-01-24 Thread Joe Witt
> > On Thu, Jan 24, 2019 at 10:39 AM Joe Witt wrote: > >> Ryan, >> >> That block of text that shows up in the log could arguably said "WARN" >> because the flow will continue to function as it did before. >> >> However, the reason it is an error is

Re: ListSFTP Question

2019-01-24 Thread Joe Witt
hey josef. yeah we need to add a min file age property to ListSftp. please file a jira. thanks On Thu, Jan 24, 2019, 11:13 AM Hi guys > > > > We need your advice,… we use the ListSFTP processor to read files on a > remote folder. The files gets written like that: > > > >- File1 >-

Re: Migrate NiFi 1.5 to 1.8 Error - A Blank Sensitive Properties Key Was Provided

2019-01-24 Thread Joe Witt
Ryan, That block of text that shows up in the log could arguably said "WARN" because the flow will continue to function as it did before. However, the reason it is an error is that you really should follow its advice and specifically follow the secure nifi configuration guidance. By not

Re: Kafka max topics

2019-01-22 Thread Joe Witt
Ah wow. It was definitely arbitrary We could make it configurable and arbitrarily larger by default :) Please file a JIRA. Good catch as this makes the thread the other day make a lot more sense. thanks On Tue, Jan 22, 2019 at 4:06 PM Boris Tyukin wrote: > Hi guys, > > does anyone know

Re: NiFI as Data Pipeline Orchestration Tool?

2019-01-11 Thread Joe Witt
Jon First things first - Sonos is awesome. Now back to the matter at hand... NiFi is quite often used for various forms of orchestration of other systems doing their thing. However, I'll state that isn't really its primary purpose so for pure orchestration cases it can leave you with a less

Re: Wait/Notify inconsistent behavior

2019-01-04 Thread Joe Witt
gt; Thanks for letting me know. > > LC > > ---------- > *De: *"Joe Witt" > *Para: *"users" > *Enviados: *Viernes, 4 de Enero 2019 23:23:02 > *Asunto: *Re: Wait/Notify inconsistent behavior > > Please avoid sending more copies of

Re: Wait/Notify inconsistent behavior

2019-01-04 Thread Joe Witt
Please avoid sending more copies of the question. Hopefully someone familiar with the processors in question will be available in time. Thanks On Fri, Jan 4, 2019 at 9:14 PM Luis Carmona wrote: > Hi everyone, > > Im having a strange behavior with Wait / Notify mechanism. Attached is > the

Re: NIfi cluster on Docker

2018-12-18 Thread Joe Witt
Hello Not aware of folks working with AWS/Fargate but I am aware of a slew of NiFi on K8S work including NiFi on EKS...so I know that works well. Curious to hear your progress. Thanks Joe On Tue, Dec 18, 2018 at 7:57 AM Jean-Sebastien Vachon wrote: > > Hi all, > > > anyone has tried running a

Re: NiFi suddenly releases a lot of disk space on shutdown

2018-12-12 Thread Joe Witt
good catch mike. that is def a thing. historically the jvm had issues with memory mapped io as well not being able to let go of files until restart. On Wed, Dec 12, 2018, 7:19 PM Mike Thomsen Mike, > > I did lsof +L1 and saw a ton of files listed that were marked (deleted), > but the OS was

Re: How to get Node ID of disconnected node to be used in a cluster/node monitoring flow

2018-12-10 Thread Joe Witt
ocs/html/administration-guide.html#proxy_configuration > <https://urldefense.proofpoint.com/v2/url?u=https-3A__nifi.apache.org_docs_nifi-2Ddocs_html_administration-2Dguide.html-23proxy-5Fconfiguration=DwMFaQ=gJN2jf8AyP5Q6Np0yWY19w=MJ04HXP0mOz9-J4odYRNRx3ln4A_OnHTjJvmsZOEG64=Fu5GvEY5W6MSQUzs1kOAlkIcfLF

Re: How to get Node ID of disconnected node to be used in a cluster/node monitoring flow

2018-12-10 Thread Joe Witt
chad in 1.8.0 you should not see much need to script that. are you still seeing disconnects in 1.8.0 that it doesnt restore on its own. thanks On Mon, Dec 10, 2018, 10:08 AM Woodhead, Chad Hey Jeremy! > > > > We occasionally see a node disconnect from the cluster (not disappear from > the

Re: nifi 1.8 cluster load balancer properities

2018-12-04 Thread Joe Witt
Hello Have you reviewed this doc for load balancing properties? If you are using a 1.7.1 or earlier properties file youll want to diff with 1.8 to see what is new. http://nifi.apache.org/docs/nifi-docs/html/administration-guide.html thanks On Tue, Dec 4, 2018, 10:02 AM Vishal Jadhav

Re: hbaseclient service is failed to enable

2018-11-25 Thread Joe Witt
Ravi it is very tough to read that stack trace as provided but the core piece of it seems to be that authorization is failing for "/hbase". Have you confirmed that the proper authorization configuration is present? Thanks On Sun, Nov 25, 2018 at 4:50 PM Ravi Papisetti (rpapiset) wrote: > > Hi,

  1   2   3   4   5   6   7   8   >