Re: [discuss] NiFi 1.20 and NiFi 2.0

2023-01-09 Thread Phil H
Hi Joe, Sorry if this is a dumb question, but is there some documentation around what 2.0 might mean? My personal interest is in backwards compatibility of existing custom processors. Thanks, Phil On Tue, 10 Jan 2023 at 06:53, Joe Witt wrote: > Team, > > As David mentioned in [1] following a

Re: dealing with a breaking change

2022-10-08 Thread Phil H
It might be too much to do prior to solving this problem, but for the general case, but could NiFi benefit from a @Depricated annotation (or similar) on PropertyDescriptor. This could be read only, requiring the admin to change to the new property. Alternatively, the deprecated property could be

Compiling / Testing a single processor

2022-04-09 Thread Phil H
Hi there, Now that my environment is working nicely, I am working on SplitContent (Nifi-1517). Is there a way to tell maven to only compile and/or run the unit tests for the SplitContent processor, rather than testing the entire nifi-standard-processors bundle? Thanks, Phil

Re: IT WORKS

2022-04-07 Thread Phil H
Oh, and maven on my laptop works fine now that it’s on my home WiFi and not running via an iOS hotspot. Go figure... > On 7 Apr 2022, at 4:34 pm, Phil H wrote: > > Upgraded to Azul JDK11 on my VM and 1.17.0 now runs! > > I am re-running the full mvn clean install process

IT WORKS

2022-04-07 Thread Phil H
in, and apologies to everyone else for the spam. Now to get on with contributing to the project (I hope). Phil On Thu, Apr 7, 2022 at 11:55 AM Phil H wrote: > So I just compiled and executed the simplest HTTP program I could find > > > import java.io.IOException; > import java.ne

Re: New build problem, Java 17

2022-04-06 Thread Phil H
? > > Running the following OpenSSL command will attempt a socket connection with > TLS to the Maven repository server, to check connectivity: > > openssl s_client -host repo1.maven.org -port 443 > > Regards, > David Handermann > > On Wed, Apr 6, 2022 at 8:31 PM Phil H w

Re: New build problem, Java 17

2022-04-06 Thread Phil H
No issue running the openssl command. I’m just on my personal laptop connecting via my hotspotted phone - I’ve never had any sort of network type issues like this before. That’s why I wondered whether it was somehow maven/java related. Makes no sense to me! > On 7 Apr 2022, at 11:40 am,

Re: New build problem, Java 17

2022-04-06 Thread Phil H
On 7 Apr 2022, at 11:27 am, David Handermann > wrote: > > Phil, > > Can you provide the version of Maven you are using? Running Maven with the > -X argument for debug will generate a lot of logging information, but it > might point to the POM resolution problem. > > Regards,

Re: New build problem, Java 17

2022-04-06 Thread Phil H
Could be something funky with how Java 17's TLS APIs are working on your > setup. > > Java 8 and Java 11 are your best bets for now anyway. Java 17 support > is still in the early stages and isn't a preferred JDK/JRE. > > On Wed, Apr 6, 2022 at 7:00 PM Phil H wrote: >>

New build problem, Java 17

2022-04-06 Thread Phil H
So I upgraded to the Azul JDK 17. New repo, following the same instructions as before. phil@Phils-MacBook-Pro nifi % mvn -T C2.0 clean install -Pinclude-grpc [INFO] Scanning for projects... Downloading from central:

Re: Missing maven dependencies when building nifi

2022-04-06 Thread Phil H
tography Extension Policy. Upgrading to the latest > > Java 8 version should resolve the illegal key size issue shown in the > > build. > > > > Regards, > > David Handermann > > > > On Wed, Apr 6, 2022 at 6:16 AM Phil H wrote: > > > > > Austral

Re: Missing maven dependencies when building nifi

2022-04-06 Thread Phil H
Australia. Never had any other issues with OpenSSL stuff, etc On Wed, 6 Apr 2022 at 21:11, Otto Fowler wrote: > What country are you in? Are you under export controls? > > From: Phil H > Reply: dev@nifi.apache.org > Date: April 6, 2022 at 07:08:45 > To: dev@nifi.apac

Re: Missing maven dependencies when building nifi

2022-04-06 Thread Phil H
ption: Illegal key size at org.apache.nifi.security.kms.FileBasedKeyProviderTest.getSecretKeysPath(FileBasedKeyProviderTest.java:60) at org.apache.nifi.security.kms.FileBasedKeyProviderTest.testGetKey(FileBasedKeyProviderTest.java:47) On Wed, Apr 6, 2022 at 5:08 PM Phil H wrote: > Hi guys,

Re: Missing maven dependencies when building nifi

2022-04-06 Thread Phil H
, 2022 at 2:31 PM James Srinivasan < > james.sriniva...@gmail.com> > wrote: > > > Possibly not your problem, but did you follow the instructions for > building > > on Windows here: > > > > https://nifi.apache.org/quickstart.html > > > > On Tue, 5

Re: Missing maven dependencies when building nifi

2022-04-05 Thread Phil H
> Too many builds in different levels are leaving things around. > > Thanks > > On Mon, Apr 4, 2022 at 7:46 AM Phil H wrote: > > > So, about that error that flashed past... > > > > [INFO] > > ---

Re: Missing maven dependencies when building nifi

2022-04-04 Thread Phil H
> Thanks > > On Mon, Apr 4, 2022 at 7:46 AM Phil H wrote: > > > So, about that error that flashed past... > > > > [INFO] > > > > [INFO] BUILD FAILURE > > [INFO] > >

Re: Missing maven dependencies when building nifi

2022-04-04 Thread Phil H
xls = -- some binary stuff -- On Tue, Apr 5, 2022 at 12:30 AM Phil H wrote: > Thanks Joe, it seems to be about half way through now - much further than > previously. I thought I saw some red text flash past, but nothing > stopped. Does

Re: Missing maven dependencies when building nifi

2022-04-04 Thread Phil H
- build it all, run contrib check, and use all the > cores. > > Thanks > > On Mon, Apr 4, 2022 at 6:38 AM Mike Thomsen > wrote: > > > Did you run "mvn install" from the root of the project before > > attempting to run a full test of the code base? > > >

Re: SplitContent doesn’t support regex?

2022-04-04 Thread Phil H
in mind you should go for it. > Sounds interesting. > > Thanks > > On Wed, Mar 16, 2022 at 3:59 PM Phil H wrote: > > > Hi, > > > > This seems like an odd omission - aside from performance (presumably?) is > > there a reason why there isn’t a regex optio

Re: Missing maven dependencies when building nifi

2022-04-04 Thread Phil H
25) Re-running resulted in the same failure a second time (points for consistency!), but I'm not sure what to do about this one. Phil On Mon, Apr 4, 2022 at 11:38 PM Mike Thomsen wrote: > Did you run "mvn install" from the root of the project before > attempting to run a full t

Re: Missing maven dependencies when building nifi

2022-04-04 Thread Phil H
2\database\migration\H2DatabaseMigrator.java:[37,49] error: cannot find symbol [ERROR] symbol: class JdbcDataSource [ERROR] location: class H2DatabaseMigrator On Sat, Apr 2, 2022 at 10:20 PM Phil H wrote: > Hi there, > > Trying to build the nifi git repo for the first time and I'm getting the

Missing maven dependencies when building nifi

2022-04-02 Thread Phil H
Hi there, Trying to build the nifi git repo for the first time and I'm getting the message below I'm used to working in offline development environments, so I'm a bit lost with this online dependency stuff. I note that https://repository.apache.org/snapshots/org/apache/nifi/ doesn't contain any

Re: SplitContent doesn’t support regex?

2022-03-16 Thread Phil H
ve a good implementation in mind you should go for it. > Sounds interesting. > > Thanks > > On Wed, Mar 16, 2022 at 3:59 PM Phil H wrote: > > > Hi, > > > > This seems like an odd omission - aside from performance (presumably?) is > > there a reason why

SplitContent doesn’t support regex?

2022-03-16 Thread Phil H
Hi, This seems like an odd omission - aside from performance (presumably?) is there a reason why there isn’t a regex option for the byte sequence? I need one but thought I’d ask before I built my own. Thanks Phil

Issue with setting nifi.sensitive.props.key

2021-12-21 Thread Phil H
Hi there, I am in the process of trying to upgrade from 13.2 to 15.1. I did not have a sensitive props key set previously. Based on the upgrade guide, I ran nifi.sh set-sensitive-properties-key APassword When I ran nifi, it was complaining about a lack of specified algorithm. I ran up a new

Re: Primary node vs shutdown

2021-09-04 Thread Phil H
ep 1, 2021 at 7:56 PM Phil H wrote: > > > Hi Joe, > > > > It’s a custom one, but it is effectively just a routing filter component > > (read the data, send the flow file out on relationship A or B based on > what > > it finds). Nothing exotic in terms of how it int

Re: Primary node vs shutdown

2021-09-01 Thread Phil H
. Phil > On 2 Sep 2021, at 12:02 pm, Joe Witt wrote: > > Phil > > What processor reads from that queue that appears unmoving? > > Thanks > > On Wed, Sep 1, 2021 at 3:51 PM Phil H wrote: > >> And once reconnected again, no data passes that queue - it all jus

Re: Primary node vs shutdown

2021-09-01 Thread Phil H
And once reconnected again, no data passes that queue - it all just piles up there (the queue count matching the number of items sent into the cluster). However if I try and list the queue, it claims there are no files in it. Very very confused! On Thu, 2 Sep 2021 at 08:39, Phil H wrote: > O

Re: Primary node vs shutdown

2021-09-01 Thread Phil H
t it down. > > Thanks > Shawn > > -Original Message- > From: Phil H > Sent: Wednesday, September 1, 2021 4:36 PM > To: dev@nifi.apache.org > Subject: Primary node vs shutdown > > Hi there, > > I am noticing a number of situations where shutting down

Re: Primary node vs shutdown

2021-09-01 Thread Phil H
you > have NiFi's cluster load balancing setup. Then you'd disconnect the node > and shut it down. > > Thanks > Shawn > > -Original Message- > From: Phil H > Sent: Wednesday, September 1, 2021 4:36 PM > To: dev@nifi.apache.org > Subject: Primary node vs shut

Primary node vs shutdown

2021-09-01 Thread Phil H
Hi there, I am noticing a number of situations where shutting down one node in a cluster is leaving data stranded in the flows on that shut down server. Is there any way to tell NiFi to ship data off to other cluster members before it shuts down? Note I am restarting via the nifi.sh script, not

Re: Using nifi in separated networks

2021-08-02 Thread Phil H
as to your > thought process and design approach. > > OK, I'll step off a little, because I might be probing too hard here. But I > was legitimately curious about the intention of the proposed processor as > it relates to the mentioned Diode device. > > Thanks, > > Adam

Re: Using nifi in separated networks

2021-08-02 Thread Phil H
Hi Marc, Thanks for the additional info. Just so you know you’re not the only one, I’ve also had to re-implement a ListenTCP alternative to get around the byte delimeter issue for binary and multiline text data. Phil On Tue, Aug 3, 2021 at 6:59 AM Marc wrote: > > Hi Adam, > > more or less it

Re: Using nifi in separated networks

2021-08-01 Thread Phil H
That is interesting stuff - out of interest, if it was sent over that UDP diode, how would you know whether or not it got to the other side? I haven’t looked into the site-to-site functionality much yet but I assume it maintains the providence info? On Mon, 2 Aug 2021 at 04:26, Marc wrote: >

Re: Unit tests for multiple processors

2021-07-25 Thread Phil H
://github.com/TouK/plumber > > > > Tony Koval > > > > On 2021/07/22 03:25:31, Phil H mailto:g...@gmail.com>> wrote: > >> Hi there,> > >> > >> I use the built in unit tests in the maven processor archetype for each of> > &g

Unit tests for multiple processors

2021-07-21 Thread Phil H
Hi there, I use the built in unit tests in the maven processor archetype for each of my processors, but I would like to set up some integration testing using multiple processors. Is this possible? If so, anyone care to sling me some example code? Eg: say I wanted to run a test connecting

Re: SSLPeerUnverifiedException following upgrade from 1.6.0 to 1.13.2

2021-06-11 Thread Phil H
t; verification: > > https://square.github.io/okhttp/changelog_3x/#version-3100 > > Regards, > David Handermann > > On Thu, Jun 10, 2021 at 11:16 PM Phil H wrote: > > > Hi there, > > > > I upgraded an older dev setup today from 1.6.0 to 1.13.2. After a

SSLPeerUnverifiedException following upgrade from 1.6.0 to 1.13.2

2021-06-10 Thread Phil H
Hi there, I upgraded an older dev setup today from 1.6.0 to 1.13.2. After a couple of config tweaks, it’s “working”, but if I try and access the interface at https://nifi2.domain.blah/ I get a message on screen stating that nifi1.domain.blah is not verified. The logs contain this same message,

Re: SocketTimeoutException (read timeout) on HTTPS

2021-06-03 Thread Phil H
gt; threads, and relax some of the timeouts. > > nifi.cluster.protocol.heartbeat.interval= > nifi.cluster.node.protocol.threads= > nifi.cluster.node.protocol.max.threads= > nifi.cluster.node.connection.timeout= > nifi.cluster.node.read.timeout= > nifi.cluster.node.max.concurrent.requests= > >

Re: SocketTimeoutException (read timeout) on HTTPS

2021-06-02 Thread Phil H
gt; Also, is NiFi using its own dedicated zookeeper or is it shared with other > services? How many nodes is the zookeeper? > > > > > On Jun 2, 2021, at 7:54 PM, Phil H wrote: > > > > Hi there, > > > > I am getting a lot of these both in the web interface to

SocketTimeoutException (read timeout) on HTTPS

2021-06-02 Thread Phil H
Hi there, I am getting a lot of these both in the web interface to my servers, and in the cluster communication between the nodes. All other aspects of the servers are fine. TCP connections to NiFi, as well as SSH connections to the servers are stable (running for days at a time). I’m lucky if I

Re: PutFile to mounted directory

2021-05-13 Thread Phil H
a NFS share, the only way is to mount the share on the nifi nodes. > > Regards, > > Eduardo Fontes > > Em qui., 13 de mai. de 2021 01:11, Phil H escreveu: > > > Hi there, > > > > I want to use PutFile write some flow files out to a network drive o

PutFile to mounted directory

2021-05-12 Thread Phil H
Hi there, I want to use PutFile write some flow files out to a network drive on my Unix system, but fail if the disk is not mounted. I can’t see any way to do this - any ideas? Cheers, Phil

Re: Maximum load

2020-09-20 Thread Phil H
and see what that shows. > > > > Sent from my iPhone > > > > > On Sep 20, 2020, at 8:30 PM, Phil H wrote: > > > > > > Various numbers - max of 96 so far. > > > > > >> On Sat, 19 Sep 2020 at 21:44, Pierre Villard < > pierre.vi

Re: Maximum load

2020-09-20 Thread Phil H
Various numbers - max of 96 so far. On Sat, 19 Sep 2020 at 21:44, Pierre Villard wrote: > How many concurrent tasks did you set on the two custom processors? > > > > Le sam. 19 sept. 2020 à 02:16, Phil H a écrit : > > > > > Basically I have two custom processors

Re: Maximum load

2020-09-18 Thread Phil H
s. > > Actually, I usually recommend changing the pool size only when you see that > > all the threads available are used. > > Overall we'd need more details about your flow and observations. > > > > Thanks, > > Pierre > > > > Le ven. 18 s

Maximum load

2020-09-17 Thread Phil H
Hi there, Regardless of what I set in the maximum thread count settings, I never see more than 128 active threads. The CPU load of the system is only 5-8% when NiFi is running at this rate. It’s a Solaris system with 64 cores. Any help getting more performance would be appreciated! Regards,

PutTCP delays when multithreaded

2020-08-13 Thread Phil H
If I have more than one thread running for a PutTCP processor, it seems to add nearly 1 second of latency, even when I am only sending a single flow file through my system. This multiplexing does increase the total throughout of my system (items per hour) but this latency is hard to explain. Any

Re: Policy to clear queues

2020-06-04 Thread Phil H
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > On Jun 4, 2020, at 4:45 PM, Phil H wrote: > > > > The action is available in the menu. I get the following pop up: > > > > *Insufficient Permissions* > > > > *Node nifiX.domain.com:443 &

Re: Policy to clear queues

2020-06-04 Thread Phil H
5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > On Jun 4, 2020, at 4:45 PM, Phil H wrote: > > > > The action is available in the menu. I get the following pop up: > > > > *Insufficient Permissions* > > > > *Node nifiX.domain.com:443 <http://nifiX.doma

Re: Policy to clear queues

2020-06-04 Thread Phil H
70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > On Jun 4, 2020, at 4:22 PM, Phil H wrote: > > > > Hi guys, > > > > So I checked this morning, and the users are members of a group that has > > “modify the data” permission at the root level (and i

Re: Policy to clear queues

2020-06-04 Thread Phil H
roxied > > from the node being used to access the UI out to the remaining nodes. > > > > -Mark > > > > > > On Thu, Jun 4, 2020 at 6:52 AM Phil H wrote: > > > > > Hi Andy, > > > > > > Thanks for your reply. I don’t recall seein

Re: Policy to clear queues

2020-06-04 Thread Phil H
; PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > On Jun 3, 2020, at 4:08 PM, Phil H wrote: > > > > Hi there, > > > > I am trying to stratify my userbase. I need to allow certain users/groups > > the ability to clear queues, but cannot fin

Policy to clear queues

2020-06-03 Thread Phil H
Hi there, I am trying to stratify my userbase. I need to allow certain users/groups the ability to clear queues, but cannot find the right policy to allow that without also allowing them to delete queues, which I absolutely don’t want to do. Am currently using 1.9.2 (putting off the upgrade

Re: Cloned processor not loading

2019-12-15 Thread Phil H
s.Processor > > -Mark > > > On Sun, Dec 15, 2019 at 6:09 PM Phil H wrote: > > > Hi there, > > > > I have a working processor, Tom, which I cloned to form a second > processor, > > Jerry. All occurrences of Tom in the code/XML were replaced with Jerry in >

Cloned processor not loading

2019-12-15 Thread Phil H
Hi there, I have a working processor, Tom, which I cloned to form a second processor, Jerry. All occurrences of Tom in the code/XML were replaced with Jerry in the clone. The build process works for Jerry fine, as do the unit tests. However when I load the Jerry into the nifi extensions

Re: Singleton data inside processors

2019-12-09 Thread Phil H
" > then this is what controller services are designed to accommodate but it is > really important you're careful here as shared state like that is where > bottlenecks can occur. Most controller services are meant for a write > once/read many type pattern. > > Thanks > &g

Re: Singleton data inside processors

2019-12-09 Thread Phil H
quite common in many processors. > > Thanks > > On Sun, Dec 8, 2019 at 7:30 PM Phil H wrote: > > > Hi there, > > > > I have a custom processor that I want to contain some shared data across > > threads (this data is based on analysis of flow files, so constantly

Singleton data inside processors

2019-12-08 Thread Phil H
Hi there, I have a custom processor that I want to contain some shared data across threads (this data is based on analysis of flow files, so constantly changing). I can’t use a static variable though, because I use the same processor more than once within my flow in different contexts. If there

Re: Forcing cluster traffic to a single node

2019-12-05 Thread Phil H
> processor on all nodes, not primary node, because it may not be the primary > node that receives the data. > > Thanks > -Mark > > Sent from my iPhone > > > On Dec 5, 2019, at 9:04 PM, Phil H wrote: > > > > Hi there, > > > > I hav

Forcing cluster traffic to a single node

2019-12-05 Thread Phil H
Hi there, I have a GetTCP processor running on both members of my cluster. I want to force data received by both to a processor running on a single node. At the moment, if I set that processor to run on only the primary, the queue from the GetTCP processor on the secondary node just fills up.

Re: Cluster issue

2019-12-04 Thread Phil H
you sure that the certificates have valid SubjectAlternativeName > entries for the proper hostnames? > > Andy LoPresto > alopre...@apache.org > alopresto.apa...@gmail.com > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > On Dec 3, 2019, at 7:53 PM

Cluster issue

2019-12-02 Thread Phil H
Hi there, I (almost) have a secure nifi 1.9.2 cluster of two servers. Server A starts up fine, but Server B fails to start with this error (I cannot copy/paste as it is an offline system) Could not start listening for incoming connections in order to load balance data across the cluster. Please

Re: Cluster question - interval vs external routing

2019-11-07 Thread Phil H
ps://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#proxy_configuration > > > > Best regards, > Endre > > > > On 7 Nov 2019, at 03:49, Phil H wrote: > > > > Hi there, > > > > We have two addresses on each of our NiFi cluster

Cluster question - interval vs external routing

2019-11-06 Thread Phil H
Hi there, We have two addresses on each of our NiFi cluster hosts. One is externally facing, which we use to access the UI, and connect to receiving processors from the wider network. We are trying to configure NiFi to use the internal network for its cluster traffic. However if I use the

Re: Load issues

2018-10-23 Thread Phil H
memory, disks), number of nodes in the cluster, JVM heap size, > configuration of repositories, specific processors that are bottle necks, > etc. > > On Tue, Oct 23, 2018 at 10:02 PM Phil H wrote: > > > Yep - although it now has less throughput with 8 threads than it > > p

Re: Load issues

2018-10-23 Thread Phil H
Das wrote: > > Did you increase the no. of Concurrent Tasks and Run Duration on slow > processors ? > > Thanks, > Milan Das > > On 10/23/18, 8:26 PM, "Phil H" wrote: > >Hi team, > >My NiFi is struggling to process data (lots of back pres

Load issues

2018-10-23 Thread Phil H
Hi team, My NiFi is struggling to process data (lots of back pressure in various queues) but it is only using a very small amount (2-5%) of CPU according to top. Any ideas?

RE: Zookeeper - help!

2018-10-03 Thread Phil H
kind but it looks like it's not: http://zookeeper.apache.org/doc/r3.4.3/zookeeperStarted.html#sc_RunningReplicatedZooKeeper On 10/2/18, 5:17 PM, "Phil H" wrote: The second port in the zookeeper server config has been a mystery to me. I thought it was a second port used for

RE: Zookeeper - help!

2018-10-02 Thread Phil H
.port=9551 - nifi.cluster.node.address=nifi2.com - nifi.cluster.node.protocol.port=11441 nifi2/state/zookeeper /myid (file contents = "2") /state-management.xml (no changes required) /version-2/ Nathan On 10/2/18, 2:07 AM, "Phil H" wrote: Hi Andy, Thanks for the addition

RE: Zookeeper - help!

2018-10-02 Thread Phil H
://pierrevillard.com/2016/08/13/apache-nifi-1-0-0-cluster-setup/ [2]  https://pierrevillard.com/2016/11/29/apache-nifi-1-1-0-secured-cluster-setup/ Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69 On Oct 1, 2018, at 2:45 PM, Phil H

RE: Zookeeper - help!

2018-10-01 Thread Phil H
uses Zookeeper and how to configure it here: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#state_management. Let us know what happens once these properties are configured correctly. Nathan On 9/30/18, 11:07 PM, "Phil H" wrote: Hi guys, Pulling m

Zookeeper - help!

2018-09-30 Thread Phil H
Hi guys, Pulling my hair out trying to solve my Zookeeper problems. I have two 1.6.0 servers that I am trying to cluster. Here is the except from the properties files – all other properties are default so omitted for clarity. The servers are set up to run HTTPS, and the interface works via

RE: Ideal hardware for NiFi

2018-09-13 Thread Phil H
. If iostat shows always 100% usage then prob things arent ideal. thanks On Thu, Sep 13, 2018, 9:16 PM Phil H wrote: > Hi joe, > > I moved the content and providence repositories off to two new disks, but > it seems like the vast majority of the writes are still occurring on the

Re: Ideal hardware for NiFi

2018-09-13 Thread Phil H
, 14 Sep 2018 at 12:37, Joe Witt wrote: > if they are physically seperate the diff should be quite noticable. > > On Thu, Sep 13, 2018, 7:36 PM Phil H wrote: > > > Potentially. We're looking to see how the multiple disks help before > > committing to spending money on new ha

Re: Ideal hardware for NiFi

2018-09-13 Thread Phil H
ore involved. > > does that help? > > thanks > > On Thu, Sep 13, 2018, 4:36 PM Phil H wrote: > > > Follow up question - how do I transition to this new structure? Should I > > shut down NiFi and move the contents of the legacy single directories > into > >

Re: Ideal hardware for NiFi

2018-09-13 Thread Phil H
te to the content repo is 400 MB/sec). Similar with > Provenance Repository. > > Doing this also will allow you to hold a larger 'archive' of content and > provenance data, because > it will span the archive across all of the listed directories, as well. > > Thanks > -Mark > >

Re: Ideal hardware for NiFi

2018-09-11 Thread Phil H
gt; writing 100 MB/sec, then > your effective write rate to the content repo is 400 MB/sec). Similar with > Provenance Repository. > > Doing this also will allow you to hold a larger 'archive' of content and > provenance data, because > it will span the archive across all of the listed

Re: Ideal hardware for NiFi

2018-09-11 Thread Phil H
disk > caching, which > can certainly speed things up, especially if you're reading the content > several times for > each FlowFile. > > Does this help at all? > > Thanks > -Mark > > > > On Sep 10, 2018, at 6:05 AM, Phil H wrote: > > > > Thanks for tha

Re: Ideal hardware for NiFi

2018-09-10 Thread Phil H
nna > > On Mon, Sep 10, 2018 at 4:09 AM Phil H wrote: > > > Hi all, > > > > I've been asked to spec some hardware for a NiFi installation. Does > anyone > > have any advice? My gut feel is lots of processor cores and RAM, with > less > > emphasis on stor

Ideal hardware for NiFi

2018-09-09 Thread Phil H
Hi all, I've been asked to spec some hardware for a NiFi installation. Does anyone have any advice? My gut feel is lots of processor cores and RAM, with less emphasis on storage (small fast disks). Are there any limitations on how many cores the JRE/NiFi can actually make use of, or any other

Status history graphs over 24hrs (v1.3.0)

2018-08-02 Thread Phil H
Hi there, Is there a way to plot graphs longer than 24 hours? We are interested in seeing things that happen over the course of a week or so Thanks, Phil

Re: GC overhead limit exceeded (1.3.0)

2018-07-03 Thread Phil H
gt; > then you can start things until you find the problem. > > thanks > > On Tue, Jul 3, 2018, 8:31 PM Phil H wrote: > > > Hi guys, > > > > I got a backlog, noticed degraded performance, restarted NiFi, and now > the > > web UI won't start. I not

GC overhead limit exceeded (1.3.0)

2018-07-03 Thread Phil H
Hi guys, I got a backlog, noticed degraded performance, restarted NiFi, and now the web UI won't start. I note that I am getting lots of out of memory errors in the log. Is there a way I can clear the backlog from outside the application (deleting files, etc)? TIA, Phil

Re: Data flow rate exceeding provenance recording rate

2018-05-14 Thread Phil H
better (UI is responsive, and no errors or warnings) but the main GetTCP only seems to be spitting out flowfiles about 1/3rd as fast as it used to (network factors have been eliminated). Any more ideas? On Tue, 15 May 2018 at 09:36, Phil H <gippyp...@gmail.com> wrote: > Thanks Mark, > >

Re: Data flow rate exceeding provenance recording rate

2018-05-14 Thread Phil H
roperty from "org.apache.nifi.provenance.PersistentProvenanceRepository" > to "org.apache.nifi.provenance.WriteAheadProvenanceRepository". > The new implementation provides better throughput, will avoid the > problematic pauses that you're encountering now, > and also is quite a bit faster

Data flow rate exceeding provenance recording rate

2018-05-14 Thread Phil H
Hi gang, I have started receiving this error after perhaps 24 hours of run time. The first queue in our flow has a very large backlog by the time this error arrives. What is odd is that the incoming message rate is fairly constant at all times and while I am watching NiFi during the day, we never

Re: SSL_ERROR_NO_CYPHER_OVERLAP

2018-05-05 Thread Phil H
usual mistake is to fix the > authorizers.xml file but not delete users.xml and authorizations.xml files > before restarting. Could it be the case here? > > Pierre > > 2018-05-05 6:36 GMT+02:00 Phil H <gippyp...@gmail.com>: > > > No worries Andy, > > > > I

Re: SSL_ERROR_NO_CYPHER_OVERLAP

2018-05-04 Thread Phil H
Server-76] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (EMAILADDRESS=phil@blah, CN=Phil H, OU=Blah) GET https://nifi1.blah/nifi-api/flow/current-user (source ip: 192.168.10.192) 2018-05-05 14:10:29,310 INFO [NiFi Web Server-76] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for EMAILADDRESS

Re: SSL_ERROR_NO_CYPHER_OVERLAP

2018-05-04 Thread Phil H
ave the "Insufficient permissions" problem, which there is loads of documentation as to how to move forwards. Thanks! Phil On Sat, May 5, 2018 at 8:27 AM, Phil H <gippyp...@gmail.com> wrote: > Further info > > In the httpd installation, I need the private key for the certi

Re: SSL_ERROR_NO_CYPHER_OVERLAP

2018-05-04 Thread Phil H
sed PKCS12 for the keystore was some Jetty debug output telling me that was preferred to JKS. Interestingly it gave me contrary advice for the trust store. On Sat, 5 May 2018 at 08:22, Phil H <gippyp...@gmail.com> wrote: > Hi Andy, > > Sorry - poor use of words (it was late when I wrote the emai

Re: SSL_ERROR_NO_CYPHER_OVERLAP

2018-05-04 Thread Phil H
e...@apache.org > *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>* > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On May 4, 2018, at 4:03 AM, Phil H <gippyp...@gmail.com> wrote: > > Hi all, > > I am trying to secure my NiFi inst

SSL_ERROR_NO_CYPHER_OVERLAP

2018-05-04 Thread Phil H
Hi all, I am trying to secure my NiFi installation. I have a client certificate (nifi1.crt) and the CA for the intranet (ca.crt). I created the trust and keystores as below: keytool -import -trustcacerts -alias nifi1 -file nifi1.crt -keystore server_keystore.p12 -storetype PKCS12 keytool

SIGSEGV in Jetty

2018-04-25 Thread Phil H
Hi there, I am getting regular (maybe every ten minutes?) crashes in NiFi 1.3.0. This just started happening unrelated to any change to the software environment (i.e.: we haven't installed new code/processors). It roughly coincides with an increase in flow file throughout and my subsequent use

Re: Programmatically adding processors

2018-01-29 Thread Phil H
te in the UI and view the REST calls made. > > Hope that helps. > > Robert > >> On Mon, Jan 29, 2018, 21:31 Phil H <gippyp...@gmail.com> wrote: >> >> Hi there, >> >> I need to add new properties/routes to RouteOnAttribute and subsequent >> PutFil

Programmatically adding processors

2018-01-29 Thread Phil H
Hi there, I need to add new properties/routes to RouteOnAttribute and subsequent PutFile processors based on an external input (a basic web interface). Hoping someone can point me in the right direction in terms of either config files to change, or API calls to make to simulate what the NiFi

Re: "One to many" provenance

2017-11-20 Thread Phil H
processors like >> UnpackContent and QueryRecord that explicitly call >> ProvenanceReporter.fork(). >> >> Regards, >> Matt >> >>> On Mon, Nov 20, 2017 at 4:32 PM, Phil H <gippyp...@gmail.com> wrote: >>> Hi team, >>> >

Re: "One to many" provenance

2017-11-20 Thread Phil H
> Matt > >> On Mon, Nov 20, 2017 at 4:32 PM, Phil H <gippyp...@gmail.com> wrote: >> Hi team, >> >> If I have a processor that takes one input flow file, and then generates >> many flow file outputs as a result (say, one output per line from a >> mul

"One to many" provenance

2017-11-20 Thread Phil H
Hi team, If I have a processor that takes one input flow file, and then generates many flow file outputs as a result (say, one output per line from a multi-line input file), how do I indicate the provenance of the new flow files? I would like to see where they have come from for

Re: Classloader issues

2017-11-06 Thread Phil H
first person to ever make that mistake. > > Nope, not true :) No worries and thanks for following up. I believe > once we have the extension registry in play this sort of thing will be > much easier/simpler. > > Thanks > >> On Mon, Nov 6, 2017 at 1:20 PM, Phil H <gippyp

  1   2   >