ListFile not fetching files when the source has future dated files vs local NiFi system time

Is there a way to get the ListFile processor to list files which are not older than max but have a min file age that would be negative if using the local system time? Some of the source location files are in the future vs the NiFi machine and they are not being listed. I don't want to resort to '

Re: Apache Nifi Openshift Deployment Error

Are you mounting a folder to the host (from each container)? If yes, perhaps the permissions/user/group are not correct on the host mounted folder. Regards, Jeremy On 7 Sep 2023, at 01:34, chris.s.jos...@lmco.com wrote:  Hi, I am new to apache nifi, and I am trying to deploy it to openshi

Remote Process Group URL from a variable

Can the Remote Process Group reference a Variable/Parameter? I am trying to deploy flows to different servers whose RPG is not the same fixed destination but this seems to need a hard coded list of URLs. Context: I'm using the NiFi Docker image to deploy NiFi with an included flow on many standal

Re: Broken pipe: InvokeHTTP

You may also want to check if there's something in the middle terminating it for you, ngnix, proxy, OpenShift route, ...Regards,JeremyOn 15 Mar 2023, at 08:07, Patrick Timmins wrote: Hello Richard, Do you have *any* requests from the InvokeHTTP processor that are act

Re: How do you use container-based NiFi ?

Hi Isha, I agree with what Adam has said. Our setups are small 4-16 node clusters, we run multiple single large flows (real-time and batch flows). For our environment we use OKD (OpenShift) for container orchestration and to provide the routes/load balancing through a service to the NiFi cluster

Re: Unable to start nifi service

Instead of the bootstrap log check the nifi-app.log. It might be out of memory or have the port it needs already in use. Regards, Jeremy On 11 Dec 2022, at 11:01, James McMahon wrote:  I am trying to start nifi on an AWS EC2 instance. My bootstrap.conf says the service does not start, but

ODBC with unixODBC in Linux

Hi, has anyone had success using unixODBC to make an ODBC connection to a DB that supports this mode of connection in Linux (CentOS/Debian). We are trying to connect NiFi to Kx's kdb+ and this is the initial feedback that they gave to connect to the time series database. Or has anyone connected t

Re: MiNiFi C2 server 0.5.0 bug or not?

Hi Tom, I think so, I sent a similar email recently to the list about this. And raised the issue here: https://issues.apache.org/jira/browse/NIFI-9075 Jeremy On Tue, Sep 21, 2021 at 8:10 PM Tomislav Novosel < tomislav.novo...@clearpeaks.com> wrote: > Hi to all, > > > > I am using MiNiFi C2 ser

Re: Minifi 1.14.0 exception - sensitive props key

Hi Tom, I recall that I used the NiFi 1.14.0 binary to set the key and then copied that over to the MiNiFi properties file to get it to work. Linux only: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#updating-the-sensitive-properties-key ./bin/nifi.sh set-sensitive-propert

Problem with 0.5.0 C2 server converting template in NiFi 1.12.1 for 1.14.0 MiNiFi

Hi, I'm trying to learn the MiNiFi + C2 NiFiRestConfigurationProvider setup. If I download my template and use the mini-toolkit-1.14.0 to convert the template there is no issue with the remote process group target port id so the template should be good. But querying the C2 server config api produ

Re: Release from holding queue after timeout

n to 20s: > > flowFile = session.get() > if(!flowFile) return > session.penalize(flowFile) > session.transfer(flowFile, REL_SUCCESS) > > Paul > > On Wed, Apr 14, 2021 at 4:22 PM Jeremy Pemberton-Pigott < > fuzzych...@gmail.com> wrote: > >> Yes it can.

Re: Release from holding queue after timeout

for 20s before the next processor will try acting on it. Paul On Wed, Apr 14, 2021 at 5:05 AM Jeremy Pemberton-Pigott wrote: > Hi, > > I have a parallel update process in 2 different flows running in a 3 node > cluster running 1.6.0, if the insert side has not completed yet the up

Release from holding queue after timeout

Hi, I have a parallel update process in 2 different flows running in a 3 node cluster running 1.6.0, if the insert side has not completed yet the update side moves the flow file to a waiting queue for retry. I want to retry every 20s all the flow files in the queue that have been waiting that lon

Re: Detect duplicate record reader

pache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.DetectDuplicate/index.html >> ? >> >> >> On 15. Feb 2021, at 04:45, Jeremy Pemberton-Pigott >> wrote: >> >> Hi everyone, I'm wondering if there is a Detect Duplicate processor tha

Detect duplicate record reader

Hi everyone, I'm wondering if there is a Detect Duplicate processor that can read records from a flow file and as output gives just the non-duplicates (can be single records or a group of non-duplicates would be better). I want to use a record reader to avoid splitting the json content into 1s

Re: TLS ConsumeMQTT processor causes NullPointerException exception

not true. > > Can you create a jira for this? > > On Mon, Sep 28, 2020 at 10:43 AM Jeremy Pemberton-Pigott > wrote: > > > > Hi, > > > > I'm trying to set up an MQTT TLS consumer in 1.12.0 with ConsumeMQTT. I > am initiating the connection

TLS ConsumeMQTT processor causes NullPointerException exception

Hi, I'm trying to set up an MQTT TLS consumer in 1.12.0 with ConsumeMQTT. I am initiating the connection to a remote EMQX MQTT TLS broker. If I configure the StandardRestrictedSSLContextService for a JKS truststore only (no keystore settings since it isn't required) and start the ConsumeMQTT I g

Calling DLL functions on a Windows PC

Hi, Is there a processor that allows me to make functions in a local DLL? Jython or maybe Python script perhaps or I would have to write a custom jar with JNI calls? NodeJS Addons has the ability to do it and it works, now I want to make the same calls from a NiFi flow without NodeJS. Jeremy

Re: MiNiFi version

...right nars should generally port well. >> >> On Tue, Jul 21, 2020 at 10:25 AM Jeremy Pemberton-Pigott < >> fuzzych...@gmail.com> wrote: >> >>> Thanks Joe, that's really helpful, I have an immediate use for the Java >>> version. If I wanted to us

Re: MiNiFi version

gt; Also under very active development. > > Thanks > > On Tue, Jul 21, 2020 at 9:39 AM Jeremy Pemberton-Pigott < > fuzzych...@gmail.com> wrote: > >> Hi, >> >> Can someone help out with the version of NiFi that MiNiFi tracks? The >> Java version seems

MiNiFi version

Hi, Can someone help out with the version of NiFi that MiNiFi tracks? The Java version seems to be at 0.5 for a long time while the C++ version is 0.7. I'm also wondering if MiNiFi is actively developed or is it risky to include it in new projects? Jeremy

Re: Session state in cluster HandleHttpRequest and HandleHttpReponse

ter > On Tue, Jun 30, 2020 at 6:04 AM Jeremy Pemberton-Pigott > wrote: > Hi, > > I have a cluster of 3 nodes and the incoming request on one node's > HandleHttpRequest may be replied to by a different node's HandleHttpResponse, > in between there is a Spark strea

Session state in cluster HandleHttpRequest and HandleHttpReponse

Hi, I have a cluster of 3 nodes and the incoming request on one node's HandleHttpRequest may be replied to by a different node's HandleHttpResponse, in between there is a Spark streaming job process. Is there any example how to do that, maybe with DistributeMapCacheService? So that I can still r

Re: Listing a folder with millions of files

rtain if that would have any effect. Edward On Mon, 9 Mar 2020, 06:34 Jeremy Pemberton-Pigott, wrote: Hi, I need to list a sub-set (few 100,000) of files in a folder with millions of files (to do some historical processing). What's the best way I can do that? ListFiles is taking way too long

Listing a folder with millions of files

Hi, I need to list a sub-set (few 100,000) of files in a folder with millions of files (to do some historical processing). What's the best way I can do that? ListFiles is taking way too long and seems to try to dump the entire list to the flow when I test it on a smaller folder list. It would b

1.6.0 odd queue behavior

Hi, I have an oversized queue due to some unknown problem. That's not the issue at the moment. I am restarting the cluster of 3 nodes (all data is stuck on 1 node) each time the queues stop processing data. What I observe is that from a queue of about 2 million flow files it will proceed throug

Re: Oversized queue between process groups

FlowFile Repository. > > Thanks > -Mark > > > [1] https://issues.apache.org/jira/browse/NIFI-5997 > > > > On Aug 30, 2019, at 11:20 PM, Jeremy Pemberton-Pigott < > fuzzych...@gmail.com> wrote: > > Thanks for your reply Mark. > > The flow was in sync

Implement heartbeat from remote servers

Does anyone have an idea of how to implement a heartbeat message posted from remote Nifi servers to another server running Nifi? This is so that I can tell if within say 1 hour there was no heartbeat received from a known list it could generate a log message with an error message of the server that

Re: Oversized queue between process groups

swapped back in on Windows but I think that was introduced after 1.6.0 and then fixed quickly. This should help to know where to focus energy on finding the problem. Thanks -Mark > On Aug 30, 2019, at 12:24 PM, Jeremy Pemberton-Pigott > wrote: > > Yes there is one but not nea

Re: Oversized queue between process groups

t to such numbers. Can you give us more details? >> What version of NiFi are you using? What's the configuration of your >> relationship between your two process groups? >> >> Thanks, >> Pierre >> >>> Le ven. 30 août 2019 à 07:46, Jeremy Pemberton-

Re: Oversized queue between process groups

gt; Pierre > > Le ven. 30 août 2019 à 07:46, Jeremy Pemberton-Pigott < > fuzzych...@gmail.com> a écrit : > >> Hi, >> >> I have a 3 node Nifi 1.6.0 cluster. It ran out of disk space when there >> was a log jam of flow files (from slow HBase lookups). My queue i

Oversized queue between process groups

Hi, I have a 3 node Nifi 1.6.0 cluster. It ran out of disk space when there was a log jam of flow files (from slow HBase lookups). My queue is configured for 20,000 but 1 node has over 206 million flow files stuck in the queue. I managed to clear up some disk space to get things going again but

Cluster remote process group load balancing under OpenShift

I've setup a 3 node Nifi cluster with 1.6.0 running in containers with OpenShift and using an external Zookeeper, been running for about a year just fine. Setup and added Remote Process Groups with HTTP transport for load balancing but 100,000s of flow files always end up on only 1 server instead