Hi guys
We got some ERRORs in our bootstrap.log for some of our NiFI 1.23.2 nodes:
2024-01-08 11:18:21,172 INFO [NiFi logging handler] org.apache.nifi.StdOut
[9581188.605s][warning][codecache] Try increasing the code heap size using
-XX:NonProfiledCodeHeapSize=
2024-01-08 11:18:21,167 ERROR [NiF
Hi guys
Today we have just one datacenter with a few NiFi clusters, so we use a
dedicated 3-node zookeeper cluster in that datacenter. We are now planning to
expand to another datacenter, so we would like to split the NiFis as well as
zookeeper. However 2 zookeeper nodes is not a good quorum nu
Hi guys
We just upgraded our 8 node NiFi cluster to v1.21.0 and we are hitting a SSH
timeout issue with the PutSFTP processor.
May be someone can help us to find the root cause for our issue or guide us
into the right direction. The error message below must be related to the “Data
Timeout”, as
Hi Bence
We still facing problems with the current NiFi v1.20.0 release and nested
process groups. Would you mind having a look at the issue?
https://issues.apache.org/jira/browse/NIFI-11251
It seems that NiFi is not correctly stopping/starting the processors within the
(nested?) PGs. As worka
Hi David
Thanks a lot for answering here in this topic.
A few comments to your reply.
* SSHD config and the “MaxSession” setting -> when we started
troubleshooting we thought as well that this could help to fix our issue.
However based on the sshd_config manpage (see below) this setting i
Hi guys
It was upgrade time again on our side, we just upgraded from 1.19.1 to 1.20.0.
Since 1.20.0 we do see significantly more SSH Connection Timeout errors on the
PutSFTP processor…
PutSFTP Processor ERROR:
2023-02-16 07:44:07,905 ERROR [Timer-Driven Process Thread-50]
o.a.nifi.processors.s
Hi Jens, Joe
Thank you for your comments and the hint regarding the “CalculateRecordStats”
processor – didn’t know that this even exists!
Jira Ticket https://issues.apache.org/jira/browse/NIFI-10996 has been created.
Thanks in advance.
Cheers Josef
From: "Jens M. Kofoed"
Reply to: "users@ni
Hi guys
I’ve pretty basic question. We use a “ConvertRecord” processor where we convert
an AVRO to a CSV. For that CSV output we would like to have the header enabled,
so we tried to set “Include Header Line – true” for the Controller Service of
the CSVRecordSetWriter. The issue is, if we have
Hi Bence
I mentioned yesterday another issue which we thought would be solved after the
upgrade steps. Sadly it happened again and after an investigation on our side
it seems to be fully reproducible, I’ve opened a Jira Bugticket
https://issues.apache.org/jira/browse/NIFI-10973 with more detail
Hi Bence
Ok got it, thanks a lot, So we will try to sync as less as possible. Let’s hope
that we will see a NiFi 1.19.1 soon…
Cheers Josef
From: Simon Bence
Reply to: "users@nifi.apache.org"
Date: Thursday, 1 December 2022 at 13:42
To: "users@nifi.apache.org"
Subject: Re: NiFi Registry Bug w
Hi Bence, Joe
Thank you guys for your fast response.
@Bence you are right, the whole issue seems to be related to nested flows which
we try to sync. We are relying heavily on this as we are doing the integration
on one NiFi system and then sync it to production via the NiFi Registry. Our
paren
Thank you guys for fixing it and for providing the workaround. We searched in
the release notes for any hints regarding the migration from 1.15.3 to 1.18.0
and found nothing, may be a short notice would be great that a upgrade from
<.1.16.0 to 1.17.0/1.18.0 could brake some of the sensitive prop
Yepp I would say it comes with milliseconds, but if you need it you can convert
afterwards to seconds like in my example “${myDate:toDate("-MM-dd
HH:mm:ss","UTC"):toNumber():divide(1000)}”
Cheers
From: James McMahon
Reply to: "users@nifi.apache.org"
Date: Monday, 3 October 2022 at 22:30
T
Hi Stephen
Not sure whether I understood you correct and if I really remember it correct,
but the UpdateRecord processor below should produce a NULL value if the string
is “” (empty). We don’t use it anymore, so I’m not 100% whether it still works
or does what you want. And yes it’s not straigh
I’ve created a Jira Ticket:
https://issues.apache.org/jira/browse/NIFI-9991
Thanks Tamás Pálfy for taking care of the problem :-). I’m curious about the
outcome.
Cheers Josef
From: Tamás Pálfy
Reply to: "users@nifi.apache.org"
Date: Thursday, 5 May 2022 at 14:07
To: "users@nifi.apache.org"
Hi guys
Since NiFi 1.14.0 there is an ASN1 decoder called JASN1Reader included into
NiFi and we are testing it now for the first time.
We have created a ConvertRecord processor and added the JASN1Reader with the
configurations below. As RecordWriter we use JSON or CSV, but NiFi always
successf
Hi Guillermo
Please check my screenshot from Microsoft Oulook calendar, you see that the 1.
Januar is still in week 52. In java it seems that this behavior is different,
the 1. January has been counted as number 1. So in outlook week 1 is in java
week 2. Hope this helps.
Cheers Josef
[Backgro
Hi Jean-Sébastien
I’ve done a quick test for you and the following doesn’t produce an error, but
it doesn’t exactly do what you want as the result for the flowfile content is
the following: '[1 2, 4 5]' |jq 'join( )'
[Graphical user interface, text, application, email Description automatically
Hi guys
We can confirm the slow browser behavior as well and it’s very annoying. We
have single node NiFis as well multiple NiFi clusters with different sizes. It
happens everywhere and is definitely browser specific. We’ve also tried to
restart NiFi, but no change at all. It so slow that in 2-
Sorry I had another issue that caused my test to fail. I can confirm that
double quoting (two single quotes) work to get a single quote.
Cheers
From: "Zahner Josef, GSB-LR-TRW-LI"
Date: Thursday, 6 May 2021 at 09:40
To: "users@nifi.apache.org"
Subject: NiFi QueryRecord Escape Single Quote in
Hi guys
We would like to insert a string as a column with the QueryRecord which
includes a single quote, please check my example below.
SELECT 'Can't create a text with single quote' AS message
FROM FLOWFILE
The yellow marked single quote generates a syntax error and I don’t know how to
fix it
I’m just jumping in, we are seeing this issue as well when we are restarting
the nifi process from time.
We are aware of the nifi.properties
“nifi.flowcontroller.graceful.shutdown.period=10 sec” parameter, but to be
honest we didn’t try to raise it up yet. Maybe it takes more than 10s to fully
Hi Henrique
Did you lift up the OS UDP Receive Buffer Size? Per default it’s way too small
under linux.
sysctl -w net.core.rmem_max=16777216
With netstat -su (“Udp:” Section) you can check whether you have UDP buffer
issues…
Cheers Josef
From: Henrique Nascimento
Reply to: "users@nifi.apach
Hi Andy & Kotaro
Thank you for your comments. So this means we can’t upgrade to nifi 1.12.0 :-(
(except if we change the certs, which is no option at the moment).
@Andy: I’m aware of the wildcard certificate notes from the documentation. We
don’t have a wildcard certificate with a ‘*’ sign in i
Good point Russell, we upgraded NiFi Registry last time in March 2020, since
then two new versions came out, that’s correct. We plan to upgrade to NiFi
Registry 0.7.0 together with NiFi v.1.12.0.
To be honest, I don’t expect that the issue is on Registry side. But I’m
curious what other people
Hi guys
We are using NiFi 1.11.4 with NiFi Registry 0.5.0. We are trying to migrate
from variables to parameters.
As soon as we are adding a “Process Group Parameter Context” to an existing
process group and change something else to be able to commit it to the NiFi
Registry, it seems that the
It tells you most probably that the CA cert from the remote HTTPS server hasn’t
been found in the truststore you’ve defined to access the site. So please check
again the CA cert and the truststore…
Cheers Josef
From: "White, Daniel"
Reply to: "users@nifi.apache.org"
Date: Thursday, 6 August
Hi Etienne
Did you tried the following in «nifi-registry.properties»:
nifi.registry.security.needClientAuth=false
Cheers Josef
From: Etienne Jouvin
Reply to: "users@nifi.apache.org"
Date: Tuesday, 30 June 2020 at 10:46
To: "users@nifi.apache.org"
Subject: Need help SSL LDAP Nifi Registry
He
Hi Mark,
it really doesn’t matter what I configure for “Max Poll Records” (tried 10,
1’000, 100’000) or for “Max Uncommitted Time” (tried 1s, 10s, 100s). The
flowfile size is always randomly between 1 and about 500 records. So those two
parameters doesn’t work at all in my case. The theory behi
Hi Mark,
thanks a lot for your explanation, makes fully sense! Did you checked as well
the “Max Poll Records” parameter? Because no matter how high I’m setting it I’m
getting always a random number of records back into one flowfile. The max. is
about 400 records which isn’t ideal for small reco
Perfect, thanks Pierre :-).
Cheers Josef
From: Pierre Villard
Reply to: "users@nifi.apache.org"
Date: Monday, 23 March 2020 at 10:54
To: "users@nifi.apache.org"
Subject: Re: maven nifi 1.11.4 libs
Hi Josef,
It can take a bit before sync is complete between repositories. It should be OK
late
As NiFi 1.11.4 has been released yesterday, we tried to upgrade our processors
to 1.11.4. Sadly maven shows no update yet for 1.11.4, am I to early? We can’t
wait to install the new release (bugfix release, yeah).
Cheers Josef
smime.p7s
Description: S/MIME Cryptographic Signature
With logback.xml you can finetune logmessages, but don't ask me the details __.
Cheers Josef
On 04.03.20, 16:35, "Dobbernack, Harald (Key-Work)"
wrote:
Hi Josef,
thank you for your input! We planned on merging/aggregating the errors into
one mail an hour (so the length of the m
Hi Harald,
I can tell you only what we do, we are sending the whole nifi-app.log to splunk
and scan there specific for alarms/warnings. We don’t use e-mail notification
as it doesn’t help too much. In the nifi-app.log you would see as well startup
issues, so we just focus on that instead.
In m
If you mean “Cron driven” scheduling strategy which has exactly a start of
08:00:00, then it won’t run afterwards. If you use “Timer driven” scheduling,
it will start again after a restart. So all depends on your scheduling strategy.
From: Mike Thomsen
Reply to: "users@nifi.apache.org"
Date:
Oh sorry, missed one of of the most important parts, we are using a 8-node
cluster with nifi 1.11.3 – so perfectly up to date.
Cheers Josef
From: Bryan Bende
Reply to: "users@nifi.apache.org"
Date: Wednesday, 4 March 2020 at 12:57
To: "users@nifi.apache.org"
Subject: Re: FetchSFTP keeps files
Hi guys,
We have an issue with the FetchSFTP processor and the max open file
descriptors. In short, it seems that the FetchSFTP keeps the file open
“forever” on our Synology NAS, so we are reaching always the default max open
files limit of 1024 from our Synlogy NAS if we try to fetch 500’000 s
Hi Valentina,
has your template/canvas disabled processors? I’ve already faced a bug which
caused the same error as yours.
https://issues.apache.org/jira/projects/NIFI/issues/NIFI-6958
Cheers Josef
From: Valentina Ivanova
Reply to: "users@nifi.apache.org"
Date: Wednesday, 5 February 2020 at
Just a short information for other people who are using the PutKudu processor
and are planning to Upgrade to NiFI 1.10.0. Before you start, check the jira
tickets below, especially the memory leak… If we had known that before, then we
probably would not have upgraded.
PutKudu related Jira Ticke
Thank you Pierre, your workaround works like a charm! JIRA request has been
created: https://issues.apache.org/jira/projects/NIFI/issues/NIFI-6895
From: Pierre Villard
Reply to: "users@nifi.apache.org"
Date: Thursday, 21 November 2019 at 10:52
To: "users@nifi.apache.org"
Subject: Re: PutKudu
Hi guys
We have just upgraded to NiFi 1.10.0 yesterday. We have seen that we got 1’000
times more logs (nifi-app.log) since the upgrade, caused mainly by the PutKudu
processor.
The logmessage we always get is:
2019-11-21 08:42:27,627 WARN [Timer-Driven Process Thread-2]
org.apache.kudu.client.
Update: we found out that the described issue (fallback to “simple” mode) is
related to Java-11 (and of course LDAP with START_TLS) only. The error message
is gone with Java 1.8.0, so in our case we will use for now Java 1.8.0. As
already mentioned earlier, another option would be to use Java 11
And additionally, below the output of the tcpdump captured on the NiFi node
during startup of NiFi 1.10.0. We use the standard LDAP port (389). And you
where right, I see in the dump that NiFi tries to authenticate with “simple”
authentication with START_TLS…
[cid:image001.png@01D59881.72F2B710
Hi Andy,
I’ve just opened a jira bugreport:
https://issues.apache.org/jira/projects/NIFI/issues/NIFI-6860
We changed nothing on the LDAP. The whole setup still works for our production
nodes with NiFi 1.9.2, we have multiple clusters/single NiFi’s running. As we
use ansible I’ve removed again N
Hi guys
We would like to upgrade from NiFi 1.9.2 to 1.10.0 and we have HTTPS with LDAP
(START_TLS) authentication successfully enabled on 1.9.2. Now after upgrading,
we have an issue which prevents nifi from startup:
2019-11-11 08:29:30,447 ERROR [main] o.s.web.context.ContextLoader Context
Yes of course… 4 servers.
From: "Wesley C. Dias de Oliveira"
Reply to: "users@nifi.apache.org"
Date: Wednesday, 30 October 2019 at 16:56
To: "users@nifi.apache.org"
Subject: Re: NiFi WebGUI Timestamp issue
Are you using NTP servers?
Em qua, 30 de out de 2019 às 11:36,
mailto:josef.zahn...@sw
Hi Wesley
I didn’t change anything on the machine during the time when I restarted the
cluster (multiple times), however NiFi showed after each restart different
timestamps/localizations.
Cheers Josef
From: "Wesley C. Dias de Oliveira"
Reply to: "users@nifi.apache.org"
Date: Wednesday, 30 Oc
Hi Guys,
We just faced an issue with the time shown in the NiFi webgui (Screenshot
below).
We are in Switzerland, so at the moment we have “CET” (UTC + 1h). After a
restart of our 8-node NiFi 1.9.2 cluster, NiFi suddenly showed “CET” - but the
timestamp was in fact UTC, so minus 1h. Then we de
Sorry there was a copy/paste mistake for the spec, in fact I tried with the
following one:
[
{
"operation": "shift",
"spec": {
"cgnat_public_ipv4": "cgnat_public_src_ipv4"
}
}
]
Spec seems to be valid as the processor runs fine.
From: Jean-Sebastien Vachon
Reply to: "use
Hi guys,
I’m jumping in here, we see the same as Giulia, just in another timeframe. We
are using the default provenance settings (nothing changed except the path)
with 4GB space on the provenance partition. Are there any suggestions what to
change? Of course as I’m checking it now we should lif
The feature requires NiFi > 1.8.x… Pierre describes it very well in his blog :
https://pierrevillard.com/2018/10/29/nifi-1-8-revolutionizing-the-list-fetch-pattern-and-more/
From: James McMahon
Reply-To: "users@nifi.apache.org"
Date: Tuesday, 9 July 2019 at 14:46
To: "users@nifi.apache.org"
S
Hi Carlos
I’m facing exact the same issue and I posted already in the cloudera forum.
However no useful response yet ☹:
https://community.cloudera.com/t5/Interactive-Short-cycle-SQL/JDBC-Driver-Upgrade-to-2-6-9-fails-on-Apache-NiFi/td-p/89754
I asked as well in the impala user group
(http://mai
Hi Matt
nifi.web.http.host has to be the hostname of your maschine and not the cluster
name! if the name is wrong you are getting serious problems.
Additional properties for external zookeeper below. That’s just what came to my
mind, hope it’s everything. Just to be sure, your NiFi Cluster was
Perfect, thank you Bryan.
Cheers Josef
On 06.02.19, 15:43, "Bryan Bende" wrote:
Hi Josef,
Thanks for the heads up. Submitting a patch or pull request and
letting everyone know is generally the correct approach.
To get it into a release will require a committer with Ku
Hi
A colleague just created a jira ticket to add an additional parameter to the
NiFi PutKudu processor. Any suggestions how to get this code change into one of
the future NiFi releases?
Jira Ticket:
https://issues.apache.org/jira/browse/NIFI-5989
Cheers Josef
smime.p7s
Description: S/MIME Cr
Hi Tom
In the wait processor you can define how long you wanna delay the fetch and
that value needs to be the longest expected writing time. Of course this works
only if you know what the max. writing time is. In my case this is clear as in
max. every hour at least one new file gets created. So
Hi Tom
I suggest to use a Wait Processor (to delay the fetch) together with
DetectDuplicate Processor. In that way you will fetch the file only once and
after it has been written completely (as long as you know how long it takes in
max. to finish writing). I know it’s not nice but that’s how we
Hi Joe
Done! https://issues.apache.org/jira/browse/NIFI-5977
Cheers Josef
From: Joe Witt
Reply-To: "users@nifi.apache.org"
Date: Thursday, 24 January 2019 at 17:18
To: "users@nifi.apache.org"
Subject: Re: ListSFTP Question
hey josef. yeah we need to add a min file age property to ListSftp.
FYI
I’ve just created a bugreport – I don’t think this is expected behavior.
https://issues.apache.org/jira/browse/NIFI-5976
I’ve also described a workaround in there.
Cheers Josef
From: "Zahner Josef, GSB-LR-TRW-LI"
Date: Saturday, 26 January 2019 at 13:07
To: "users@nifi.apache.org"
Subjec
Hi Jim
Did you solve the issue? I have exactly the same behavior… I wanted to try
Redis but ended up with this error.
Cheers Josef
On 2018/11/29 21:54:25, "Williams, Jim"
mailto:j...@alertlogic.com>> wrote:
> Hello,>
>
> >
>
> I'm trying to set up the>
> RedisConnectionPoolService/RedisDistr
Hi guys
We need your advice,… we use the ListSFTP processor to read files on a remote
folder. The files gets written like that:
* File1
* File2
* File3 (at the time of “ls” command this file is growing, we don’t know
how big it gets or when writing is finished)
So the application
Thanks Chad for confirming it.
@Bryan Bende: how shall we continue here? I understand that it isn't a high
prio issue, however it would be great to get it fixed or at least know that it
will be fixed in one of the future releases...
Cheers Josef
On 08.01.19, 20:34, "Chad Woodhead" wrote:
I've tried it now on another secured cluster exactly like you did:
1) Create PG "A" and save to registry
2) Import PG "A" from registry and rename to "B"
3) Add new processor (Execute Script) to original PG "A" and save to
registry
4) Change version on PG "B" to ne
Hi Bryan
In my case it happens all the time, doesn't matter what kind of change. On my
first test below I've changed a variable on a processor inside the PG and the
second time (a few seconds ago) I've added a connection to my "Execute Script"
processor. All the time my second PG with the new n
Hi guys
I’ve faced again an (at least for me) unexpected behavior of NiFi 1.8.0
together with NiFi Registry.
Following use case:
1. Create a process group with name “abc” and add some processors to the pg
2. Commit the pg to the NiFi registry
3. Create a new pg and import the pg from
Mark,
Yes we are using Load Balancing capability and we do that after the ListSFTP
processor, so yes we loadbalance 0-Byte files. Seems that we probably facing
your Bug here.
Thanks a lot for explaining in detail what happens regarding the
flowfile/content repo in NiFi.
Additionally we have s
Hi Jianan
I just say that as soon as “Minimum Number of Entries” is reached the flow can
be flushed out, and further if the minimum number isn’t reached I would expect
that the “Max Bin Age” takes place. Have you tried that?
Cheers Josef
From: Jianan Zhang
Reply-To: "users@nifi.apache.org"
D
Hi Jianan
As you have “Minimum Number of Entries: 1” it is normal that you can see merges
with only one flowfile. In my opinion the “Minimum Number of Entries” is
stronger than the “Max Bin Age” (first is written bold and second not).
Additionally it is called “Max Bin Age” and not “Bin Age”. S
Hi Asanka
For a CRON which runs once a day at 7 AM I use this pattern “0 0 7 * * ?”. So I
don’t see any reason why your CRON doesn’t work. Can you try with an example
like 1 AM instead of midnight and see what’s happening?
Cheers Josef
From: Asanka Sanjaya
Reply-To: "users@nifi.apache.org"
Hi Dano
Seems that the problem has been seen by a few people but until now nobody from
NiFi team really cared about it – except Mark Payne. He mentioned the part
below with the diagnostics, however in my case this doesn’t even work (tried it
on standalone unsecured cluster as well as on secured
@Mark: I’m now able to query the REST-API. However, the call
nifi-api/processors//diagnostics does reply on all nodes “An
unexpected error has occurred. Please check the logs for additional details.”
If I remove the /diagnostics I’m getting a useful reply in JSON format. I tried
on several nifi
Looks pretty much the same in our case, just that we have a 8 node cluster (all
nodes connected).
* Flowfile expiration: 0
* Back Pressure Object Threshold: 100
* Size Threshold: 1 GB
* Load Balance Strategy: Round Robin
* Load Balance Compression: Do not compress
We ha
Hi Mark
We see the issue again, even after a fresh started cluster (where we started
everything at the same time). The files stuck for multiple seconds/minutes in
the queue and the light blue loadbalancing icon on the right side shows that it
is actually loadbalancing the whole time (even if it
Same Issue here! Had to reboot the whole cluster to fix that issue. The files
stuck a few seconds/minutes in the queue until they get processed. In my case I
assume that it was caused by a one-by-one reboot of the cluster nodes, means we
normally reboot only one of 8 nodes and wait a few seconds
74 matches
Mail list logo
