I'll review this closely once again when I get back to this system tonight
- thanks very much for your reply, Isha.
I also feel I need to look more closely in nifi.properties, at values I
have set for keys nifi.security.identity.mapping.[value, transform,
pattern].CN1
I noticed some odd behavior
Hi James,
Have you changed these settings in authorizers.xml since you first started
NiFi? If so, you may need to delete users.xml and authorizations.xml.
A new admin user will not be created if those files already exist.
Otherwise, the trickiest part is usually that the user DN needs to match
Thank you Pierre. I appreciate your reply. I have not yet done this but
intend to, as soon as I can figure out how to get my NiFi 2.0 instance to
come up securely with certs and TLS. I do and did not want to load up my
flows into a nonsecured NiFi instance, which is why I have taken so long to
> Regards,
>
>
>
> Isha
>
>
>
>
>
> *Van:* James McMahon
> *Verzonden:* vrijdag 19 april 2024 01:17
> *Aan:* users
> *Onderwerp:* Re: Unable to securely connect to NiFi 2.0 instance
>
>
>
> I started from scratch. Got nifi to start, no errors at
which CAs it will accept for client certs.
Regards,
Isha
Van: James McMahon
Verzonden: vrijdag 19 april 2024 01:17
Aan: users
Onderwerp: Re: Unable to securely connect to NiFi 2.0 instance
I started from scratch. Got nifi to start, no errors at all in my nifi-app.log.
Configured the client
I started from scratch. Got nifi to start, no errors at all in my
nifi-app.log. Configured the client certs in my Chrome browser, also added
cacert.pem to my Root Trusted CAs.
Tried to hit https://ec2-44-219-227-80.compute-1.amazonaws.com:8443/nifi ,
continue to get rejected with this message from
In NiFi 1.x, right click on process group, download flow definition. It'll
give you a JSON file. In NiFi 2, drag and drop a process group on the
canvas, and click the upload icon on the right of the name input. You'll be
able to select your JSON file.
HTH
Le jeu. 11 avr. 2024 à 02:14, James
Hi Shamsudeen,
Thanks for the reply and clarifying the use of the protocol. UDP does
not offer any kind of delivery guarantees at the protocol level, so it
is even less reliable than TCP for syslog. In other words, data loss
can happen without any type of error being logged.
If the data loss is
Hi David,
Thanks for the information. I'm actually using the UDP port to transfer
logs to the syslog system. Planning to upgrade to TCP in the future but
just need to figure out why it's hit \ miss.
Regards,
*Shamsudeen Jameer*
On Thu, Mar 28, 2024 at 10:37 PM David Handermann <
Hi Shamsudeen,
Although TCP incorporates packet acknowledgement as a core feature of
the protocol, syslog does not provide any type of delivery guarantees
on top of TCP. This means that it is possible for a system such as
NiFi to send a syslog message, without logging a socket communication
.
>
> Regards
>
>
> *Envoyé:* mardi 19 mars 2024 à 14:53
> *De:* "Joe Witt"
> *À:* users@nifi.apache.org
> *Objet:* Re: java.lang.OutOfMemoryError: Java heap space : NIFI 1.23.2
> Hello
>
> The key output is
>
> java.lang.OutOfMemoryError: Jav
; to 1 and it is worked.
Regards
Envoyé: mardi 19 mars 2024 à 14:53
De: "Joe Witt"
À: users@nifi.apache.org
Objet: Re: java.lang.OutOfMemoryError: Java heap space : NIFI 1.23.2
Hello
The key output is
java.lang.OutOfMemoryError: Java heap space
Review batch prope
Specifically set Fetch Size to something like 1000, by default setting
Fetch Size to zero will cause Postgres to fetch the entire ResultSet into
memory [1]. We should probably change that default to avoid problems like
this and with other drivers (for example, Oracle's default is 10 rows which
is
Hello
The key output is
java.lang.OutOfMemoryError: Java heap space
Review batch property options to limit response sizes in the database calls.
Thanks
On Tue, Mar 19, 2024 at 6:15 AM wrote:
> Hello
>
> I got the executeSQL processor does the sql command "select * from
> public.table1"
>
Hi Mark,
thanks for your feedback.
In nifi-app.log, the last stacktrace seems pretty explicit:
Caused by: org.apache.nifi.encrypt.EncryptionException: Key Password length
less than required [12]
at
Franck,
Can you take a look at the nifi-app.log? That should provide more information
than the bootstrap log does.
Thanks
-Mark
> On Mar 14, 2024, at 5:09 AM, Franck Routier via users
> wrote:
>
> Any idea anyone ?
>
> Le jeudi 7 mars 2024, 09:08:45 CET Franck Routier a écrit :
>> Hi,
>>
Any idea anyone ?
Le jeudi 7 mars 2024, 09:08:45 CET Franck Routier a écrit :
> Hi,
>
> here is the result of the launch:
>
> root@nifi:/opt/nifi# bin/nifi.sh run
> nifi.sh: JAVA_HOME not set; results may vary
>
> Java home:
> NiFi home: /opt/nifi-2.0.0-M2
>
> Bootstrap Config File:
I’m not an expert with MergeRecord, but looking at your screenshots, I’d guess
that your setup is taking that long to reach one of the defined “maximum”
settings, e.g. 2GB, 5,000,000 records, or 3600 seconds (1 hour).
How large (number of records and content size in bytes) are the typical
David,
Makes sense. Large values should not be added as attributes. Attributes are
designed for String values. Think 100-200 characters, generally. A couple KB
can be fine, as well, but it can significantly reduce performance. If the
intent is to “stash the content” so that you can change it
Mark,
Yes, it was the flowfile repository.
Of all your points, the large attributes is most likely our issue. One of
our folks was caching the flowfile (which can be large occasionally) in an
attribute ahead of a DB lookup (which would overwrite the contents) and
then reinstating the content
Thanks David.
I will try then share it again. :)
Regards,
DongCheol Gong
2024년 3월 5일 (화) 오후 10:38, David Handermann 님이
작성:
> Thanks for following up on this issue, the additional logging is helpful.
>
> The example URL provided does not include the port number (
> https://nifi.my-site.com)
Similar situation here. Not a devops guy. So I guess it’s fair to say this is expected behavior if policies are not written to allow nifi.sh and such to access the folders NiFi needs to access. Thanks for the help.Sent from my iPhoneOn Mar 8, 2024, at 5:38 PM, Russell Bateman wrote:
That's what lies in those "SELinux policies."
I think it's simple: Use SELinux to lock the filesystem (and other
stuff) up so no one can get in or go around. Then create specific
policies that allow, in this case, NiFi, access to its filesystem (like
//opt/nifi/current-nifi//) so that it can
I think the admin told me that even a simple nifi.sh start won’t work. Just
won’t even start the script and it is marked executable. I was wondering if
there were any gotchas to getting a basic setup running.
Sent from my iPhone
> On Mar 8, 2024, at 4:29 PM, Russell Bateman wrote:
>
> We
We have run on CentOS with SELinux set to enforcing and have run NiFi in
that environment for probably 8 or 9 years now. We do install some
SELinux policies that allow NiFi to access the filesystem underneath
itself and not outside that filesystem.
What specifically are you asking?
On 3/8/24
Dave,
When you say that the journal files are huge, I presume you mean the FlowFile
repository?
There are generally 4 things that can cause this:
- OutOfMemoryError causing the FlowFile repo not to properly checkpoint
- Out of Disk Space causing the FlowFile repo not to properly checkpoint
-
ticket for the improvement. Look for the
>
> *Dynamic Properties:*
> Supports Sensitive Dynamic Properties: *Yes*
>
> in the processor usage.
> Lehel
> --
> *From:* Roman Wesołowski
> *Sent:* Thursday, March 7, 2024 17:48
> *To:* users@nifi.apache.org
Sent: Thursday, March 7, 2024 17:48
To: users@nifi.apache.org
Subject: Re: RestLookupService question
Hi,
I have an additional question regarding RestLookupService. I will use this
thread.
When we add a header in InvokeHTTP processor we can set "Sensitive Value"
Yes/No, however dur
; *From:* Gregory Foreman
> *Sent:* Thursday, March 7, 2024 12:53
> *To:* users@nifi.apache.org
> *Subject:* Re: RestLookupService question
>
> I assume this is a bug. I just created NIFI-12875 (
> https://issues.apache.org/jira/browse/NIFI-12875) to track.
>
> On M
The response code is only used in debug logs, there is room to improve error
handling,
Thanks for reporting!
From: Gregory Foreman
Sent: Thursday, March 7, 2024 12:53
To: users@nifi.apache.org
Subject: Re: RestLookupService question
I assume this is a bug. I
I assume this is a bug. I just created NIFI-12875
(https://issues.apache.org/jira/browse/NIFI-12875) to track.
> On Mar 1, 2024, at 8:04 AM, Gregory Foreman
> wrote:
>
> Hello:
>
> The RestLookupService service seems to treat all http response status codes
> the same. Is this intentional?
Hi,
here is the result of the launch:
root@nifi:/opt/nifi# bin/nifi.sh run
nifi.sh: JAVA_HOME not set; results may vary
Java home:
NiFi home: /opt/nifi-2.0.0-M2
Bootstrap Config File: /opt/nifi-2.0.0-M2/conf/bootstrap.conf
2024-03-07 08:05:28,742 INFO [main]
Hey there Franck,
Can you provide the full error message with stack trace that gets printed?
Thanks
-Mark
On Mar 6, 2024, at 1:26 PM, Franck Routier via users
wrote:
Hi,
I'm a first time wanabe user of Nifi, and I followed the installation guide,
until the point when I try to launch Nifi:
Thanks for following up on this issue, the additional logging is helpful.
The example URL provided does not include the port number (
https://nifi.my-site.com) which seems to imply the presence of a gateway or
reverse proxy server in front of NiFi. This configuration is supported, but
in some
Hi guys.
I solved and share my story.
I was able to check the logs that the destination information was different
as shown below, but there was no problem with the settings in OKTA.
2024-03-04 12:14:33,051 DEBUG [NiFi Web Server-26]
o.s.s.s.p.s.a.OpenSamlAuthenticationProvider Found 2
Thanks David,
I know it's not going to be easy to resolve my issue.
I'll change the loglevel as you suggested and test again.
Have a nice and happy weekend.
Regard,
DongCheol Gong
2024년 3월 1일 (금) 오전 10:36, David Handermann 님이
작성:
> Thanks for providing some background on the issue with SAML
Thanks for providing some background on the issue with SAML configuration.
The following post describes the steps required for configuring NiFi to
integrate with Okta, including example configuration settings:
rn...@domain.com as "Initial User Identity 1" and "Initial Admin
> Identity" in authorizers.xml?
>
>
>
> 2.2) If one is using DN "Initial User Identity 1" and "Initial Admin
> Identity" in authorizers.xml and the "USE_DN" for "
Hi Anders, you are not alone :-)
I've the same issue and have tried to create some other jira's for a long
time ago.
FetchSmb: Addind completing strategy -
https://issues.apache.org/jira/browse/NIFI-12231?filter=-2
PutSmb: Add a rename strategy
Mark:
Ok, thanks for clarifying.
Greg
> On Feb 26, 2024, at 11:38 AM, Mark Payne wrote:
>
> Greg,
>
> Yes, the Relationship is generally only populated for ROUTE events, such as
> RouteOnAttribute.
>
> Thanks
> -Mark
>
>
>> On Feb 26, 2024, at 11:34 AM, Gregory Foreman
>> wrote:
>>
>>
Greg,
Yes, the Relationship is generally only populated for ROUTE events, such as
RouteOnAttribute.
Thanks
-Mark
> On Feb 26, 2024, at 11:34 AM, Gregory Foreman
> wrote:
>
> Hello:
>
> I am having an issue with the InvokeHttp processor provenance events. No
> Relationship field is
hat should one use as "User Search Filter"
in login-identity-providers.xml?
Thanks
From: Matthew Clarke
Sent: Sunday, February 25, 2024 11:19 PM
To: users@nifi.apache.org
Cc: Alexei Rozenvaser
Subject: Re: Insufficient Permissions - Unable to view the user interface
Hello Alexei,
Since you are loving in using your sAMAccountName, I recommend using them
identity string instead of the full DN of your AD users. In your
ldap-provider, change USE_DN to USE_USERNAME. Upon successfully
authentication the username you provided in the login window will be posted
to
t;Authentication Strategy" for LDAP-provider at my
login-identity-providers.xml don't FileUserGroupProvider +
FileAccessPolicyProvider enough at my authorizers.xml?
From: Michael Moser
Sent: Thursday, February 22, 2024 4:24 PM
To: users@nifi.apache.org; Alexei Rozenvaser
Subject: Re: Insufficie
ary 22, 2024 4:33 PM
To: users@nifi.apache.org
Subject: Re: Insufficient Permissions - Unable to view the user interface - at
WebUI
You don't often get email from
j...@thefribergs.com<mailto:j...@thefribergs.com>. Learn why this is
important<https://aka.ms/LearnAboutSenderIdentification&
Hey Pierre,
This has solved the issue for me. Thank you!
From: Pierre Villard
Sent: 23 February 2024 13:59
To: users@nifi.apache.org
Subject: Re: Using PutIceberg to insert into a table in S3 fails with
ClassNotFoundException
I believe that you'd need to build the processor with specific
I believe that you'd need to build the processor with specific profiles
enabled which is not the case in the default build that we provide for
convenience.
In nifi-inceberg-bundle, you could do something like
maven clean install -Pinclude-hadoop-aws
And use the generated NARs in your NiFi
consider to done.
Also, I have tested the Notify and Wait, but I remember it is not working in the cluster environment.
Anyway, I'm trying to deal with these informations.
Thanks
Minh
Envoyé: mercredi 21 février 2024 à 16:49
De: "Lehel Boér"
À: "users@nifi.apache
Hi,
Thanks for bringing this to our attention and for the investigation.
I have identified a potential root cause and created a JIRA ticket for it.
https://issues.apache.org/jira/browse/NIFI-12836
The observed behavior, as described by you and confirmed through my debugging,
aligns with a
Did you use a different user to setup ldap then your AD user your trying to
login with now?
On Thu, Feb 22, 2024 at 6:59 AM Alexei Rozenvaser
wrote:
> Hi Everyone
>
>
>
> I have a brand new installation of NiFi 2.0.
>
> I configured LDAP user authentication.
> I have my AD DN defined as both
Hello Alexei,
If you have configured an org.apache.nifi.ldap.LdapProvider in your NiFi
login-identity-providers.xml for *authentication* then you will also have
to configure an org.apache.nifi.ldap.tenants.LdapUserGroupProvider in your
NiFi authorizers.xml for *authorization*. Some instructions
My ./config/authorizers.xml
file-user-group-provider
org.apache.nifi.authorization.FileUserGroupProvider
./conf/users.xml
CN=My
Name,OU=MyOU,DC=MyDomain
file-access-policy-provider
Subject: RE: Easiest way to secure NiFi automatically
Hi Stanislav,
I recently set up Nginx on my cluster and found there were extra end points to
configure for reverse proxy other than /nifi/ as described in the admin guide.
I think they are associated with the underlying API that the GUI
Hi Stanislav,
I recently set up Nginx on my cluster and found there were extra end points to
configure for reverse proxy other than /nifi/ as described in the admin guide.
I think they are associated with the underlying API that the GUI incorporates.
I used this config for my proxy.conf file.
Hi,
In NiFi, counters serve monitoring purposes but are not directly accessible
through expression language, and their values are lost upon restart. Instead, I
suggest using an UpdateAttribute processor for this task. Configure it to
'Store State Locally' with an initial value of 0. Add a
Thank you Arvind. I removed the comment but the processor continues to fail
its configuration check with the same error.
Our nifi version is 1.16.3.
I experimented with other configuration properties of the processor and
seem to have hit on success. I had tried setting Jolt Transformation DSL to
Hi James,
Sharing your Nifi version would help identify the issue.
However, one thing which I encountered while working with Nifi version
1.23.2 that the processor does not like embedded comments in the
transformations. I needed to remove all the comments.
I tested yours by removing the comment
Good to know - very helpful. Thank you both, Joe W. and Mark P.
On Thu, Feb 15, 2024 at 10:42 AM Joe Witt wrote:
> This [1] blog seems amazingly appropriate and wow do we need these/any
> such fields we intend to truly honor in a prominent place in the docs.
> Super useful...
>
> [1]
Etienne.
Is this what you're looking for?
https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#hostname
Thanks,
Phil
On Wed, Feb 14, 2024 at 5:41 AM Etienne Jouvin
wrote:
> Hello.
>
> Imagine, I use UpdateAttributes to set an attribute where I put the NiFi
> instance id
This is proving to be difficult to do in practice. Many of the filenames in
the zip contain spaces and other characters, and these are failing to be
passed to the tar successfully.
This is the command I am testing at the command line to first extract the
filenames:
unzip -l
Please send an email to users-unsubscr...@nifi.apache.org !
More info: https://nifi.apache.org/community/contact/
On Thu, Feb 15, 2024 at 4:54 PM Martin, Stanley L <
stanley.mar...@hexagonusfederal.com> wrote:
>
>
>
>
>
>
Richard,
It sounds like your scripted reader is responsible for parsing the Avro? In
short, the Record appears to have an Avro Utf8 value, not a String, in the
field you’re looking at. You could call .toString() on that Utf8 object, or you
could configure the Avro reader to return Strings
This [1] blog seems amazingly appropriate and wow do we need these/any such
fields we intend to truly honor in a prominent place in the docs. Super
useful...
[1] https://jameswing.net/nifi/nifi-internal-fields.html
Thanks
On Thu, Feb 15, 2024 at 8:35 AM Mark Payne wrote:
> Jim,
>
> You can
Jim,
You can actually reference “lastQueueDate” in Expression Language. It is
formatted as number of milliseconds since epoch.
So you might have a RouteOnAttribute that has a property named “old” with a
value of:
${lastQueueDate:lt( ${now():minus(1)} )}
So any FlowFile that has been
That would work - what a good suggestion. I'll do that. I can format the
resulting number and then RouteOnAttribute by the desired subset of the
result.
Something like this to set attribute dt.failure:
${now():toNumber():toDate("-MM-ddHH:mm:ss"):format("MMddHHmmss","EST")}
Then I can
You could add an UpdateAttribute processor first in the failure path to add a
new attribute which contains the time the error occurred by using the ${now()}
or ${now():toNumber()} expression language function.
https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#now
Then
As it turns out lineageStartDate and Queued Duration are very different.
Without being able to get at Queued Duration as an attribute, it appears we
cannot RouteOnAttribute to filter thousands in a queue by anything like
hours they have been in queue.
Why would this be helpful? Let us say we have
Hi,
This is a test pipeline reading pdf files from disk. It begins with a
GetFile processor supplying a ConvertRecord processor with a scripted
reader input and an avrorecordsetwriter, generic output.
The scripted reader places the file content in a "content" field:
List recordFields = []
Hey Richard,
I think you’d need to explain more about what you’re doing in your groovy
script. What processor are you using? What’s the script doing? Is it parsing
Avro data?
On Jan 29, 2024, at 12:26 AM, Richard Beare wrote:
Anyone able to offer assistance with this?
I think my problem
What a great workaround, thank you once again Mike. I'll put this in and
use it now.
Jim
On Tue, Feb 13, 2024 at 4:41 PM Michael Moser wrote:
> Hello James,
>
> I'm not aware of a way to access Queued Duration using expression
> language, but you can access the Lineage Duration information.
Do the expression language functions ip() and hostname() give you what you need?
Steve Hindmarch
From: Etienne Jouvin
Sent: Wednesday, February 14, 2024 10:41 AM
To: users@nifi.apache.org
Subject: Re: Access instance name
You don't often get email from
lapinoujou...@gmail.com
Hello.
Imagine, I use UpdateAttributes to set an attribute where I put the NiFi
instance id or name.
This attribute may be used after, for example when I use PutRecord and if I
want to add information regarding the NiFi node that do the work.
In cluster mode, you can have a node identifier in
Etienne,
What instance name / id are you referring to?
On Tue, Feb 13, 2024 at 8:43 AM Etienne Jouvin
wrote:
> Hello all.
>
> Just simple question, is there a way to access, from controller service,
> to the instance name / id event in not cluster implementation?
>
> Regards
>
> Etienne Jouvin
Hello James,
I'm not aware of a way to access Queued Duration using expression language,
but you can access the Lineage Duration information. The Getting Started
Guide mentions both entryDate and lineageStartDate as immutable attributes
on all flowfiles. These are numbers of milliseconds since
Hello,
One approach to ensure default attribute values for specific processors is to
create a processor with the desired configuration in the NiFi UI and then copy
it whenever needed. This method retains the configured values, saving time
during replication.
Another effective strategy is to
Yes, that's exactly what those commands do. Your linux commands like unzip
and tar can probably read directly from /dev/stdin and write directly to
/dev/stdout if you want to.
-- Mike
On Fri, Feb 2, 2024 at 9:22 AM James McMahon wrote:
> Hi Michael. This is a very clever approach: convert
Hi Michael. This is a very clever approach: convert from a zip (which
UnpackContent does not preserve file metadata for extracted files) to a tar
(for which UnpackContent does preserve file metadata), then employ the
UnpackContent.
One quick followup question. The ExecuteStreamCommand will be in
Hi Jim,
The ExecuteStreamCommand will only output 1 flowfile, so using it to unzip
in this fashion won't yield the results you need.
Instead, you might try a workaround with ExecuteStreamCommand to unzip your
file and then tar to repackage it. Then UnpackContent should be able to
read the tar
If anyone can show me how to get my ExecuteStreamCommand configured
properly as a workaround, I am still interested in that.
Jim
On Wed, Jan 31, 2024 at 12:39 PM James McMahon wrote:
> I tried to find a Create option for tickets here,
>
I tried to find a Create option for tickets here,
https://issues.apache.org/jira/projects/NIFI/issues/NIFI-11859?filter=allopenissues
.
I did not find one, and suspect maybe I have no such privilege perhaps?
In any case, thank you for creating that.
Jim
On Wed, Jan 31, 2024 at 12:37 PM Joe Witt
I went ahead and wrote it up here
https://issues.apache.org/jira/browse/NIFI-12709
Thanks
On Wed, Jan 31, 2024 at 10:30 AM James McMahon wrote:
> Happy to do that Joe. How do I create and submit a JIRA for consideration?
> I have not done one - at least, not for years.
> If you get me started,
Happy to do that Joe. How do I create and submit a JIRA for consideration?
I have not done one - at least, not for years.
If you get me started, I will do a concise and thorough description in the
ticket.
Sincerely,
Jim
On Wed, Jan 31, 2024 at 12:12 PM Joe Witt wrote:
> James,
>
> Makes sense
James,
Makes sense to create a JIRA to improve UnpackContent to extract these
attributes in the event of a zip file that happens to present them. The
concept of lastModifiedDate does appear easily accessed if available in the
metadata. Owner/Creator/Creation information looks less standard in
Anyone able to offer assistance with this?
I think my problem relates to correctly specifying types using expression
languages and using schema inference from groovy.
On Tue, Jan 23, 2024 at 2:20 PM Richard Beare
wrote:
> Hi,
> What is the right way to deal with dates in the following context.
typo: so you can't run the DB and Git providers simultaneously.
Le mar. 16 janv. 2024 à 18:08, Bryan Bende a écrit :
> Hello,
>
> You can have only one persistence provider defined, so you can run the
> DB and Git providers simultaneously.
>
> Thanks,
>
> Bryan
>
> On Tue, Jan 16, 2024 at 6:52
Hello,
You can have only one persistence provider defined, so you can run the
DB and Git providers simultaneously.
Thanks,
Bryan
On Tue, Jan 16, 2024 at 6:52 AM Roman Wesołowski
wrote:
>
> Hello,
>
> We would like to have 3 nodes cluster with Nfii Registry one for each
> environment. So for
Out of curiosity - I'm seeing the metric "nifi_amount_threads_active" at
the max of 40 for processors that are disabled. Does that make sense? That
seems very odd to me since those processors should be doing anything at all.
On Mon, Jan 15, 2024 at 12:01 PM Aaron Rich wrote:
> Yeah - that gets
Yeah - that gets the performance to where we need it.
But the question I have is why did the performance drop in the first place.
Everything was working fine, and then it suddenly dropped. I'm having to
adjust nifi parameters to try to get back to where performance was but I
can't find what is
Aaron,
It doesn’t sound like you’re back to the drawing board at all - sounds like you
have the solution in hand. Just increase the size of your Timer Driven Thread
Pool and leave it there.
Thanks
-Mark
On Jan 15, 2024, at 11:16 AM, Aaron Rich wrote:
@Mark - thanks for that note. I hadn't
@Mark - thanks for that note. I hadn't tried restarting. When I did that,
the performance dropped back down. So I'm back to the drawing board.
@Phillip - I didn't have any other services/components/dataflows going. It
was just those 2 processors going (I tried to remove every variable I could
to
Juan, to do as you suggest would I set Content-Encoding in the processor
configuration to utf-8? Does that achieve this?
Joe O., I don't see any property in the Properties of the InvokeHTTP
processor that allows me to expect and handle spaces in the header. I also
don't explicitly create a
0x20 is a space. Maybe that's somewhere in your header?
On 1/12/2024 4:33 PM, James McMahon wrote:
I have a text flowfile that I am trying to send to a translation
service on a remote EC2 instance from my nifi insurance on my EC2. I
am failing with only this somewhat-cryptic error:
it seems charset issue. if it is a json add charset=utf-8
On Fri, Jan 12, 2024, 6:33 PM James McMahon wrote:
> I have a text flowfile that I am trying to send to a translation service
> on a remote EC2 instance from my nifi insurance on my EC2. I am failing
> with only this somewhat-cryptic
Ditto...
@Aaron... so outside of the GenerateFlowFile -> PutFile, were there
additional components/dataflows handling data at the same time as the
"stress-test". These will all share the same thread-pool. So depending
upon your dataflow footprint and any variability regarding data volumes...
20
Aaron,
Interestingly, up to version 1.21 of NiFi, if you increase the size of the
thread pool, it increased immediately. But if you decreased the size of the
thread pool, the decrease didn’t take effect until you restart NiFi. So that’s
probably why you’re seeing the behavior you are. Even
So the good news is it's working now. I know what I did but I don't know
why it worked so I'm hoping others can enlighten me based on what I did.
TL;DR - "turn it off/turn in on" for Max Timer Driven Thread Count fixed
performance. Max Timer Driven Thread Count was set to 20. I changed it to
30 -
Hi Joe,
Nothing is load balanced- it's all basic queues.
Mark,
I'm using NiFi 1.19.1.
nifi.performance.tracking.percentage sounds exactly what I might need. I'll
give that a shot.
Richard,
I hadn't looked at the rotating logs and/or cleared them out. I'll give
that a shot too.
Thank you all.
I had a similar sounding issue, although not in a Kube cluster. Nifi was
running in a docker container and the issue was the log rotation
interacting with the log file being mounted from the host. The mounted log
file was not deleted on rotation, meaning that once rotation was triggered
by log
Aaron,
What version of NiFi are you running? One thing that you can look into if
you’re running a pretty recent version, (though the user friendliness is not
great) is to update nifi.properties and set the
“nifi.performance.tracking.percentage” property from 0 to something like 5 or
10.
Hi Joe,
It's a pretty fixed size objects at a fixed interval- One 5mb-ish file, we
break down to individual rows.
I went so far as to create a "stress test" where I have a generateFlow(
creating a fix, 100k fille, in batches of 1000, every .1s) feeding right
into a putFile. I wanted to see the
1 - 100 of 10977 matches
Mail list logo
