Shawn,
have a look here:
https://nifi.apache.org/docs/nifi-docs/rest-api/index.html
Under "Versions" I suppose it could be:
[cid:image001.jpg@01D6F61A.7F265710]
Kind Regards
Von: Shawn Weeks
Gesendet: Donnerstag, 28. Januar 2021 18:32
An: users@nifi.apache.org
Betreff: How to Stop Version
Hey
I have an issue with nested PG where the root PG continuously is showing it
has local changes but it has not.
I have now found why, and how you can reproduce the bug
1. Create an empty Process Group (PG-Root)
2. Add version control to the PG-Root
3. Add an empty PG (PG-Sub) in PG-Root
4. Add
Hey
I have found the reason to why it is not working.
to reproduce this issue:
1. Create an empty Process Group (PG-Root)
2. Add version control to the PG-Root
3. Add an empty PG (PG-Sub) in PG-Root
4. Add version control to PG-Sub
5. Update PG-Root
6. Check that both PG-Root and PG-Sub are
Hey Vijay,
I’ve seen a few people lately running into issues with InvokeHTTP. The common
thread for all of them is that they are hitting servers that are using HTTP 2.
Reading threads from OkHttp (the underlying HTTP library that we use), I see
that a lot of people are running into issues with
The likely relevant thread is here
"Timer-Driven Process Thread-9" #70 prio=5 os_prio=31 cpu=12025.30ms
elapsed=4403.88s tid=0x7fe44f16b000 nid=0xe103 in Object.wait()
[0x7fed4000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(java.base@11.0.5/Native
Zilvinas,
Also I figured I would throw out links to a few videos that I’ve posted on
YouTube regarding NiFi Anti-Patterns, as they may be helpful for you.
Part 4: Scheduling / Concurrent Tasks / Thread Pools. Discusses how to
determine the appropriate size of the Timer-Driven Thread Pool and
Thanks Mark. I see what you're saying. I'll try out your ideas tomorrow and
see how it goes. Thanks a lot!
On Thu, Jan 28, 2021 at 8:55 PM Mark Payne wrote:
> … And now that I’m reading more about how your flow is setup (I didn’t see
> that the source of the data was SQS at first)… I would not
Joe,
Absolutely. I can provide configuration of every single processor. Could
you point me to anything I can read through to see how actual content can
be cached in memory? Perhaps a link to github. If there's a condition where
processors can avoid reading from local disk to fetch actual content
… And now that I’m reading more about how your flow is setup (I didn’t see that
the source of the data was SQS at first)… I would not recommend using
load-balanced connections at all.
Instead, you should be able to rely solely on backpressure to allow for evenly
distributing the data. If you
Zilvinas,
That is accurate - when a connection is load balanced, the data is pushed to a
particular node based on the selected algorithm. It is not continually
rebalanced.
So for a flow like this, my recommendation would be:
1) Set the backpressure threshold from FetchS3Object -> PublishKafka
Saltys
It can be possible because those things can still be cached. The way this
thing really works at scale can be quite awesome actually.
However, definitely want to help you understand what is happening but the
pictures alone dont cut it. We appreciate you have sensitivities/stuff you
have
We're still on an old version of Kafka that's why we're still using old
processors.
File sizes vary .. Generally they are all within +-100mb range before they
are uncompressed. There can be some small files but they are not a
majority. From logging I can see that all hosts are processing files of
Not saying this is the issue, but is your Kafka cluster using Kafka 0.11?
Looking at the screenshot, you're using the Kafka processors from the 0.11
bundle, you might want to look at the processors for Kafka 2.x instead.
Are your files more or less evenly distributed in terms of sizes?
I suppose
You're saying it wouldn't kick in because of low rates of files? The
interesting bit for us is that we don't process a lot of files but the
files themselves are quite large. And in our case some nodes keep getting
behind and it seems that load balancer is not kicking in to rebalance the
cluster.
My other issue is that the balancing is not rebalancing the queue? Perhaps
I misunderstand how balancing should work and it only balances round robin
new incoming files? I can easily manually rebalance by disabling balancing
and enabling it again but after a while it gets back to the same
Hi Joe,
Yes it is the same issue. We have used your advice and reduced the amount
of threads on our large processors: fetch/compress/publish to a minimum and
then increased gradually to 4 until the processing rate became acceptable
(about 2000 files per 5 min). This is a cluster of 25 nodes of 36
Maksym,
Very difficult to look at these brief/limited details and offer meaningful
responses. In the example you show about the data volumes are so small
that I dont even know that load balancing would kick in. But yes generally
speaking the combination of load balancing and back pressure
I'm assuming also this is the same thing Maksym was asking about
yesterday. Let's try to keep the thread together as this gets discussed.
On Thu, Jan 28, 2021 at 1:10 PM Pierre Villard
wrote:
> Hi Zilvinas,
>
> I'm afraid we would need more details to help you out here.
>
> My first question
Hi Zilvinas,
I'm afraid we would need more details to help you out here.
My first question by quickly looking at the graph would be: there is a host
(green line) where the number of queued flow files is more or less
constantly growing. Where in the flow are the flow files accumulating for
this
How do I stop version control for a process group from the rest api? On older
versions of NiFi importing a template that contains a registry will actually
prevent the UI from loading and I'm having trouble figure out what the
equivalent rest call for "stop version control" is.
Thanks
Shawn
I am running NiFi 1.12.1 without NiFi Registry and have a connection that
is configured to *Round Robin *flow files. After some time I see some nodes
performing worse than the other and the queue is piling up on 1-2 nodes.
[image: niifi-queue.jpg]
The question I have is how rebalancing actually
Hello!
I have the following issue with configuring ListFile and the following folder
structure organized year/month/day/hour:
/abc/def/2021/01/15/13/Test
I need to recursively retrieve files from /abc/def/2021/01/15/13 but not from
/abc/def/2021/01/15/13/Test and have the following parameters
22 matches
Mail list logo
