Dear all,
Thank you very much for all of the detailed responses. About JVM heap space
recommendation, I am aware that it is possible to optimize 64GB heap space
for JVM, but it may cause lots of pauses in some cases. Anyway, my question
was not about how much memory space is required for Nifi JVM
The validity of that advice depends on a lot of factors. G1 changed the
game a bit for pause times for sure but you can still see larger pause
times than acceptable for some cases. In any event I agree that we should
be more careful with how we describe heap usage.
Thanks
Joe
On Oct 14, 2016 7:
Yeah, I spent a bit of time this morning before posting looking for a
magic 8-10Gb advisory and generally for GC gotchas related to larger
heap sizes in the 64-bit world, but couldn't find any. We're using 12Gb
right now for NiFi and haven't noticed any trouble. We vaguely conceive
of increasin
We actually use heap sizes from 32 to 64Gb for ours but our volumes and graphs
are both extremely large. Although I believe the smaller heap sizes were a
limitation of the garbage collection in Java 7. We also moved to ssd drives,
which did help through put quite a bit. Our systems were actually
Russ,
You can definitely find a lot of material on the Internet about Java heap
sizes, types of garbage collectors, application usage patterns. By all
means please do experiment with different sizes appropriate for your case.
We're not saying NiFi itself has any problem with large heaps.
Thanks
Ali,
"not recommended to dedicate more than 8-10 GM to JVM heap space" by
whom? Do you have links/references establishing this? I couldn't find
anyone saying this or why.
Russ
On 10/13/2016 05:47 PM, Ali Nazemian wrote:
Hi,
I have another question regarding the hardware recommendation. As
I'd also add to Mark's great reply that another good use of RAM beyond the
HEAP and disk caching and avoiding swapping is that you can do things like
off-heap native storage of things like reference datasets that can wired
into NiFi flows for high speed enrichment where you can even do hot
swapping
Hi Ali,
Typically, we see people using a 4-8 GB heap with NiFi. 8 GB is pretty typical
for a flow that is expected to have
pretty high throughput in terms of the number of FlowFiles, or a large number
of processors. However, one thing
that you will want to consider in terms of RAM is disk cachin
Hi,
I have another question regarding the hardware recommendation. As far as I
found out, Nifi uses on-heap memory currently, and it will not try to load
the whole object in memory. From the garbage collection perspective, it is
not recommended to dedicate more than 8-10 GB to JVM heap space. In t
Thank you very much.
I would be more than happy to provide some benchmark results after the
implementation.
Sincerely yours,
Ali
On Thu, Oct 13, 2016 at 11:32 PM, Joe Witt wrote:
> Ali,
>
> I agree with your assumption. It would be great to test that out and
> provide some numbers but intuitive
Ali,
I agree with your assumption. It would be great to test that out and
provide some numbers but intuitively I agree.
I could envision certain scatter/gather data flows that could challenge
that sequential access assumption but honestly with how awesome disk
caching is in Linux these days in t
Dear Joe,
Thank you very much. That was a really great explanation.
I investigated the Nifi architecture, and it seems that most of the
read/write operations for flow file repo and provenance repo are random.
However, for content repo most of the read/write operations are sequential.
Let's say cos
Ali,
You have a lot of nice resources to work with there. I'd recommend the
series of RAID-1 configuration personally provided you keep in mind this
means you can only lose a single disk for any one partition. As long as
they're being monitored and would be quickly replaced this in practice
work
Dear Nifi Users/ developers,
Hi,
I was wondering is there any benchmark about the question that is it better
to dedicate disk control to Nifi or using RAID for this purpose? For
example, which of these scenarios is recommended from the performance point
of view?
Scenario 1:
24 disk in total
2 disk
14 matches
Mail list logo
