[
https://issues.apache.org/jira/browse/IGNITE-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16251176#comment-16251176
]
Alexander Belyak commented on IGNITE-6898:
------------------------------------------
We suggest data streamer as best mechanism to load large amount of data to grid
so I think it will be enough to optimize it first
You suggest to add some back pressure or load control on application level. OK,
sounds good.
> Datastreamers can lead to OOM on server side
> --------------------------------------------
>
> Key: IGNITE-6898
> URL: https://issues.apache.org/jira/browse/IGNITE-6898
> Project: Ignite
> Issue Type: Bug
> Security Level: Public(Viewable by anyone)
> Components: general
> Affects Versions: 2.1
> Reporter: Alexander Belyak
>
> If grid server node process many datastreamer in same time (from many
> clients, with many cache backups and persistence, i.e. if processing take
> some time) it can lead to OutOfMemoryError in server JVM. To fix we can:
> 1) specify buffer sized in bytes instead of entries
> 2) use pageMemory to store streamer buffers
> I get this problem on 16 server node grid with 45g heap each and 15 clients
> with 2 datastreamer each with this settings:
> autoFlushFrequency=0
> allowOverwrite=false
> perNodeParallelOperations=8
> perNodeBufferSize=10000
> Each client have 64g heap.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
