[
https://issues.apache.org/jira/browse/IGNITE-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16095996#comment-16095996
]
Yakov Zhdanov edited comment on IGNITE-5658 at 7/21/17 9:28 AM:
----------------------------------------------------------------
Tests has shown that batching per partition on client side does not bring any
significant improvement, but otherwise slows down the process a bit for low
load scenarios. Possible reasons are:
* increased number of requests
* more instantiations
* less entries per message (in order to occupy less heap) and as a consequence
more IO calls.
Solution would be to implement per stripe batching on processing server side.
This seems to be the another "putAll" problem and whatever solution would be
these two problems should be solved in the same way.
Probably, it makes sense to batch per stripe (spanning more than 1 partition).
E.g. for 4 servers topology this will end up in 16 stripes per node on client
instead of 256 partitions.
Bench results - 16 threads, allowOverwrite=false, 500_000_000 entries, 1 client
1 server
* master branch - 519 sec
* 5658 - 477 sec
was (Author: yzhdanov):
Tests has shown that batching per partition on client side does not bring any
significant improvement, but otherwise slows down the process a bit for low
load scenarios. Possible reasons are:
* increased number of requests
* more instantiations
* less entries per message (in order to occupy less heap) and as a consequence
more IO calls.
Solution would be to implement per stripe batching on processing server side.
This seems to be the another "putAll" problem and whatever solution would be
these two problems should be solved in the same way.
Probably, it makes sense to batch per stripe (spanning more than 1 partition).
E.g. for 4 servers topology this will end up in 16 stripes per node on client
instead of 256 partitions.
Bench results - 16 threads, allowOverwrite=false, 500_000_000 entries
* master branch - 519 sec
* 5658 - 477 sec
> Optimizations for data streamer
> -------------------------------
>
> Key: IGNITE-5658
> URL: https://issues.apache.org/jira/browse/IGNITE-5658
> Project: Ignite
> Issue Type: Improvement
> Reporter: Yakov Zhdanov
> Assignee: Yakov Zhdanov
> Labels: performance
> Fix For: 2.2
>
>
> Data streamer can be improved with this:
> -batch on per-partition basis and send batches to striped pool (for
> allowOverwirte=false)
> -New default - perNodeParallelOps should be twice CPU count of a remote node
> (if not set by user, if set then parallelOps for every node is the value
> provided by user)
> -wait stable topology error should be output as warn
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
