[
https://issues.apache.org/jira/browse/CASSANDRA-6665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis updated CASSANDRA-6665:
--------------------------------------
Reproduced In: 2.0.4, 2.0.3, 2.0.2 (was: 2.0.2, 2.0.3, 2.0.4)
Fix Version/s: 2.0.6
> Batching in CqlRecordWriter
> ---------------------------
>
> Key: CASSANDRA-6665
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6665
> Project: Cassandra
> Issue Type: Improvement
> Components: Hadoop
> Environment: Cluster of 12 nodes, each node with 256-384 vnodes. RPC
> threads capped at 2048.
> Reporter: Christian Rolf
> Assignee: Christian Rolf
> Priority: Minor
> Fix For: 2.0.6
>
> Attachments: batchWrite.txt, inet_patch.diff
>
>
> We're writing from Pig map tasks, about 20 million records of one integer
> each.
> For the case of 12 nodes, with 256-384 vnodes per node, we get around 4000
> threads per mapper. This obviously overloads the nodes, since the number of
> RPC threads are capped, and the write fails.
> Also, each transfer is only in the order of a few bytes of payload. Clearly
> batching is a good solution.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
