[ https://issues.apache.org/jira/browse/CASSANDRA-6665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Christian Rolf updated CASSANDRA-6665: -------------------------------------- Attachment: batchWrite.txt Never submitted a patch here before, let me know if the format's bad. > Batching in CqlRecordWriter > --------------------------- > > Key: CASSANDRA-6665 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6665 > Project: Cassandra > Issue Type: Improvement > Components: Hadoop > Environment: Cluster of 12 nodes, each node with 256-384 vnodes. RPC > threads capped at 2048. > Reporter: Christian Rolf > Priority: Minor > Attachments: batchWrite.txt > > > We're writing from Pig map tasks, about 20 million records of one integer > each. > For the case of 12 nodes, with 256-384 vnodes per node, we get around 4000 > threads per mapper. This obviously overloads the nodes, since the number of > RPC threads are capped, and the write fails. > Also, each transfer is only in the order of a few bytes of payload. Clearly > batching is a good solution. -- This message was sent by Atlassian JIRA (v6.1.5#6160)