[
https://issues.apache.org/jira/browse/CASSANDRA-1473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13083826#comment-13083826
]
Stu Hood commented on CASSANDRA-1473:
-------------------------------------
bq. I don't think we can do this, without knowing the total number of keys as
well.
Would it be possible to generate the partitions during the job setup, and pass
them in as config to the h.m.Partitioner instances?
I know that Hadoop's terasort benchmark performs ordered partitioning by using
a sampling phase before the actual job, so that might be a good place to start
looking.
> Implement a Cassandra aware Hadoop mapreduce.Partitioner
> --------------------------------------------------------
>
> Key: CASSANDRA-1473
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1473
> Project: Cassandra
> Issue Type: Improvement
> Components: Hadoop
> Reporter: Stu Hood
> Assignee: Patricio Echague
> Fix For: 1.0
>
>
> When using a IPartitioner that does not sort data in byte order
> (RandomPartitioner for example) with Cassandra's Hadoop integration, Hadoop
> is unaware of the output order of the data.
> We can make Hadoop aware of the proper order of the output data by
> implementing Hadoop's mapreduce.Partitioner interface: then Hadoop will
> handle sorting all of the data according to Cassandra's IPartitioner, and the
> writing clients will be able to connect to smaller numbers of Cassandra nodes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
