BTW you can also use rdd.partitions() to get a list of Partition objects and
see how many there are.
On September 4, 2014 at 5:18:30 PM, Matei Zaharia (matei.zaha...@gmail.com)
wrote:
Partitioners also work in local mode, the only question is how to see which
data fell into each partition, since most RDD operations hide the fact that
it's partitioned. You can do rdd.glom().collect() -- the glom() operation turns
an RDD of elements of type T into an RDD of List<T>, with a separate list for
each partition. You should get back 26 lists, with the right elements in each
one.
Matei
On September 4, 2014 at 4:58:19 PM, Steve Lewis (lordjoe2...@gmail.com) wrote:
Assume I define a partitioner like
/**
* partition on the first letter
*/
public class PartitionByStart extends Partitioner {
@Override public int numPartitions() {
return 26;
}
@Override public int getPartition(final Object key) {
String s = (String)key;
if(s.length() == 0)
throw new IllegalStateException("problem"); // ToDo change
int ret = s.charAt(0) - 'A';
ret = Math.min(25,ret) ;
ret = Math.max(0,ret);
return 25 - ret;
}
}
how, short or running on a large cluster can I test that code which might
look like (Unrolling all the chained methods)
ones = ones.partitionBy(new PartitionByStart());
JavaPairRDD<String, Integer> sorted = ones.sortByKey();
JavaRDD<WordNumber> answer = sorted.mapPartitions(new
WordCountFlatMapFinction());
partitions properly - in other words on a local instance how would
partitoning work and what do I expect to see in switching from one partition to
another as the code runs?
On Sat, Aug 30, 2014 at 10:30 AM, Matei Zaharia <matei.zaha...@gmail.com> wrote:
In 1.1, you'll be able to get all of these properties using sortByKey, and then
mapPartitions on top to iterate through the key-value pairs. Unfortunately
sortByKey does not let you control the Partitioner, but it's fairly easy to
write your own version that does if this is important.
In previous versions, the values for each key had to fit in memory (though we
could have data on disk across keys), and this is still true for groupByKey,
cogroup and join. Those restrictions will hopefully go away in a later release.
But sortByKey + mapPartitions lets you just iterate through the key-value pairs
without worrying about this.
Matei
On August 30, 2014 at 9:04:37 AM, Steve Lewis (lordjoe2...@gmail.com) wrote:
When programming in Hadoop it is possible to guarantee
1) All keys sent to a specific partition will be handled by the same machine
(thread)
2) All keys received by a specific machine (thread) will be received in sorted
order
3) These conditions will hold even if the values associated with a specific key
are too large enough to fit in memory.
In my Hadoop code I use all of these conditions - specifically with my larger
data sets the size of data I wish to group exceeds the available memory.
I think I understand the operation of groupby but my understanding is that this
requires that the results for a single key, and perhaps all keys fit on a
single machine.
Is there away to perform like Hadoop ad not require that an entire group fir in
memory?
--
Steven M. Lewis PhD
4221 105th Ave NE
Kirkland, WA 98033
206-384-1340 (cell)
Skype lordjoe_com
