The problem with using collect is that it will fail for large data sets, as
you'll attempt to copy the entire RDD to the memory of your driver program.
The following works (Scala syntax, but similar to Python):
scala> val i1 = input.distinct.groupByKey
scala> i1.foreach(println)
(1,CompactBuffer(beta, alpha, foo))
(3,CompactBuffer(foo))
(2,CompactBuffer(alpha, bar))
scala> val i2 = i1.map(tup => (tup._1, tup._2.size))
scala> i1.foreach(println)
(1,3)
(3,1)
(2,2)
The "i2" line passes a function that takes a tuple argument, then
constructs a new output tuple with the first element and the size of the
second (each CompactBuffer). An alternative pattern match syntax would be.
scala> val i2 = i1.map { case (key, buffer) => (key, buffer.size) }
This should work as long as none of the CompactBuffers are too large, which
could happen for extremely large data sets.
dean
Dean Wampler, Ph.D.
Author: Programming Scala, 2nd Edition
<http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
Typesafe <http://typesafe.com>
@deanwampler <http://twitter.com/deanwampler>
http://polyglotprogramming.com
On Mon, Apr 13, 2015 at 11:45 AM, Marco Shaw <[email protected]> wrote:
> **Learning the ropes**
>
> I'm trying to grasp the concept of using the pipeline in pySpark...
>
> Simplified example:
> >>>
> list=[(1,"alpha"),(1,"beta"),(1,"foo"),(1,"alpha"),(2,"alpha"),(2,"alpha"),(2,"bar"),(3,"foo")]
>
> Desired outcome:
> [(1,3),(2,2),(3,1)]
>
> Basically for each key, I want the number of unique values.
>
> I've tried different approaches, but am I really using Spark effectively?
> I wondered if I would do something like:
> >>> input=sc.parallelize(list)
> >>> input.groupByKey().collect()
>
> Then I wondered if I could do something like a foreach over each key
> value, and then map the actual values and reduce them. Pseudo-code:
>
> input.groupbykey()
> .keys
> .foreach(_.values
> .map(lambda x: x,1)
> .reducebykey(lambda a,b:a+b)
> .count()
> )
>
> I was somehow hoping that the key would get the current value of count,
> and thus be the count of the unique keys, which is exactly what I think I'm
> looking for.
>
> Am I way off base on how I could accomplish this?
>
> Marco
>