Hi Sujit,
It is wonderful for you!I must show my sincere appreciation towards your kind
help.
Thank you very much!Best Regards,Zhiliang
On Wednesday, September 23, 2015 10:15 PM, Sujit Pal
<sujitatgt...@gmail.com> wrote:
Hi Zhiliang,
How about doing something like this?
val rdd3 = rdd1.zip(rdd2).map(p => p._1.zip(p._2).map(z => z._1 - z._2))
The first zip will join the two RDDs and produce an RDD of (Array[Float],
Array[Float]) pairs. On each pair, we zip the two Array[Float] components
together to form an Array[(Float, Float)] and then we subtract the first
element from the second in the inner map (the inner map is a Scala map not a
Spark one).
I tried this out on a notebook:
val rdd1 = sc.parallelize(List(Array(1.0, 2.0, 3.0), Array(4.0, 5.0, 6.0),
Array(7.0, 8.0, 9.0)))val rdd2 = sc.parallelize(List(Array(1.0, 4.0, 3.0),
Array(4.0, 10.0, 6.0), Array(7.0, 16.0, 9.0)))val rdd3 = rdd1.zip(rdd2).map(p
=> p._1.zip(p._2).map(z => z._1 - z._2))rdd3.collect()
gives me:res0: Array[Array[Double]] = Array(Array(0.0, -2.0, 0.0), Array(0.0,
-5.0, 0.0), Array(0.0, -8.0, 0.0))
-sujit
On Wed, Sep 23, 2015 at 12:23 AM, Zhiliang Zhu <zchl.j...@yahoo.com> wrote:
there is matrix add API, might map rdd2 each row element to be negative , then
make rdd1 and rdd2 and call add ?
Or some more ways ...
On Wednesday, September 23, 2015 3:11 PM, Zhiliang Zhu
<zchl.j...@yahoo.com> wrote:
Hi All,
There are two RDDs : RDD<Array<float>> rdd1, and RDD<Array<float>> rdd2,that
is to say, rdd1 and rdd2 are similar with DataFrame, or Matrix with same row
number and column number.
I would like to get RDD<Array<float>> rdd3, each element in rdd3 is the
subtract between rdd1 and rdd2 of thesame position, which is similar Matrix
subtract:rdd3<i, j> = rdd1<i, j> - rdd2<i, j> ...
It seemed very difficult to operate this kinds of matrix arithmetic, even is
about add, subtract, multiple , diff etc...
I shall appreciate your help very much~~Zhiliang
