In some given partition, it seems difficult to access the last element in another partition, but in my application I need do as that.Exactly how to do it ? Just by repartition /shuffle the rdd into one partition and get the specific "last" element ? Will this will change the previous order among the elements, and will it also not work ? Thanks very much in advance!
On Monday, December 7, 2015 11:32 AM, Zhiliang Zhu <zchl.j...@yahoo.com> wrote: On Monday, December 7, 2015 10:37 AM, DB Tsai <dbt...@dbtsai.com> wrote: Only beginning and ending part of data. The rest in the partition can be compared without shuffle. Would you help write a few pseudo-code about it...It seems that there is not shuffle related API , or repartition ? Thanks a lot in advance! ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Sincerely, DB Tsai ---------------------------------------------------------- Web: https://www.dbtsai.com PGP Key ID: 0xAF08DF8D On Sun, Dec 6, 2015 at 6:27 PM, Zhiliang Zhu <zchl.j...@yahoo.com> wrote: > > > > > On Saturday, December 5, 2015 3:00 PM, DB Tsai <dbt...@dbtsai.com> wrote: > > > This is tricky. You need to shuffle the ending and beginning elements > using mapPartitionWithIndex. > > > Does this mean that I need to shuffle the all elements in different > partitions into one partition, then compare them by way of any two adjacent > elements? > It seems good, if it is like that. > > One more issue, will it loss parallelism since there become only one > partition ... > > Thanks very much in advance! > > > > > > > Sincerely, > > DB Tsai > ---------------------------------------------------------- > Web: https://www.dbtsai.com > PGP Key ID: 0xAF08DF8D > > > On Fri, Dec 4, 2015 at 10:30 PM, Zhiliang Zhu <zchl.j...@yahoo.com> wrote: >> Hi All, >> >> I would like to compare any two adjacent elements in one given rdd, just >> as >> the single machine code part: >> >> int a[N] = {...}; >> for (int i=0; i < N - 1; ++i) { >> compareFun(a[i], a[i+1]); >> } >> ... >> >> mapPartitions may work for some situations, however, it could not compare >> elements in different partitions. >> foreach also seems not work. >> >> Thanks, >> Zhiliang > >> >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org