Also same thing can be done using rdd.top(1)(reverseOrdering)
On Thu, Apr 24, 2014 at 11:28 AM, Sourav Chandra < sourav.chan...@livestream.com> wrote: > You can use rdd.takeOrdered(1)(reverseOrdrering) > > reverseOrdering is you Ordering[T] instance where you define the ordering > logic. This you have to pass in the method > > > > On Thu, Apr 24, 2014 at 11:21 AM, Frank Austin Nothaft < > fnoth...@berkeley.edu> wrote: > >> If you do this, you could simplify to: >> >> RDD.collect().last >> >> However, this has the problem of collecting all data to the driver. >> >> Is your data sorted? If so, you could reverse the sort and take the >> first. Alternatively, a hackey implementation might involve a >> mapPartitionsWithIndex that returns an empty iterator for all partitions >> except for the last. For the last partition, you would filter all elements >> except for the last element in your iterator. This should leave one >> element, which is your last element. >> >> Frank Austin Nothaft >> fnoth...@berkeley.edu >> fnoth...@eecs.berkeley.edu >> 202-340-0466 >> >> On Apr 23, 2014, at 10:44 PM, Adnan Yaqoob <nsyaq...@gmail.com> wrote: >> >> This function will return scala List, you can use List's last function to >> get the last element. >> >> For example: >> >> RDD.take(RDD.count()).last >> >> >> On Thu, Apr 24, 2014 at 10:28 AM, Sai Prasanna >> <ansaiprasa...@gmail.com>wrote: >> >>> Adnan, but RDD.take(RDD.count()) returns all the elements of the RDD. >>> >>> I want only to access the last element. >>> >>> >>> On Thu, Apr 24, 2014 at 10:33 AM, Sai Prasanna >>> <ansaiprasa...@gmail.com>wrote: >>> >>>> Oh ya, Thanks Adnan. >>>> >>>> >>>> On Thu, Apr 24, 2014 at 10:30 AM, Adnan Yaqoob <nsyaq...@gmail.com>wrote: >>>> >>>>> You can use following code: >>>>> >>>>> RDD.take(RDD.count()) >>>>> >>>>> >>>>> On Thu, Apr 24, 2014 at 9:51 AM, Sai Prasanna <ansaiprasa...@gmail.com >>>>> > wrote: >>>>> >>>>>> Hi All, Some help ! >>>>>> RDD.first or RDD.take(1) gives the first item, is there a straight >>>>>> forward way to access the last element in a similar way ? >>>>>> >>>>>> I coudnt fine a tail/last method for RDD. !! >>>>>> >>>>> >>>>> >>>> >>> >> >> > > > -- > > Sourav Chandra > > Senior Software Engineer > > · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · > > sourav.chan...@livestream.com > > o: +91 80 4121 8723 > > m: +91 988 699 3746 > > skype: sourav.chandra > > Livestream > > "Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd > Block, Koramangala Industrial Area, > > Bangalore 560034 > > www.livestream.com > -- Sourav Chandra Senior Software Engineer · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · sourav.chan...@livestream.com o: +91 80 4121 8723 m: +91 988 699 3746 skype: sourav.chandra Livestream "Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd Block, Koramangala Industrial Area, Bangalore 560034 www.livestream.com
