You can use rdd.takeOrdered(1)(reverseOrdrering) reverseOrdering is you Ordering[T] instance where you define the ordering logic. This you have to pass in the method
On Thu, Apr 24, 2014 at 11:21 AM, Frank Austin Nothaft < fnoth...@berkeley.edu> wrote: > If you do this, you could simplify to: > > RDD.collect().last > > However, this has the problem of collecting all data to the driver. > > Is your data sorted? If so, you could reverse the sort and take the first. > Alternatively, a hackey implementation might involve a > mapPartitionsWithIndex that returns an empty iterator for all partitions > except for the last. For the last partition, you would filter all elements > except for the last element in your iterator. This should leave one > element, which is your last element. > > Frank Austin Nothaft > fnoth...@berkeley.edu > fnoth...@eecs.berkeley.edu > 202-340-0466 > > On Apr 23, 2014, at 10:44 PM, Adnan Yaqoob <nsyaq...@gmail.com> wrote: > > This function will return scala List, you can use List's last function to > get the last element. > > For example: > > RDD.take(RDD.count()).last > > > On Thu, Apr 24, 2014 at 10:28 AM, Sai Prasanna <ansaiprasa...@gmail.com>wrote: > >> Adnan, but RDD.take(RDD.count()) returns all the elements of the RDD. >> >> I want only to access the last element. >> >> >> On Thu, Apr 24, 2014 at 10:33 AM, Sai Prasanna >> <ansaiprasa...@gmail.com>wrote: >> >>> Oh ya, Thanks Adnan. >>> >>> >>> On Thu, Apr 24, 2014 at 10:30 AM, Adnan Yaqoob <nsyaq...@gmail.com>wrote: >>> >>>> You can use following code: >>>> >>>> RDD.take(RDD.count()) >>>> >>>> >>>> On Thu, Apr 24, 2014 at 9:51 AM, Sai Prasanna >>>> <ansaiprasa...@gmail.com>wrote: >>>> >>>>> Hi All, Some help ! >>>>> RDD.first or RDD.take(1) gives the first item, is there a straight >>>>> forward way to access the last element in a similar way ? >>>>> >>>>> I coudnt fine a tail/last method for RDD. !! >>>>> >>>> >>>> >>> >> > > -- Sourav Chandra Senior Software Engineer · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · sourav.chan...@livestream.com o: +91 80 4121 8723 m: +91 988 699 3746 skype: sourav.chandra Livestream "Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd Block, Koramangala Industrial Area, Bangalore 560034 www.livestream.com
