What i observe is, this way of computing is very inefficient. It returns all the elements of the RDD to a List which takes considerable amount of time. Then it calculates the last element.
I have a file of size 3 GB in which i ran a lot of aggregate operations which dint took the time that this take(RDD.count) took. Is there an efficient way ? My guess is there should be one, since its a basic operation. On Thu, Apr 24, 2014 at 11:14 AM, Adnan Yaqoob <nsyaq...@gmail.com> wrote: > This function will return scala List, you can use List's last function to > get the last element. > > For example: > > RDD.take(RDD.count()).last > > > On Thu, Apr 24, 2014 at 10:28 AM, Sai Prasanna <ansaiprasa...@gmail.com>wrote: > >> Adnan, but RDD.take(RDD.count()) returns all the elements of the RDD. >> >> I want only to access the last element. >> >> >> On Thu, Apr 24, 2014 at 10:33 AM, Sai Prasanna >> <ansaiprasa...@gmail.com>wrote: >> >>> Oh ya, Thanks Adnan. >>> >>> >>> On Thu, Apr 24, 2014 at 10:30 AM, Adnan Yaqoob <nsyaq...@gmail.com>wrote: >>> >>>> You can use following code: >>>> >>>> RDD.take(RDD.count()) >>>> >>>> >>>> On Thu, Apr 24, 2014 at 9:51 AM, Sai Prasanna >>>> <ansaiprasa...@gmail.com>wrote: >>>> >>>>> Hi All, Some help ! >>>>> RDD.first or RDD.take(1) gives the first item, is there a straight >>>>> forward way to access the last element in a similar way ? >>>>> >>>>> I coudnt fine a tail/last method for RDD. !! >>>>> >>>> >>>> >>> >> >
