Since no one else has answered...
I assume:
data.mapPartitions(_.toList.sortBy(...).toIterator)
would work, but I also suspect there's a better way.
On Fri, Oct 25, 2013 at 5:01 AM, Arun Kumar <[email protected]> wrote:
> Hi,
>
> I am trying to process some logs and the data is sorted(*almost*) by
> timestamp.
> If I do a full sort it takes a lot of time. Is there some way to sort more
> efficiently (like restricting sort to per partition).
>
> Thanks in advance
>
--
Nathan Kronenfeld
Senior Visualization Developer
Oculus Info Inc
2 Berkeley Street, Suite 600,
Toronto, Ontario M5A 4J5
Phone: +1-416-203-3003 x 238
Email: [email protected]
