Another option is merge partfiles after your app ends.
On 5 Jun 2015 20:37, Akhil Das ak...@sigmoidanalytics.com wrote:
you can simply do rdd.repartition(1).saveAsTextFile(...), it might not be
efficient if your output data is huge since one task will be doing the
whole writing.
Thanks
Best
Hi all
I'm running spark in a single local machine, no hadoop, just reading and
writing in local disk.
I need to have a single file as output of my calculation.
if I do rdd.saveAsTextFile(...) all runs ok but I get allot of files.
Since I need a single file I was considering to do something
Just repartition to 1 partition before writing.
On Fri, Jun 5, 2015 at 3:46 PM, marcos rebelo ole...@gmail.com wrote:
Hi all
I'm running spark in a single local machine, no hadoop, just reading and
writing in local disk.
I need to have a single file as output of my calculation.
if I do
you can simply do rdd.repartition(1).saveAsTextFile(...), it might not be
efficient if your output data is huge since one task will be doing the
whole writing.
Thanks
Best Regards
On Fri, Jun 5, 2015 at 3:46 PM, marcos rebelo ole...@gmail.com wrote:
Hi all
I'm running spark in a single local