Re: Writing to a single file from multiple executors

2015-03-12 Thread Tathagata Das
t; I way to mitigate the issue is to repartition & have a single writer task, >> but as my data is huge that is not a feasible option. >> >> Any suggestions welcomed. >> >> Regards, >> Sam >> >> >> >> -- >> View this message

Re: Writing to a single file from multiple executors

2015-03-12 Thread Maiti, Samya
n context: http://apache-spark-user-list.1001560.n3.nabble.com/Writing-to-a-single-file-from-multiple-executors-tp22003.html Sent from the Apache Spark User List mailing list archive at Nabble.com<http://Nabble.com>. - To unsu

Re: Writing to a single file from multiple executors

2015-03-11 Thread Tathagata Das
> > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Writing-to-a-single-file-from-multiple-executors-tp22003.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > -

Writing to a single file from multiple executors

2015-03-11 Thread SamyaMaiti
have a single writer task, but as my data is huge that is not a feasible option. Any suggestions welcomed. Regards, Sam -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Writing-to-a-single-file-from-multiple-executors-tp22003.html Sent from the Apache Spark