Re: Why is huge data shuffling in Spark when using union()/coalesce(1,false) on DataFrame?

2015-09-09 Thread Umesh Kacha
D); >> DataFrame updatedDataFrame = >> hiveContext.createDataFrame(updatedDsqlRDD,renamedSourceFrame.schema()); >> >> updatedDataFrame.write().mode(SaveMode.Append).format("orc").partitionBy("entity", >> "date").save("baseTable"); >&

Re: Why is huge data shuffling in Spark when using union()/coalesce(1,false) on DataFrame?

2015-09-08 Thread Richard Marscher
rceFrame.schema()); > > updatedDataFrame.write().mode(SaveMode.Append).format("orc").partitionBy("entity", > "date").save("baseTable"); > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.co

Why is huge data shuffling in Spark when using union()/coalesce(1,false) on DataFrame?

2015-09-04 Thread unk1102
t.1001560.n3.nabble.com/Why-is-huge-data-shuffling-in-Spark-when-using-union-coalesce-1-false-on-DataFrame-tp24581.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spa