how to merge two dataframes

Hi folks, I have a need to "append" two dataframes -- I was hoping to use UnionAll but it seems that this operation treats the underlying dataframes as sequence of columns, rather than a map. In particular, my problem is that the columns in the two DFs are not in the same order --notice that my

Re: how to merge two dataframes

How about the following ? scala> df.registerTempTable("df") scala> df1.registerTempTable("df1") scala> sql("select customer_id, uri, browser, epoch from df union select customer_id, uri, browser, epoch from df1").show() +---+-+---+-+ |customer_id|

Re: how to merge two dataframes

Not a bad idea I suspect but doesn't help me. I dumbed down the repro to ask for help. In reality one of my dataframes is a cassandra DF. So cassDF.registerTempTable("df1") registers the temp table in a different SQL Context (new CassandraSQLContext(sc)). scala> sql("select customer_id, uri,

Re: how to merge two dataframes

I see - you were trying to union a non-Cassandra DF with Cassandra DF :-( On Fri, Oct 30, 2015 at 12:57 PM, Yana Kadiyska wrote: > Not a bad idea I suspect but doesn't help me. I dumbed down the repro to > ask for help. In reality one of my dataframes is a cassandra DF.

Re: how to merge two dataframes

l.com>" <yana.kadiy...@gmail.com<mailto:yana.kadiy...@gmail.com>> Date: Friday, October 30, 2015 at 3:57 PM To: Ted Yu <yuzhih...@gmail.com<mailto:yuzhih...@gmail.com>> Cc: "user@spark.apache.org<mailto:user@spark.apache.org>" <user@spark.