You need to add a select clause to at least one dataframe to give them the same schema before you can union them (much like in SQL).
On Tue, May 5, 2015 at 3:24 AM, Wilhelm <niznik.pa...@gmail.com> wrote: > Hey there, > > 1.) I'm loading 2 avro files with that have slightly different schema > > df1 = sqlc.load(file1, "com.databricks.spark.avro") > df2 = sqlc.load(file2, "com.databricks.spark.avro") > > 2.) I want to unionAll them > > nfd = dfs1.unionAll(dfs2) > > 3.) Getting the following error > > --------------------------------------------------------------------------- > Py4JJavaError Traceback (most recent call last) > <ipython-input-190-a86d9adbea83> in <module>() > 17 > 18 > ---> 19 nfd = dfs1.unionAll(dfs2) > 20 > 21 > > /home/hadoop/spark/python/pyspark/sql/dataframe.pyc in unionAll(self, > other) > 669 This is equivalent to `UNION ALL` in SQL. > 670 """ > --> 671 return DataFrame(self._jdf.unionAll(other._jdf), > self.sql_ctx) > 672 > 673 def intersect(self, other): > > /home/hadoop/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py in > __call__(self, *args) > 536 answer = self.gateway_client.send_command(command) > 537 return_value = get_return_value(answer, > self.gateway_client, > --> 538 self.target_id, self.name) > 539 > 540 for temp_arg in temp_args: > > /home/hadoop/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py in > get_return_value(answer, gateway_client, target_id, name) > 298 raise Py4JJavaError( > 299 'An error occurred while calling {0}{1}{2}.\n'. > --> 300 format(target_id, '.', name), value) > 301 else: > 302 raise Py4JError( > > Py4JJavaError: An error occurred while calling o76196.unionAll. > : org.apache.spark.sql.AnalysisException: unresolved operator 'Union ; > at > > org.apache.spark.sql.catalyst.analysis.CheckAnalysis.failAnalysis(CheckAnalysis.scala:37) > at > > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$apply$3.apply(CheckAnalysis.scala:97) > at > > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$apply$3.apply(CheckAnalysis.scala:43) > at > org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:88) > at > > org.apache.spark.sql.catalyst.analysis.CheckAnalysis.apply(CheckAnalysis.scala:43) > at > > org.apache.spark.sql.SQLContext$QueryExecution.assertAnalyzed(SQLContext.scala:1069) > at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:133) > at > org.apache.spark.sql.DataFrame.logicalPlanToDataFrame(DataFrame.scala:157) > at org.apache.spark.sql.DataFrame.unionAll(DataFrame.scala:641) > at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231) > at > py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379) > at py4j.Gateway.invoke(Gateway.java:259) > at > py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133) > at py4j.commands.CallCommand.execute(CallCommand.java:79) > at py4j.GatewayConnection.run(GatewayConnection.java:207) > at java.lang.Thread.run(Thread.java:745) > --------------------------------------------------------------------------- > > 4.) Is it possible to automatically merge 2 DFs with different schemas like > that? Am I doing sth. wrong? > > Much appreciated! > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Two-DataFrames-with-different-schema-unionAll-issue-tp22765.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >