hi, is the table that you are trying to overwrite an external table or temporary table created in hivecontext?
Regards, Gourav Sengupta On Sat, Mar 5, 2016 at 3:01 PM, Dhaval Modi <dhavalmod...@gmail.com> wrote: > Hi Team, > > I am facing a issue while writing dataframe back to HIVE table. > > When using "SaveMode.Overwrite" option the table is getting dropped and > Spark is unable to recreate it thus throwing error. > > JIRA: https://issues.apache.org/jira/browse/SPARK-13699 > > > E.g. > tgtFinal.write.mode(SaveMode.Overwrite).saveAsTable("tgt_table") > > Error: > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > 16/03/05 13:57:26 INFO spark.SparkContext: Created broadcast 138 from run > at ThreadPoolExecutor.java:1145 > 16/03/05 13:57:26 INFO log.PerfLogger: <PERFLOG method=OrcGetSplits > from=org.apache.hadoop.hive.ql.io.orc.ReaderImpl> > *java.lang.RuntimeException: serious problem* > * at * > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1021) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1048) > at > org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:199) > at > org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239) > at > org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237) > at scala.Option.getOrElse(Option.scala:120) > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > Caused by: java.util.concurrent.ExecutionException: > java.io.FileNotFoundException: File does not exist: hdfs:// > sandbox.hortonworks.com:8020/apps/hive/warehouse/tgt_table > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:188) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:998) > ... 138 more > *Caused by: java.io.FileNotFoundException: File does not exist: > hdfs://sandbox.hortonworks.com:8020/apps/hive/warehouse/tgt_table > <http://sandbox.hortonworks.com:8020/apps/hive/warehouse/tgt_table>* > at > org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1122) > at > org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114) > ++++++++++++++++++++++++++++++++++++++++++++++++++ > > > Regards, > Dhaval Modi > dhavalmod...@gmail.com >
