Alex Liu created SPARK-5624: ------------------------------- Summary: Can't find new column Key: SPARK-5624 URL: https://issues.apache.org/jira/browse/SPARK-5624 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 1.1.1 Reporter: Alex Liu Priority: Minor
The following test fails {code} 0: jdbc:hive2://localhost:10000> DROP TABLE IF EXISTS alter_test_table; +---------+ | Result | +---------+ +---------+ No rows selected (0.175 seconds) 0: jdbc:hive2://localhost:10000> DROP TABLE IF EXISTS alter_test_table_ctas; +---------+ | Result | +---------+ +---------+ No rows selected (0.155 seconds) 0: jdbc:hive2://localhost:10000> DROP TABLE IF EXISTS alter_test_table_renamed; +---------+ | Result | +---------+ +---------+ No rows selected (0.162 seconds) 0: jdbc:hive2://localhost:10000> CREATE TABLE alter_test_table (foo INT, bar STRING) COMMENT 'table to test DDL ops' PARTITIONED BY (ds STRING) STORED AS TEXTFILE; +---------+ | result | +---------+ +---------+ No rows selected (0.247 seconds) 0: jdbc:hive2://localhost:10000> LOAD DATA LOCAL INPATH '/Users/alex/project/automaton/resources/tests/data/files/kv1.txt' OVERWRITE INTO TABLE alter_test_table PARTITION (ds='2008-08-08'); +---------+ | result | +---------+ +---------+ No rows selected (0.367 seconds) 0: jdbc:hive2://localhost:10000> CREATE TABLE alter_test_table_ctas as SELECT * FROM alter_test_table; +------+------+-----+ | foo | bar | ds | +------+------+-----+ +------+------+-----+ No rows selected (0.641 seconds) 0: jdbc:hive2://localhost:10000> ALTER TABLE alter_test_table ADD COLUMNS (new_col1 INT); +---------+ | result | +---------+ +---------+ No rows selected (0.226 seconds) 0: jdbc:hive2://localhost:10000> INSERT OVERWRITE TABLE alter_test_table PARTITION (ds='2008-08-15') SELECT foo, bar, 333333333 FROM alter_test_table_ctas WHERE ds='2008-08-08'; +------+------+------+ | foo | bar | c_2 | +------+------+------+ +------+------+------+ No rows selected (0.522 seconds) 0: jdbc:hive2://localhost:10000> select * from alter_test_table ; Error: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 35.0 failed 4 times, most recent failure: Lost task 0.3 in stage 35.0 (TID 66, 127.0.0.1): java.lang.RuntimeException: cannot find field new_col1 from [0:foo, 1:bar] org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:367) org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector.getStructFieldRef(LazySimpleStructObjectInspector.java:168) org.apache.spark.sql.hive.HadoopTableReader$$anonfun$9.apply(TableReader.scala:275) org.apache.spark.sql.hive.HadoopTableReader$$anonfun$9.apply(TableReader.scala:275) scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) scala.collection.TraversableLike$class.map(TraversableLike.scala:244) scala.collection.AbstractTraversable.map(Traversable.scala:105) org.apache.spark.sql.hive.HadoopTableReader$.fillObject(TableReader.scala:275) org.apache.spark.sql.hive.HadoopTableReader$$anonfun$3$$anonfun$apply$1.apply(TableReader.scala:193) org.apache.spark.sql.hive.HadoopTableReader$$anonfun$3$$anonfun$apply$1.apply(TableReader.scala:187) org.apache.spark.rdd.RDD$$anonfun$13.apply(RDD.scala:596) org.apache.spark.rdd.RDD$$anonfun$13.apply(RDD.scala:596) org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262) org.apache.spark.rdd.RDD.iterator(RDD.scala:229) org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:87) org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262) org.apache.spark.rdd.RDD.iterator(RDD.scala:229) org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31) org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262) org.apache.spark.rdd.RDD.iterator(RDD.scala:229) org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62) org.apache.spark.scheduler.Task.run(Task.scala:54) org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Driver stacktrace: (state=,code=0) {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org