Repository: hive Updated Branches: refs/heads/master 068d007b8 -> 5ced7bf92
HIVE-19577: CREATE TEMPORARY TABLE LIKE and INSERT generate output format mismatch errors (Steve Yeom, reviewed by Jason Dere) Project: http://git-wip-us.apache.org/repos/asf/hive/repo Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/5ced7bf9 Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/5ced7bf9 Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/5ced7bf9 Branch: refs/heads/master Commit: 5ced7bf923c4ec4477ebfd68ff50aa2047be4eae Parents: 068d007 Author: Jason Dere <jd...@hortonworks.com> Authored: Tue May 29 10:58:24 2018 -0700 Committer: Jason Dere <jd...@hortonworks.com> Committed: Tue May 29 10:58:24 2018 -0700 ---------------------------------------------------------------------- data/files/students.txt | 199 +++++++++++++++++++ .../test/resources/testconfiguration.properties | 3 +- .../hadoop/hive/ql/parse/SemanticAnalyzer.java | 12 +- ql/src/test/queries/clientpositive/cttl.q | 29 +++ .../test/results/clientpositive/llap/cttl.q.out | 78 ++++++++ 5 files changed, 315 insertions(+), 6 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/hive/blob/5ced7bf9/data/files/students.txt ---------------------------------------------------------------------- diff --git a/data/files/students.txt b/data/files/students.txt new file mode 100644 index 0000000..f2d8a2d --- /dev/null +++ b/data/files/students.txt @@ -0,0 +1,199 @@ +ulysses thompson 64 1.90 +katie carson 25 3.65 +luke king 65 0.73 +holly davidson 57 2.43 +fred miller 55 3.77 +holly white 43 0.24 +luke steinbeck 51 1.14 +nick underhill 31 2.46 +holly davidson 59 1.26 +calvin brown 56 0.72 +rachel robinson 62 2.25 +tom carson 35 0.56 +tom johnson 72 0.99 +irene garcia 54 1.06 +oscar nixon 39 3.60 +holly allen 32 2.58 +oscar hernandez 19 0.05 +alice ichabod 65 2.25 +wendy thompson 30 2.39 +priscilla hernandez 73 0.23 +gabriella van buren 68 1.32 +yuri thompson 42 3.65 +yuri laertes 60 1.16 +sarah young 23 2.76 +zach white 32 0.20 +nick van buren 68 1.75 +xavier underhill 41 1.51 +bob ichabod 56 2.81 +zach steinbeck 61 2.22 +alice garcia 42 2.03 +jessica king 29 3.61 +calvin nixon 37 0.30 +fred polk 66 3.69 +bob zipper 40 0.28 +alice young 75 0.31 +nick underhill 37 1.65 +mike white 57 0.69 +calvin ovid 41 3.02 +fred steinbeck 47 3.57 +sarah ovid 65 0.00 +wendy nixon 63 0.62 +gabriella zipper 77 1.51 +david king 40 1.99 +jessica white 30 3.82 +alice robinson 37 3.69 +zach nixon 74 2.75 +irene davidson 27 1.22 +priscilla xylophone 43 1.60 +oscar zipper 25 2.43 +fred falkner 38 2.23 +ulysses polk 58 0.01 +katie hernandez 47 3.80 +zach steinbeck 55 0.68 +fred laertes 69 3.62 +quinn laertes 70 3.66 +nick garcia 50 0.12 +oscar young 55 2.22 +bob underhill 47 0.24 +calvin young 77 1.60 +mike allen 65 2.95 +david young 77 0.26 +oscar garcia 69 1.59 +ulysses ichabod 26 0.95 +wendy laertes 76 1.13 +sarah laertes 20 0.24 +zach ichabod 60 1.60 +tom robinson 62 0.78 +zach steinbeck 69 1.01 +quinn garcia 57 0.98 +yuri van buren 32 1.97 +luke carson 39 0.76 +calvin ovid 73 0.82 +luke ellison 27 0.56 +oscar zipper 50 1.31 +fred steinbeck 52 3.14 +katie xylophone 76 1.38 +luke king 54 2.30 +ethan white 72 1.43 +yuri ovid 37 3.64 +jessica garcia 54 1.08 +luke young 29 0.80 +mike miller 39 3.35 +fred hernandez 63 0.17 +priscilla hernandez 52 0.35 +ethan garcia 43 1.70 +quinn hernandez 25 2.58 +calvin nixon 33 1.01 +yuri xylophone 47 1.36 +ulysses steinbeck 63 1.05 +jessica nixon 25 2.13 +bob johnson 53 3.31 +jessica ichabod 56 2.21 +zach miller 63 3.87 +priscilla white 66 2.82 +ulysses allen 21 1.68 +katie falkner 47 1.49 +tom king 51 1.91 +bob laertes 60 3.33 +luke nixon 27 3.54 +quinn johnson 42 2.24 +wendy quirinius 71 0.10 +victor polk 55 3.63 +rachel robinson 32 1.11 +sarah king 57 1.37 +victor young 38 1.72 +priscilla steinbeck 38 2.11 +fred brown 19 2.72 +xavier underhill 55 3.56 +irene ovid 67 3.80 +calvin brown 37 2.22 +katie thompson 20 3.27 +katie carson 66 3.55 +tom miller 57 2.83 +rachel brown 56 0.74 +holly johnson 38 2.51 +irene steinbeck 29 1.97 +wendy falkner 37 0.14 +ethan white 29 3.62 +bob underhill 26 1.10 +jessica king 64 0.69 +luke steinbeck 19 1.16 +luke laertes 70 3.58 +rachel polk 74 0.92 +calvin xylophone 52 0.58 +luke white 57 3.86 +calvin van buren 52 3.13 +holly quirinius 59 1.70 +mike brown 44 1.93 +yuri ichabod 61 0.70 +ulysses miller 56 3.53 +victor hernandez 64 2.52 +oscar young 34 0.34 +luke ovid 36 3.17 +quinn ellison 50 1.13 +quinn xylophone 72 2.07 +nick underhill 48 0.15 +rachel miller 23 3.38 +mike van buren 68 1.74 +zach van buren 38 0.34 +irene zipper 32 0.54 +sarah garcia 31 3.87 +rachel van buren 56 0.35 +fred davidson 69 1.57 +nick hernandez 19 2.11 +irene polk 40 3.89 +katie young 26 2.88 +priscilla ovid 49 3.28 +jessica hernandez 39 3.13 +yuri allen 29 3.51 +victor garcia 66 3.45 +zach johnson 77 0.95 +yuri zipper 48 3.44 +alice falkner 28 3.72 +gabriella allen 58 3.61 +bob nixon 34 3.34 +bob white 67 2.93 +holly steinbeck 57 1.81 +wendy van buren 40 1.09 +calvin brown 61 2.08 +irene young 25 2.66 +holly van buren 40 2.37 +katie underhill 30 0.63 +quinn hernandez 73 0.31 +fred nixon 53 1.76 +luke ellison 59 1.10 +quinn nixon 24 0.30 +ethan underhill 68 2.25 +fred underhill 28 3.88 +jessica brown 59 3.66 +katie falkner 49 3.96 +calvin ellison 27 2.23 +zach carson 59 0.46 +priscilla polk 47 2.99 +rachel zipper 49 3.26 +holly king 73 1.23 +zach carson 64 2.60 +fred xylophone 61 3.15 +gabriella miller 43 1.73 +david laertes 56 3.43 +tom garcia 63 2.78 +ethan king 66 3.13 +david hernandez 26 2.52 +wendy ichabod 57 2.81 +alice young 69 0.25 +tom xylophone 50 2.78 +ulysses carson 62 2.26 +nick garcia 43 2.23 +gabriella ellison 33 1.18 +ethan miller 28 2.15 +tom zipper 19 2.56 +wendy white 19 1.12 +luke ovid 31 1.68 +wendy xylophone 75 2.58 +quinn garcia 22 3.65 +holly ellison 68 0.26 +yuri hernandez 75 2.50 +tom underhill 71 2.68 +ulysses king 31 1.76 +fred thompson 46 1.55 http://git-wip-us.apache.org/repos/asf/hive/blob/5ced7bf9/itests/src/test/resources/testconfiguration.properties ---------------------------------------------------------------------- diff --git a/itests/src/test/resources/testconfiguration.properties b/itests/src/test/resources/testconfiguration.properties index f7def35..6a70a4a 100644 --- a/itests/src/test/resources/testconfiguration.properties +++ b/itests/src/test/resources/testconfiguration.properties @@ -429,7 +429,8 @@ minillap.query.files=acid_bucket_pruning.q,\ unionDistinct_1.q,\ load_fs2.q,\ llap_stats.q,\ - multi_count_distinct_null.q + multi_count_distinct_null.q,\ + cttl.q minillaplocal.query.files=\ dp_counter_non_mm.q,\ http://git-wip-us.apache.org/repos/asf/hive/blob/5ced7bf9/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java ---------------------------------------------------------------------- diff --git a/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java b/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java index 863ab1b..d5ed581 100644 --- a/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java +++ b/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java @@ -12777,7 +12777,8 @@ public class SemanticAnalyzer extends BaseSemanticAnalyzer { */ private Map<String, String> addDefaultProperties( Map<String, String> tblProp, boolean isExt, StorageFormat storageFormat, - String qualifiedTableName, List<Order> sortCols, boolean isMaterialization) { + String qualifiedTableName, List<Order> sortCols, boolean isMaterialization, + boolean isTemporaryTable) { Map<String, String> retValue; if (tblProp == null) { retValue = new HashMap<String, String>(); @@ -12797,7 +12798,8 @@ public class SemanticAnalyzer extends BaseSemanticAnalyzer { } } boolean makeInsertOnly = HiveConf.getBoolVar(conf, ConfVars.HIVE_CREATE_TABLES_AS_INSERT_ONLY); - boolean makeAcid = MetastoreConf.getBoolVar(conf, MetastoreConf.ConfVars.CREATE_TABLES_AS_ACID) && + boolean makeAcid = !isTemporaryTable && + MetastoreConf.getBoolVar(conf, MetastoreConf.ConfVars.CREATE_TABLES_AS_ACID) && HiveConf.getBoolVar(conf, ConfVars.HIVE_SUPPORT_CONCURRENCY) && DbTxnManager.class.getCanonicalName().equals(HiveConf.getVar(conf, ConfVars.HIVE_TXN_MANAGER)); if ((makeInsertOnly || makeAcid) @@ -13109,7 +13111,7 @@ public class SemanticAnalyzer extends BaseSemanticAnalyzer { case CREATE_TABLE: // REGULAR CREATE TABLE DDL tblProps = addDefaultProperties( - tblProps, isExt, storageFormat, dbDotTab, sortCols, isMaterialization); + tblProps, isExt, storageFormat, dbDotTab, sortCols, isMaterialization, isTemporary); addDbAndTabToOutputs(qualifiedTabName, TableType.MANAGED_TABLE, tblProps); CreateTableDesc crtTblDesc = new CreateTableDesc(dbDotTab, isExt, isTemporary, cols, partCols, @@ -13133,7 +13135,7 @@ public class SemanticAnalyzer extends BaseSemanticAnalyzer { case CTLT: // create table like <tbl_name> tblProps = addDefaultProperties( - tblProps, isExt, storageFormat, dbDotTab, sortCols, isMaterialization); + tblProps, isExt, storageFormat, dbDotTab, sortCols, isMaterialization, isTemporary); addDbAndTabToOutputs(qualifiedTabName, TableType.MANAGED_TABLE, tblProps); if (isTemporary) { @@ -13213,7 +13215,7 @@ public class SemanticAnalyzer extends BaseSemanticAnalyzer { } tblProps = addDefaultProperties( - tblProps, isExt, storageFormat, dbDotTab, sortCols, isMaterialization); + tblProps, isExt, storageFormat, dbDotTab, sortCols, isMaterialization, isTemporary); addDbAndTabToOutputs(qualifiedTabName, TableType.MANAGED_TABLE, tblProps); tableDesc = new CreateTableDesc(qualifiedTabName[0], dbDotTab, isExt, isTemporary, cols, partCols, bucketCols, sortCols, numBuckets, rowFormatParams.fieldDelim, http://git-wip-us.apache.org/repos/asf/hive/blob/5ced7bf9/ql/src/test/queries/clientpositive/cttl.q ---------------------------------------------------------------------- diff --git a/ql/src/test/queries/clientpositive/cttl.q b/ql/src/test/queries/clientpositive/cttl.q new file mode 100644 index 0000000..b3da8ec --- /dev/null +++ b/ql/src/test/queries/clientpositive/cttl.q @@ -0,0 +1,29 @@ +set hive.support.concurrency=true; +set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; + +set metastore.create.as.acid=true; +set hive.default.fileformat=textfile; +set hive.default.fileformat.managed=orc; + +dfs ${system:test.dfs.mkdir} hdfs:///tmp/hive19577_cttl; +dfs -copyFromLocal ../../data/files/students.txt hdfs:///tmp/hive19577_cttl/; +dfs -ls hdfs:///tmp/hive19577_cttl/; + +drop table if exists students; +create external table students( + name string, + age int, + gpa double) + row format delimited + fields terminated by '\t' + stored as textfile + location 'hdfs:///tmp/hive19577_cttl'; + +create temporary table temp1 like students; +insert into table temp1 select * from students; +select * from temp1 order by name limit 10; + +drop table students; +dfs -ls hdfs:///tmp/hive19577_cttl/; + +dfs -rmr hdfs:///tmp/hive19577_cttl; http://git-wip-us.apache.org/repos/asf/hive/blob/5ced7bf9/ql/src/test/results/clientpositive/llap/cttl.q.out ---------------------------------------------------------------------- diff --git a/ql/src/test/results/clientpositive/llap/cttl.q.out b/ql/src/test/results/clientpositive/llap/cttl.q.out new file mode 100644 index 0000000..8203b80 --- /dev/null +++ b/ql/src/test/results/clientpositive/llap/cttl.q.out @@ -0,0 +1,78 @@ +Found 1 items +-rw-r--r-- 3 ### USER ### ### GROUP ### 4316 ### HDFS DATE ### hdfs://### HDFS PATH ### +PREHOOK: query: drop table if exists students +PREHOOK: type: DROPTABLE +POSTHOOK: query: drop table if exists students +POSTHOOK: type: DROPTABLE +PREHOOK: query: create external table students( + name string, + age int, + gpa double) + row format delimited + fields terminated by '\t' + stored as textfile + location 'hdfs://### HDFS PATH ###' +PREHOOK: type: CREATETABLE +PREHOOK: Input: hdfs://### HDFS PATH ### +PREHOOK: Output: database:default +PREHOOK: Output: default@students +POSTHOOK: query: create external table students( + name string, + age int, + gpa double) + row format delimited + fields terminated by '\t' + stored as textfile + location 'hdfs://### HDFS PATH ###' +POSTHOOK: type: CREATETABLE +POSTHOOK: Input: hdfs://### HDFS PATH ### +POSTHOOK: Output: database:default +POSTHOOK: Output: default@students +PREHOOK: query: create temporary table temp1 like students +PREHOOK: type: CREATETABLE +PREHOOK: Output: database:default +PREHOOK: Output: default@temp1 +POSTHOOK: query: create temporary table temp1 like students +POSTHOOK: type: CREATETABLE +POSTHOOK: Output: database:default +POSTHOOK: Output: default@temp1 +PREHOOK: query: insert into table temp1 select * from students +PREHOOK: type: QUERY +PREHOOK: Input: default@students +PREHOOK: Output: default@temp1 +POSTHOOK: query: insert into table temp1 select * from students +POSTHOOK: type: QUERY +POSTHOOK: Input: default@students +POSTHOOK: Output: default@temp1 +POSTHOOK: Lineage: temp1.age SIMPLE [(students)students.FieldSchema(name:age, type:int, comment:null), ] +POSTHOOK: Lineage: temp1.gpa SIMPLE [(students)students.FieldSchema(name:gpa, type:double, comment:null), ] +POSTHOOK: Lineage: temp1.name SIMPLE [(students)students.FieldSchema(name:name, type:string, comment:null), ] +PREHOOK: query: select * from temp1 order by name limit 10 +PREHOOK: type: QUERY +PREHOOK: Input: default@temp1 +PREHOOK: Output: hdfs://### HDFS PATH ### +POSTHOOK: query: select * from temp1 order by name limit 10 +POSTHOOK: type: QUERY +POSTHOOK: Input: default@temp1 +POSTHOOK: Output: hdfs://### HDFS PATH ### +alice falkner 28 3.72 +alice garcia 42 2.03 +alice ichabod 65 2.25 +alice robinson 37 3.69 +alice young 69 0.25 +alice young 75 0.31 +bob ichabod 56 2.81 +bob johnson 53 3.31 +bob laertes 60 3.33 +bob nixon 34 3.34 +PREHOOK: query: drop table students +PREHOOK: type: DROPTABLE +PREHOOK: Input: default@students +PREHOOK: Output: default@students +POSTHOOK: query: drop table students +POSTHOOK: type: DROPTABLE +POSTHOOK: Input: default@students +POSTHOOK: Output: default@students +Found 1 items +-rw-r--r-- 3 ### USER ### ### GROUP ### 4316 ### HDFS DATE ### hdfs://### HDFS PATH ### +#### A masked pattern was here ####
