Re: Review Request 65478: HIVE-18553 VectorizedParquetReader fails after adding a new column to table

[Vihang Karajgaonkar via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65478/#review196867
---




ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetDataColumnReaderFactory.java
Lines 114 (patched)


nit, It should probably say "Implementation is consistent with .." Same 
with TypesFromInt64PageReader and TypesFromFloatPageReader below. Thanks



ql/src/test/queries/clientpositive/schema_evol_par_vec_table.q
Lines 9 (patched)


Does this test work when dictionary encoding is both enabled/disabled? You 
can change parquet dictionary encoding by setting table property 
"parquet.enable.dictionary"="true" or "false". Can you please modify the test 
to include that as well?



ql/src/test/queries/clientpositive/schema_evol_par_vec_table.q
Lines 49 (patched)


I think this comment can be removed now so that there is no confusion in 
the future.



ql/src/test/queries/clientpositive/schema_evol_par_vec_table.q
Lines 71 (patched)


Can you please test using timestamps as well. Specifically, the following 
should work.

drop table test_alter2;
create table test_alter2 (ts timestamp) stored as parquet;
insert into test_alter2 values ('2018-01-01 13:14:15.123456'), ('2018-01-02 
14:15:16.123456'), ('2018-01-03 16:17:18.123456');
select * from test_alter2;
alter table test_alter2 replace columns (ts string);
select * from test_alter2;

drop table test_alter2;
create table test_alter2 (ts timestamp) stored as parquet;
insert into test_alter2 values ('2018-01-01 13:14:15.123456'), ('2018-01-02 
14:15:16.123456'), ('2018-01-03 16:17:18.123456');
select * from test_alter2;
alter table test_alter2 replace columns (ts varchar(19));
-- this should truncate the microseconds
select * from test_alter2;

drop table test_alter2;
create table test_alter2 (ts timestamp) stored as parquet;
insert into test_alter2 values ('2018-01-01 13:14:15.123456'), ('2018-01-02 
14:15:16.123456'), ('2018-01-03 16:17:18.123456');
select * from test_alter2;
alter table test_alter2 replace columns (ts char(25);
select * from test_alter2;


- Vihang Karajgaonkar


On Feb. 5, 2018, 8:46 a.m., cheng xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65478/
> ---
> 
> (Updated Feb. 5, 2018, 8:46 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> VectorizedParquetReader throws an exception when trying to reading from a 
> parquet table on which new columns are added.
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/BaseVectorizedColumnReader.java
>  907a9b8 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetDataColumnReaderFactory.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedDummyColumnReader.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java
>  08ac57b 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestVectorizedColumnReader.java
>  9e414dc 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/VectorizedColumnReaderTestBase.java
>  5d3ebd6 
>   ql/src/test/queries/clientpositive/schema_evol_par_vec_table.q PRE-CREATION 
>   ql/src/test/results/clientpositive/schema_evol_par_vec_table.q.out 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65478/diff/2/
> 
> 
> Testing
> ---
> 
> Newly added UT passed and qtest passed locally.
> 
> 
> Thanks,
> 
> cheng xu
> 
>

Re: Review Request 65130: HIVE-18350 : load data should rename files consistent with insert statements

[Deepak Jaiswal]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65130/
---

(Updated Feb. 6, 2018, 7:03 a.m.)


Review request for hive, Ashutosh Chauhan, Eugene Koifman, Gopal V, Jason Dere, 
and Thejas Nair.


Changes
---

Implemented review comments by Sergey and Ashutosh.
- Made BucketingVersion enum.
- Removed hardcoded code in CustomPatitionVertex to use Utilities code to fetch 
bucket number from file name.


Bugs: HIVE-18350
https://issues.apache.org/jira/browse/HIVE-18350


Repository: hive-git


Description
---

Made changes for both bucketed and non-bucketed tables.
Added a positive test for non-bucketed table which renames the loaded file.
Added couple of negative tests for bucketed table which reject a load with 
inconsistent file name.


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomPartitionVertex.java 
26afe90faa 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomVertexConfiguration.java 
ef5e7edcd6 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 9885038588 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 632a21390d 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 
dc698c8de8 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/metainfo/annotation/OpTraitsRulesProcFactory.java
 69d9f3125a 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkMapJoinOptimizer.java
 bacc44482a 
  ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
54f5bab6de 
  ql/src/java/org/apache/hadoop/hive/ql/plan/OpTraits.java 9621c3be53 
  ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHive.java b5b478fbad 
  ql/src/test/queries/clientpositive/auto_sortmerge_join_2.q e5fdcb57e4 
  ql/src/test/queries/clientpositive/auto_sortmerge_join_4.q abf09e5534 
  ql/src/test/queries/clientpositive/auto_sortmerge_join_5.q b85c4a7aa3 
  ql/src/test/queries/clientpositive/auto_sortmerge_join_7.q bd780861e3 
  ql/src/test/results/clientnegative/bucket_mapjoin_mismatch1.q.out b9c2e6f827 
  ql/src/test/results/clientpositive/auto_sortmerge_join_2.q.out 5cfc35aa73 
  ql/src/test/results/clientpositive/auto_sortmerge_join_4.q.out 0d586fd26b 
  ql/src/test/results/clientpositive/auto_sortmerge_join_5.q.out 45704d1253 
  ql/src/test/results/clientpositive/auto_sortmerge_join_7.q.out 1959075912 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_2.q.out 
054b0d00be 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_4.q.out 
95d329862c 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_5.q.out 
e711715aa5 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_7.q.out 
53c685cb11 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_2.q.out 
8cfa113794 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_4.q.out 
fce5e0cfc4 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_5.q.out 
8250eca099 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_7.q.out 
eb813c1734 
  standalone-metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp 
af0fd6b0e0 
  standalone-metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h 4c09bc8fe6 
  standalone-metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp 
aadf8f17c4 
  
standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java
 a132e5e838 
  standalone-metastore/src/gen/thrift/gen-php/metastore/Types.php a5b578ef37 
  standalone-metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py 
5598859042 
  standalone-metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb bc58cfe0ef 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 3d1c67f97c 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MTable.java
 aea16ade7d 
  standalone-metastore/src/main/thrift/hive_metastore.thrift 371b97590c 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java
 bd61df654a 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestTablesCreateDropAlterTruncate.java
 00f38eeec5 


Diff: https://reviews.apache.org/r/65130/diff/11/

Changes: https://reviews.apache.org/r/65130/diff/10-11/


Testing
---


Thanks,

Deepak Jaiswal

[jira] [Created] (HIVE-18632) TYPE CASTING ERROR: CONVERTING FORM INT TO STRING

[shiva (JIRA)]

shiva created HIVE-18632:


 Summary: TYPE CASTING ERROR: CONVERTING FORM INT TO STRING
 Key: HIVE-18632
 URL: https://issues.apache.org/jira/browse/HIVE-18632
 Project: Hive
  Issue Type: Bug
  Components: Database/Schema
Affects Versions: 1.1.0
Reporter: shiva


HI,

AVRO source file field data type have int but we want to load the data using 
the string,

we have created the AVSC schema for that field has string.

getting error while doing the select * from table name;

1 >> Created a table in Hive with AVRO data.
CREATE EXTERNAL TABLE tweets (username string, tweet string, timestamp bigint)
COMMENT 'A table backed by Avro data with the Avro schema stored in HDFS'
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS
INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
LOCATION '/home/neha/test_data/avro_create_data'
TBLPROPERTIES 
('avro.schema.literal'='{"namespace":"com.miguno.avro","name":"Tweet","type":"record","fields":[

{"name" : "username","type" : "string","doc" : "Name of the user account on 
Twitter.com"}
,

{"name" : "tweet","type":"string","doc" : "The content of the Twitter message"}
,

{"name" : "timestamp", "type" : "long", "doc" : "Unix epoch time in seconds"}
]}');

2 >> Altered type of a column (to a compatible type) using ALTER TABLE. In this 
example, altered type for column "timestamp" from "long" to "int".

ALTER TABLE tweets SET TBLPROPERTIES 
('avro.schema.literal'='{"namespace":"com.miguno.avro","name":"Tweet","type":"record","fields":[

{"name" : "username","type" : "string","doc" : "Name of the user account on 
Twitter.com"}
,

{"name" : "tweet","type":"string","doc" : "The content of the Twitter message"}
,

{"name" : "timestamp", "type" : "int", "doc" : "Unix epoch time in seconds"}
]}');

3 >> Now, a select query on this table fails with following error.

hive> select * from tweets;
OK
Failed with exception java.io.IOException:org.apache.avro.AvroTypeException: 
Found "long", expecting "int"
Time taken: 4.514 seconds



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18631) Hive metastore schema initialization failing on mysql

[Deepesh Khandelwal (JIRA)]

Deepesh Khandelwal created HIVE-18631:
-

 Summary: Hive metastore schema initialization failing on mysql
 Key: HIVE-18631
 URL: https://issues.apache.org/jira/browse/HIVE-18631
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 3.0.0
Reporter: Deepesh Khandelwal
Assignee: Deepesh Khandelwal


Hive metastore schema on mysql is broken after the commit for HIVE-18546. 
Following error is seen during schema initialization:
{noformat}
0: jdbc:mysql://localhost.localdomain> CREATE TABLE IF NOT EXISTS `TBLS` 
 ( `TBL_ID` bigint(20) NOT NULL, `CREATE_TIME` int(11) NOT NULL, `DB_ID` 
bigint( 
20) DEFAULT NULL, `LAST_ACCESS_TIME` int(11) NOT NULL, `OWNER` varchar(767) 
CHAR 
ACTER SET latin1 COLLATE latin1_bin DEFAULT NULL, `RETENTION` int(11) NOT NULL, 
 
`SD_ID` bigint(20) DEFAULT NULL, `TBL_NAME` varchar(256) CHARACTER SET latin1 
CO 
LLATE latin1_bin DEFAULT NULL, `TBL_TYPE` varchar(128) CHARACTER SET latin1 
COLL 
ATE latin1_bin DEFAULT NULL, `VIEW_EXPANDED_TEXT` mediumtext, 
`VIEW_ORIGINAL_TEX 
T` mediumtext, `IS_REWRITE_ENABLED` bit(1) NOT NULL DEFAULT 0 PRIMARY KEY 
(`TBL_ 
ID`), UNIQUE KEY `UNIQUETABLE` (`TBL_NAME`,`DB_ID`), KEY `TBLS_N50` (`SD_ID`), 
K 
EY `TBLS_N49` (`DB_ID`), CONSTRAINT `TBLS_FK1` FOREIGN KEY (`SD_ID`) REFERENCES 
 
`SDS` (`SD_ID`), CONSTRAINT `TBLS_FK2` FOREIGN KEY (`DB_ID`) REFERENCES `DBS` 
(` 
DB_ID`) ) ENGINE=InnoDB DEFAULT CHARSET=latin1
Error: You have an error in your SQL syntax; check the manual that corresponds 
to your MySQL server version for the right syntax to use near '(`TBL_ID`), 
UNIQUE KEY `UNIQUETABLE` (`TBL_NAME`,`DB_ID`), KEY `TBLS_N50` (`SD_I' at line 1 
(state=42000,code=1064)
Closing: 0: 
jdbc:mysql://localhost.localdomain/hivedb?createDatabaseIfNotExist=true
org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization 
FAILED! Metastore state would be inconsistent !!
Underlying cause: java.io.IOException : Schema script failed, errorcode 2
org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization 
FAILED! Metastore state would be inconsistent !!
at 
org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:586)
at 
org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:559)
at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1183)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:304)
at org.apache.hadoop.util.RunJar.main(RunJar.java:218)
Caused by: java.io.IOException: Schema script failed, errorcode 2
at 
org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:957)
at 
org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:935)
at 
org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:582)
... 8 more
*** schemaTool failed ***{noformat}
In the file metastore/scripts/upgrade/mysql/hive-schema-3.0.0.mysql.sql one of 
the column definitions in the `TBLS` table is missing a comma at the end
{code:java}
`IS_REWRITE_ENABLED` bit(1) NOT NULL DEFAULT 0{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18630) Resource plan validate/activate does not verify trigger expressions.

[Harish Jaiprakash (JIRA)]

Harish Jaiprakash created HIVE-18630:


 Summary: Resource plan validate/activate does not verify trigger 
expressions.
 Key: HIVE-18630
 URL: https://issues.apache.org/jira/browse/HIVE-18630
 Project: Hive
  Issue Type: Bug
Reporter: Harish Jaiprakash


Resource plan trigger expressions are not validated in WM validate. They are 
stored as strings in the database, standalone-metastore does not have access to 
the expression parser. This results in validate not checking the counter names 
and pool names in the validate/activate command.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65413: HIVE-18575 ACID properties usage in jobconf is ambiguous for MM tables

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65413/
---

(Updated Feb. 6, 2018, 3:29 a.m.)


Review request for hive and Eugene Koifman.


Repository: hive-git


Description
---

.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 99e8457c7b 
  
hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FosterStorageHandler.java
 5ee8aadfa7 
  
hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/HiveEndPoint.java
 8943423244 
  
hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
 5e12614cfe 
  
hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/mutate/StreamingAssert.java
 c98d22be2e 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 1305902351 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
 d252279be9 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 68bb168bd2 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java d3aa571913 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java a7dace955d 
  ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java 270b576199 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java abd42ec651 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 430e0fc551 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java c3b846c4d2 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ff2cc0455c 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java 61565ef030 
  
ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
 da200049bc 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcInputFormat.java 
7b157e6486 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 3968b0e899 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 07999e2e9e 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketingSortingReduceSinkOptimizer.java
 0fdff7d853 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
69447d9d34 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
190771ea6b 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
3eb869d3f6 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java b67a03f213 
  ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/TableExport.java 
e1cea22005 
  ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 661446df0b 
  ql/src/java/org/apache/hadoop/hive/ql/stats/Partish.java 78f48b169a 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
0e456df19c 
  ql/src/test/org/apache/hadoop/hive/ql/io/TestAcidUtils.java 8945fdf1e7 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
92f005d1dc 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcRawRecordMerger.java 
c6a866a164 
  
ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestVectorizedOrcAcidRowBatchReader.java
 65508f4ddd 


Diff: https://reviews.apache.org/r/65413/diff/3/

Changes: https://reviews.apache.org/r/65413/diff/2-3/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 65413: HIVE-18575 ACID properties usage in jobconf is ambiguous for MM tables

[Sergey Shelukhin]

> On Feb. 5, 2018, 6:52 p.m., Eugene Koifman wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java
> > Line 862 (original), 863 (patched)
> > 
> >
> > is 'job' cloned somewhere between invocations?  If not this seems like 
> > a bug

This is existing code. Another strange thing here is that there's job (the 
field) and jobConf (the parameter), and some code in this methods updates one 
while some updates the other.
They are not obviously identical. 
Does it make sense to investigate/fix in a follow-up jira?


> On Feb. 5, 2018, 6:52 p.m., Eugene Koifman wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java
> > Line 202 (original), 202 (patched)
> > 
> >
> > on the client we have isTransactionalTable() to mean MM or full acid 
> > and isAcidTable() to mean full acid.
> > 
> > Shouldn't this be isAcidRead() to be consistent?

I;m going to change these too.. I see that some code calls is(Full)Acid... && 
!isInsertOnly.
These methods should be explicit.


> On Feb. 5, 2018, 6:52 p.m., Eugene Koifman wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java
> > Line 507 (original), 507 (patched)
> > 
> >
> > stupid RB won't let me edit my own comment...
> > Previsouly this check was testing for full acid but now just for 
> > transactional.  Why?

It's used to skip global limit optimization. As far as I understand this should 
also be skipped for insert-only tables


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65413/#review196819
---


On Jan. 29, 2018, 11:12 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65413/
> ---
> 
> (Updated Jan. 29, 2018, 11:12 p.m.)
> 
> 
> Review request for hive and Eugene Koifman.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> .
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> b7d3e99e1a505f576a06c530080fc72dddcd85ba 
>   
> hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FosterStorageHandler.java
>  5ee8aadfa774a85a0bdbcaf78a636ff6593c43e2 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  5e12614cfe17030f8fcb56ef8c83b53b8b870c97 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/mutate/StreamingAssert.java
>  c98d22be2e6216e95d9c13f3a26540ca03e7405e 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  13059023516edbb58a9129ba9aa49de7e40129e6 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
>  d252279be973201227da52d8aecf83b3fcc4656b 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  68bb168bd23b84dd150cdc4da63d73657f1b33bb 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 
> a7dace955d6fb3dabc4c5e77ef68f83617eb48d1 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java 
> 270b576199c57c109195b85d43e216743a607955 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 
> abd42ec651927503e7c8c2d9a7d3d415cc9c4ac4 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 
> eb75308e8393cadf8e69e0e30b303474b89df03e 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 
> c3b846c4d2fee8691b4952b9f6cf4dd1d8bd632f 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 
> ff2cc0455c64ed210d8ff14a9f112cd91b7314be 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java 
> 61565ef0305006a57b7f608e60ddcdf2b6ff474d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
>  da200049bcbc8f2fe1d793acc7b84f8b99ae67cc 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcInputFormat.java 
> 7b157e648646c5a199aaebf04484b81ff1c12478 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 923372d5b6da42446997051d0758e9aab4881e2e 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> 190771ea6b1cbf4b669a8919271b25a689af941b 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 
> 661446df0b9fbb5cf248d76205e47dbaa113026f 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> 9152b4e08a7a1044fc7f844d47ae8e180162b78b 
>   ql/src/test/org/apache/hadoop/hive/ql/io/TestAcidUtils.java 
> 26a96a47f1935de8e985d382b40c8aae604a9880 
>   

Re: Review Request 54341: HIVE-15353: Metastore throws NPE if StorageDescriptor.cols is null

[Anthony Hsu via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54341/
---

(Updated 二月 6, 2018, 3:01 a.m.)


Review request for hive, Carl Steinbach and Ratandeep Ratti.


Changes
---

Rebased on HEAD.


Bugs: HIVE-15353
https://issues.apache.org/jira/browse/HIVE-15353


Repository: hive-git


Description (updated)
---

Updated HiveAlterHandler.updateOrGetPartitionColumnStats to handle null 
`oldCols`.


Diffs (updated)
-

  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
 89354a2d34249903a9ff13c4ed913a68de93057e 


Diff: https://reviews.apache.org/r/54341/diff/4/

Changes: https://reviews.apache.org/r/54341/diff/3-4/


Testing
---

After making these changes, I no longer encounter NullPointerExceptions when 
setting cols to null in create_table, alter_table, and alter_partition calls.


Thanks,

Anthony Hsu

Re: Review Request 65130: HIVE-18350 : load data should rename files consistent with insert statements

[Deepak Jaiswal]

> On Feb. 6, 2018, 1:26 a.m., Sergey Shelukhin wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomPartitionVertex.java
> > Lines 551 (patched)
> > 
> >
> > what's the logic behind "- 2"? needs to be documented.
> > Is it "_0" at the end? cannot that potentially have 2+-digit numbers?

Upon checking, you are correct, I should have replaced this with the new 
function I introduced, Utilities.getBucketFileNameFromPathSubString()


> On Feb. 6, 2018, 1:26 a.m., Sergey Shelukhin wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomPartitionVertex.java
> > Lines 588 (patched)
> > 
> >
> > this doesn't actually throw an error

The idea is to fallback to old logic which will work for existing customers who 
have bucketed tables with names which dont fit required format.


> On Feb. 6, 2018, 1:26 a.m., Sergey Shelukhin wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomPartitionVertex.java
> > Lines 595 (patched)
> > 
> >
> > why is this 1?

Becuase the 1st iteration happened in the for loop above at line 546. These 
iterations are needed only when the small table has less number of buckets than 
big table.


> On Feb. 6, 2018, 1:26 a.m., Sergey Shelukhin wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java
> > Lines 619 (patched)
> > 
> >
> > assert?

sure. I will add a Precondition check.


> On Feb. 6, 2018, 1:26 a.m., Sergey Shelukhin wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/metainfo/annotation/OpTraitsRulesProcFactory.java
> > Lines 350 (patched)
> > 
> >
> > is there a test for this scenario? I couldn't find an explicit check 
> > for -1, so I wonder if it's possible to get -1s everywhere and have them 
> > match.
> > Perhaps I'm misunderstanding the logic here.

Thanks for pointing out this code. It was supposed to be taken out but fell off 
my radar. The check is done explicitly in checkConvertJoinSMBJoin. We dont care 
what the value in join is.


> On Feb. 6, 2018, 1:26 a.m., Sergey Shelukhin wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java
> > Lines 173 (patched)
> > 
> >
> > nit: Arrays.fill

Still a JAVA n00b :(


> On Feb. 6, 2018, 1:26 a.m., Sergey Shelukhin wrote:
> > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MTable.java
> > Lines 40 (patched)
> > 
> >
> > ideally, bucketing versions should be an enum (like e.g. ORC writer 
> > versions).
> > They can be stored as ints in metastore, but used as enum in QL, to 
> > check explicitly that we get a supported value (and have a fixed list of 
> > values)

Should have thought. Thanks for the suggestion.


> On Feb. 6, 2018, 1:26 a.m., Sergey Shelukhin wrote:
> > standalone-metastore/src/main/thrift/hive_metastore.thrift
> > Line 331 (original), 333 (patched)
> > 
> >
> > what does expertMode actually mean? ie what does it affect. Perhaps 
> > there should be a comment in Table.java that explains it.

As commented by Ashutosh, the name is not clear enough. I will quote what I 
wrote there. Will add more comments.

"This is something which Gopal suggested. It is not supposed to block loads. 
This config is set to true when user uses "load data" in bucketing strict mode 
set to false. The plumbing is not complete so this field is just a placeholder 
for the moment.
The idea is to check this value when user complaints about wrong results when 
using bucket based joins as load data without data check can potentially ruin 
the bucketing scheme.
Agree with name, will think of something more appropriate."


- Deepak


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65130/#review196846
---


On Feb. 4, 2018, 2:14 a.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65130/
> ---
> 
> (Updated Feb. 4, 2018, 2:14 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Eugene Koifman, Gopal V, Jason 
> Dere, and Thejas Nair.
> 
> 
> Bugs: HIVE-18350
> https://issues.apache.org/jira/browse/HIVE-18350
> 
> 
> Repository: hive-git
> 

Re: Review Request 65478: HIVE-18553 VectorizedParquetReader fails after adding a new column to table

[cheng xu]

> On Feb. 6, 2018, 1:46 a.m., Vihang Karajgaonkar wrote:
> >

Thanks Vihang for your review. Comments left below.


> On Feb. 6, 2018, 1:46 a.m., Vihang Karajgaonkar wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetDataColumnReaderFactory.java
> > Lines 24 (patched)
> > 
> >
> > Do we need to override the methods for other reads as well? What is the 
> > criteria to identify the methods which need to be overriden for this and 
> > TypesFromInt64PageReader?
> 
> Jerry Chen wrote:
> Current Ferdinand follows the same conversion principles of 
> ETypeConverter in Hive. The basic conversion implemented in ETypeConverter 
> is: low precision data type can convert to high precison data type. for 
> int32: it can be converted to int64, float, double. for int64: it can be 
> converted to float, double. For float: it can be converted to double.
> 
> Although other conversions can logically supported such as int64 to int32 
> or double to float. But that is just not always safe.

Yes, as Jerry mentioned, our current implementation stays the same rules as non 
vectorization path. To support other conversion, we may need to ensure both 
vectorization and non vectorization have the same behavior since vectorization 
is just a performance feature.


> On Feb. 6, 2018, 1:46 a.m., Vihang Karajgaonkar wrote:
> > ql/src/test/queries/clientpositive/schema_evol_par_vec_table.q
> > Lines 9 (patched)
> > 
> >
> > can you please confirm that the result of this q file matches with 
> > non-vectorized execution?

Yes, I checked the output as well by disabling the vectorization. And they have 
the same output as vectorization enabled case.


> On Feb. 6, 2018, 1:46 a.m., Vihang Karajgaonkar wrote:
> > ql/src/test/results/clientpositive/schema_evol_par_vec_table.q.out
> > Lines 140 (patched)
> > 
> >
> > this is interesting. Do you know why this row is returned first?

The order is the same as non vectorization read path. It may result from the 
order of generated files.


- cheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65478/#review196718
---


On Feb. 5, 2018, 4:46 p.m., cheng xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65478/
> ---
> 
> (Updated Feb. 5, 2018, 4:46 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> VectorizedParquetReader throws an exception when trying to reading from a 
> parquet table on which new columns are added.
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/BaseVectorizedColumnReader.java
>  907a9b8 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetDataColumnReaderFactory.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedDummyColumnReader.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java
>  08ac57b 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestVectorizedColumnReader.java
>  9e414dc 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/VectorizedColumnReaderTestBase.java
>  5d3ebd6 
>   ql/src/test/queries/clientpositive/schema_evol_par_vec_table.q PRE-CREATION 
>   ql/src/test/results/clientpositive/schema_evol_par_vec_table.q.out 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65478/diff/2/
> 
> 
> Testing
> ---
> 
> Newly added UT passed and qtest passed locally.
> 
> 
> Thanks,
> 
> cheng xu
> 
>

[jira] [Created] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-18629:
---

 Summary: copyValues in BytesColumnVector may be missing null checks
 Key: HIVE-18629
 URL: https://issues.apache.org/jira/browse/HIVE-18629
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin


{noformat}
Caused by: java.lang.NullPointerException
at java.lang.System.arraycopy(Native Method)
at 
org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173)
at 
org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333)
at 
org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83)
at 
org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133)
{noformat}

IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case has 
isNull checks for v2/v3 buffers that copySelected is missing. 




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18628) Make tez dag status check interval configurable

[Prasanth Jayachandran (JIRA)]

Prasanth Jayachandran created HIVE-18628:


 Summary: Make tez dag status check interval configurable
 Key: HIVE-18628
 URL: https://issues.apache.org/jira/browse/HIVE-18628
 Project: Hive
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran


DAG status check interval in TezJobMonitor is not configurable. This can cause 
cached data to be returned and fail some trigger validation tests. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65478: HIVE-18553 VectorizedParquetReader fails after adding a new column to table

[Jerry Chen]

> On Feb. 5, 2018, 5:46 p.m., Vihang Karajgaonkar wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetDataColumnReaderFactory.java
> > Lines 24 (patched)
> > 
> >
> > Do we need to override the methods for other reads as well? What is the 
> > criteria to identify the methods which need to be overriden for this and 
> > TypesFromInt64PageReader?

Current Ferdinand follows the same conversion principles of ETypeConverter in 
Hive. The basic conversion implemented in ETypeConverter is: low precision data 
type can convert to high precison data type. for int32: it can be converted to 
int64, float, double. for int64: it can be converted to float, double. For 
float: it can be converted to double.

Although other conversions can logically supported such as int64 to int32 or 
double to float. But that is just not always safe.


- Jerry


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65478/#review196718
---


On Feb. 5, 2018, 8:46 a.m., cheng xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65478/
> ---
> 
> (Updated Feb. 5, 2018, 8:46 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> VectorizedParquetReader throws an exception when trying to reading from a 
> parquet table on which new columns are added.
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/BaseVectorizedColumnReader.java
>  907a9b8 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetDataColumnReaderFactory.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedDummyColumnReader.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java
>  08ac57b 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestVectorizedColumnReader.java
>  9e414dc 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/VectorizedColumnReaderTestBase.java
>  5d3ebd6 
>   ql/src/test/queries/clientpositive/schema_evol_par_vec_table.q PRE-CREATION 
>   ql/src/test/results/clientpositive/schema_evol_par_vec_table.q.out 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65478/diff/2/
> 
> 
> Testing
> ---
> 
> Newly added UT passed and qtest passed locally.
> 
> 
> Thanks,
> 
> cheng xu
> 
>

Re: Review Request 65130: HIVE-18350 : load data should rename files consistent with insert statements

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65130/#review196846
---




ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomPartitionVertex.java
Lines 551 (patched)


what's the logic behind "- 2"? needs to be documented.
Is it "_0" at the end? cannot that potentially have 2+-digit numbers?



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomPartitionVertex.java
Lines 588 (patched)


this doesn't actually throw an error



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomPartitionVertex.java
Lines 595 (patched)


why is this 1?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java
Lines 619 (patched)


assert?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/metainfo/annotation/OpTraitsRulesProcFactory.java
Lines 350 (patched)


is there a test for this scenario? I couldn't find an explicit check for 
-1, so I wonder if it's possible to get -1s everywhere and have them match.
Perhaps I'm misunderstanding the logic here.



ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java
Lines 173 (patched)


nit: Arrays.fill



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MTable.java
Lines 40 (patched)


ideally, bucketing versions should be an enum (like e.g. ORC writer 
versions).
They can be stored as ints in metastore, but used as enum in QL, to check 
explicitly that we get a supported value (and have a fixed list of values)



standalone-metastore/src/main/thrift/hive_metastore.thrift
Line 331 (original), 333 (patched)


what does expertMode actually mean? ie what does it affect. Perhaps there 
should be a comment in Table.java that explains it.


- Sergey Shelukhin


On Feb. 4, 2018, 2:14 a.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65130/
> ---
> 
> (Updated Feb. 4, 2018, 2:14 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Eugene Koifman, Gopal V, Jason 
> Dere, and Thejas Nair.
> 
> 
> Bugs: HIVE-18350
> https://issues.apache.org/jira/browse/HIVE-18350
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Made changes for both bucketed and non-bucketed tables.
> Added a positive test for non-bucketed table which renames the loaded file.
> Added couple of negative tests for bucketed table which reject a load with 
> inconsistent file name.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomPartitionVertex.java 
> 26afe90faa 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomVertexConfiguration.java 
> ef5e7edcd6 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 9885038588 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 9b0ffe0e91 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 
> dc698c8de8 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/metainfo/annotation/OpTraitsRulesProcFactory.java
>  69d9f3125a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkMapJoinOptimizer.java
>  bacc44482a 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
> 54f5bab6de 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/OpTraits.java 9621c3be53 
>   ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHive.java aa95d2fcdc 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_2.q e5fdcb57e4 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_4.q abf09e5534 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_5.q b85c4a7aa3 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_7.q bd780861e3 
>   ql/src/test/results/clientnegative/bucket_mapjoin_mismatch1.q.out 
> b9c2e6f827 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_2.q.out 5cfc35aa73 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_4.q.out 0d586fd26b 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_5.q.out 45704d1253 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_7.q.out 1959075912 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_2.q.out 
> 054b0d00be 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_4.q.out 
> 95d329862c 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_5.q.out 
> e711715aa5 
>   

Re: Review Request 65130: HIVE-18350 : load data should rename files consistent with insert statements

[Ashutosh Chauhan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65130/#review196844
---




ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomPartitionVertex.java
Line 544 (original), 547 (patched)


Comment should also include an e.g., filename which adheres to this scheme.



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomPartitionVertex.java
Lines 551 (patched)


Does this assume bucket number are in last 2 positions? What if # of 
buckets > 99 ?



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomPartitionVertex.java
Lines 557 (patched)


LOG.debug



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomPartitionVertex.java
Lines 570 (patched)


Should add a comment about assumption of filenames being sorted alphaumeric 
order and mapped to bucket#.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java
Lines 586 (patched)


LOG.debug



ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java
Lines 631 (patched)


LOG.debug



ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java
Lines 183 (patched)


LOG.debug



standalone-metastore/src/main/thrift/hive_metastore.thrift
Lines 332 (patched)


1. Seems like you want to use this to allow/disallow load statements for 
malformed names. But seems like its not used anywhere.
2. You also need to provide upgrade scripts for all metastore for this new 
field in metastore.
2. In any case, I don't see a reason to store this in metadata. This should 
rather be a config variable which user may toggle to force malformed load.
3. Name expertMode is not descriptive. Pick better name : 
allowIllegalLoadInBucketedTable?


- Ashutosh Chauhan


On Feb. 4, 2018, 2:14 a.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65130/
> ---
> 
> (Updated Feb. 4, 2018, 2:14 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Eugene Koifman, Gopal V, Jason 
> Dere, and Thejas Nair.
> 
> 
> Bugs: HIVE-18350
> https://issues.apache.org/jira/browse/HIVE-18350
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Made changes for both bucketed and non-bucketed tables.
> Added a positive test for non-bucketed table which renames the loaded file.
> Added couple of negative tests for bucketed table which reject a load with 
> inconsistent file name.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomPartitionVertex.java 
> 26afe90faa 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomVertexConfiguration.java 
> ef5e7edcd6 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 9885038588 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 9b0ffe0e91 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 
> dc698c8de8 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/metainfo/annotation/OpTraitsRulesProcFactory.java
>  69d9f3125a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkMapJoinOptimizer.java
>  bacc44482a 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
> 54f5bab6de 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/OpTraits.java 9621c3be53 
>   ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHive.java aa95d2fcdc 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_2.q e5fdcb57e4 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_4.q abf09e5534 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_5.q b85c4a7aa3 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_7.q bd780861e3 
>   ql/src/test/results/clientnegative/bucket_mapjoin_mismatch1.q.out 
> b9c2e6f827 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_2.q.out 5cfc35aa73 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_4.q.out 0d586fd26b 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_5.q.out 45704d1253 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_7.q.out 1959075912 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_2.q.out 
> 054b0d00be 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_4.q.out 
> 95d329862c 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_5.q.out 
> e711715aa5 
>   

Re: Review Request 65431: HIVE-16605 : NOT NULL constraint enforcement

[Vineet Garg]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65431/
---

(Updated Feb. 6, 2018, 12:30 a.m.)


Review request for hive, Ashutosh Chauhan and Jesús Camacho Rodríguez.


Changes
---

Added micromanaged table test and addressed comments


Bugs: HIVE-16605
https://issues.apache.org/jira/browse/HIVE-16605


Repository: hive-git


Description
---

This patch introduces support for enable NOT NULL constraint enforcement for 
following type of statements:

* INSERT INTO
* INSERT as SELECT
* INSERT OVERWRITE
* UPDATE
* MERGE


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 99e8457c7b 
  itests/src/test/resources/testconfiguration.properties fed9394544 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 76e85636d1 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java b1e05dffe9 
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 
372cfad866 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java b67a03f213 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 
9fcde76d21 
  ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java e3a9e62bdb 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEnforceNotNullConstraint.java
 PRE-CREATION 
  ql/src/test/queries/clientnegative/alter_notnull_constraint_violation.q 
PRE-CREATION 
  ql/src/test/queries/clientnegative/create_with_constraints_enable.q 
59ebb1e9b2 
  ql/src/test/queries/clientnegative/create_with_constraints_enforced.q  
  ql/src/test/queries/clientnegative/create_with_fk_constraints_enforced.q 
PRE-CREATION 
  ql/src/test/queries/clientnegative/create_with_unique_constraints_enforced.q 
PRE-CREATION 
  ql/src/test/queries/clientnegative/insert_into_acid_notnull.q PRE-CREATION 
  ql/src/test/queries/clientnegative/insert_into_notnull_constraint.q 
PRE-CREATION 
  ql/src/test/queries/clientnegative/insert_multi_into_notnull.q PRE-CREATION 
  ql/src/test/queries/clientnegative/insert_overwrite_notnull_constraint.q 
PRE-CREATION 
  ql/src/test/queries/clientnegative/merge_constraint_notnull.q PRE-CREATION 
  ql/src/test/queries/clientnegative/update_notnull_constraint.q PRE-CREATION 
  ql/src/test/queries/clientpositive/enforce_constraint_notnull.q PRE-CREATION 
  ql/src/test/results/clientnegative/alter_notnull_constraint_violation.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/create_with_constraints_enable.q.out  
  ql/src/test/results/clientnegative/create_with_constraints_enforced.q.out  
  ql/src/test/results/clientnegative/create_with_fk_constraints_enforced.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/insert_into_acid_notnull.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/insert_into_notnull_constraint.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/insert_multi_into_notnull.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/insert_overwrite_notnull_constraint.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/merge_constraint_notnull.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/update_notnull_constraint.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/llap/enforce_constraint_notnull.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/show_functions.q.out 43e4a5de39 


Diff: https://reviews.apache.org/r/65431/diff/4/

Changes: https://reviews.apache.org/r/65431/diff/3-4/


Testing
---


Thanks,

Vineet Garg

[jira] [Created] (HIVE-18627) PPD: Handle FLOAT boxing differently for single/double precision constants

[Gopal V (JIRA)]

Gopal V created HIVE-18627:
--

 Summary: PPD: Handle FLOAT boxing differently for single/double 
precision constants
 Key: HIVE-18627
 URL: https://issues.apache.org/jira/browse/HIVE-18627
 Project: Hive
  Issue Type: Bug
  Components: Types
Reporter: Gopal V


Constants like 0.1 and 0.3 are differently boxed based on intermediate 
precision of the compiler codepath.

Disabling CBO produces 0.1BD constants which fail to box correctly to 
Double/Float.

Enabling CBO fixes this issue, but cannot be applied all queries in Hive.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

[j . prasanth . j]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/
---

(Updated Feb. 5, 2018, 11:05 p.m.)


Review request for hive, Sergey Shelukhin and Thejas Nair.


Changes
---

Added /peers endpoint to list all hs2 instances and their leadership status


Bugs: HIVE-18281
https://issues.apache.org/jira/browse/HIVE-18281


Repository: hive-git


Description
---

HIVE-18281: HiveServer2 HA for LLAP and Workload Manager


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 99e8457 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
PRE-CREATION 
  
itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java 
6cab8cd 
  itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java 
5d7f813 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
 c88198f 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
 80a6aba 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
 8339230 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java 
32d5caa 
  llap-client/src/java/org/apache/hadoop/hive/registry/RegistryUtilities.java 
PRE-CREATION 
  llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
908b3bb 
  llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstanceSet.java 
34fba5c 
  
llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
 db3d788 
  llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
0724cf5 
  
llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
 417e571 
  llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
17269dd 
  llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java 
9666517 
  llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
0120639 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
 58bf8dc 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
 66de3b8 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
82fdf6c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b98fb58 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java b33f027 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
 a8d729d 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
d261623 
  
service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java 
PRE-CREATION 
  
service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistryClient.java
 PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 2a528cd 
  service/src/java/org/apache/hive/service/server/HiveServer2HAInstanceSet.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2Instance.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/servlet/HS2LeadershipStatus.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/servlet/HS2Peers.java PRE-CREATION 


Diff: https://reviews.apache.org/r/65498/diff/2/

Changes: https://reviews.apache.org/r/65498/diff/1-2/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 65479: HIVE-18523 Fix summary row in case there are no inputs

[Ashutosh Chauhan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65479/#review196840
---




ql/src/java/org/apache/hadoop/hive/ql/exec/IConfigureJobConf.java
Lines 24 (patched)


Add: Intended only for compilation phase.



ql/src/java/org/apache/hadoop/hive/ql/exec/IConfigureJobConf.java
Lines 24 (patched)


Add: Only during compilation.



ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java
Lines 259 (patched)


Is this needed?



ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecMapRunner.java
Lines 29 (patched)


Why do we need this class?



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HiveSplitGenerator.java
Line 247 (original), 246-248 (patched)


This may result in extra memory allocation. If this change is not 
necessary, can we leave it as is?



ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorGroupByOperator.java
Lines 453 (patched)


Please add comment.



ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java
Lines 580 (patched)


Add comment on need for this.



ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java
Lines 505 (patched)


Add comment on need for this.



ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceWork.java
Lines 213 (patched)


Do we need this? Reducers are always launched when when there is no mapper. 
So, this seems unnecessary.


- Ashutosh Chauhan


On Feb. 2, 2018, 12:23 p.m., Zoltan Haindrich wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65479/
> ---
> 
> (Updated Feb. 2, 2018, 12:23 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Prasanth_J.
> 
> 
> Bugs: HIVE-18523
> https://issues.apache.org/jira/browse/HIVE-18523
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> * ensure that mapper operators are started up - but only if empty grouping is 
> present
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java 
> 6a0f0de2a5e84770c6446af41710d972d813c7bc 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/IConfigureJobConf.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 
> d7b3e4b2fd3ee1a8e2795095a6c55442de2b38e0 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 
> 976b537033abda5d5ab8b77a7e7d6fb9c84e5a19 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecMapRunner.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecMapper.java 
> 150382a8d58fd4ba44e4d9b78a80173ab984e776 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HiveSplitGenerator.java 
> 98f4bc01c8526422348a38f8d8632e0899d695ee 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorGroupByOperator.java 
> 45d809a1820fcb6ea5e1e5c15aee7de91a4c36c8 
>   ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java 
> e4dfc009d95f4302bd1fcdff2276e11bed68d2e0 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 
> c3b846c4d2fee8691b4952b9f6cf4dd1d8bd632f 
>   ql/src/java/org/apache/hadoop/hive/ql/io/NullRowsInputFormat.java 
> 6a372a3f47e3ac2ae2b2e583541b3a19e5d525f3 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java 
> f2b2fc57a03b368707968eb503139e51218008ca 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceWork.java 
> ecfb118b41bfa5b7d593b7e801a37f0a7b5b0b5e 
>   ql/src/test/queries/clientpositive/groupby_rollup_empty.q 
> 432d8c448a05f51db9ecf9940bce599dfd598a70 
>   ql/src/test/results/clientpositive/groupby_rollup_empty.q.out 
> 7359140e29fc63eebbab42ab385187be6bfc66e1 
>   ql/src/test/results/clientpositive/llap/groupby_rollup_empty.q.out 
> d2b57455a3640387d8bc5f2d415a7af25eb55341 
> 
> 
> Diff: https://reviews.apache.org/r/65479/diff/1/
> 
> 
> Testing
> ---
> 
> added new testcase for union
> 
> 
> Thanks,
> 
> Zoltan Haindrich
> 
>

Re: Review Request 65500: Vectorized execution handles overflows in a different manner than non-vectorized execution

[Vihang Karajgaonkar via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65500/
---

(Updated Feb. 5, 2018, 8:54 p.m.)


Review request for hive, Aihua Xu, Gopal V, Matt McCline, and Sahil Takiar.


Changes
---

patch version 2 with fix for the tests. Changed default value of 
hive.vectorized.use.checked.expressions. Modified test templates for checked 
Expressions.


Bugs: HIVE-18449
https://issues.apache.org/jira/browse/HIVE-18449


Repository: hive-git


Description
---

See JIRA.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
4f2e6d31af85cf1d9866ad1419f5c06a18eea347 
  ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticColumn.txt 
b5011c3adcedf8974d3241994733e0021a851cbd 
  ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticScalar.txt 
cbec1abcc2b66f3ffc91b4778daf5017eff4379d 
  ql/src/gen/vectorization/ExpressionTemplates/ColumnDivideColumn.txt 
3e955578933dd7990939865527c3bd11023b3a90 
  ql/src/gen/vectorization/ExpressionTemplates/ColumnUnaryMinus.txt 
f0ab4711e79c8a1bfceebcde9a3dda2b4e15a38a 
  ql/src/gen/vectorization/ExpressionTemplates/ScalarArithmeticColumn.txt 
e95baa6199e138a4e0c009e62ce495b626e5909c 
  ql/src/gen/vectorization/TestTemplates/TestClass.txt 
62c58fb293fbe2d4d948c6a3409ee31466424a02 
  
ql/src/gen/vectorization/TestTemplates/TestColumnColumnOperationVectorExpressionCheckedEvaluation.txt
 PRE-CREATION 
  
ql/src/gen/vectorization/TestTemplates/TestColumnScalarOperationVectorExpressionCheckedEvaluation.txt
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorExpressionDescriptor.java
 bbe78c8720e16163b642f54d27fdf6b65ba9850b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java 
d46eb8d737cf2885b7f34004a9bd9eadbcf6af7a 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/LongColModuloLongColumnChecked.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/OverflowUtils.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/PosModDoubleToDouble.java
 75ec419aa9ea5c3fcc5e7314fbac756d6a5d36d5 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/PosModLongToLong.java
 6b4d714c9a79a55593c4a4d254267a3035abb10f 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorExpression.java
 710165033627b33d9b238cc847dbac36c07ee5f6 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPMinus.java 
af8552caa02f2896f393a5099abdb1ae5abd4c16 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPMod.java 
e2a638da518a2071ff15b8da6899646ec45c832a 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPMultiply.java 
99d1ad7f203d946fd89d26074bd0e00dec8b3a1a 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPNegative.java 
4e45788936559bbb7cfe65e9ffd083747b37dcc2 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPPlus.java 
b1200e673e6b470b5fd1cc856270a6da615f16cb 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestUnaryMinus.java
 ab6f6b79316818cac458390dc2d087091057c63b 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestVectorArithmeticExpressions.java
 02dec659ce421eef06f924bb6973070878d57be3 
  ql/src/test/queries/clientpositive/vectorization_numeric_overflows.q 
PRE-CREATION 
  ql/src/test/results/clientpositive/vectorization_numeric_overflows.q.out 
PRE-CREATION 
  vector-code-gen/src/org/apache/hadoop/hive/tools/GenVectorCode.java 
657ea34e11f7465e6c77d45128b298e7326a057b 
  vector-code-gen/src/org/apache/hadoop/hive/tools/GenVectorTestCode.java 
d97646f8b1c4a074da59b4685939fc4359c9c30d 


Diff: https://reviews.apache.org/r/65500/diff/2/

Changes: https://reviews.apache.org/r/65500/diff/1-2/


Testing
---


Thanks,

Vihang Karajgaonkar

Re: Intellij + Checkstyle setup

[Andrew Sherman]

I will try to make things clearer...

When debugging checkstyle it can be useful to have the IDE run the checks.
Checkstyle uses a file usually called checkstyle.xml for its configuration.
The version of checkstyle in Intellij (or in the checkstyle plugin, IDK)
does not like the checkstyle.xml that Hive uses.
(I think that Hive's checkstyle version may come from the maven plugin)
Vineet provided a version of Hive's checkstyle.xml that Intellij does like.
You can even load it directly in Intellj using the url
https://raw.githubusercontent.com/vineetgarg02/misc/master/hive_tools/checkstyle.xml

I have tried it out and it is useful (thanks Vineet).

Hopefully eventually Intellj and Hive will converge on a mutually
acceptable version of checkstyle

-Andrew



On Thu, Feb 1, 2018 at 8:37 PM, Vineet Garg  wrote:

> Hi Vihang,
>
> I am unable to import eclipse-styles.xml. I am using IntelliJ 2017.3 with
> Checkstyle 8.7. May be you are using different version?
>
> Vineet
>
> > On Feb 1, 2018, at 5:58 PM, Vihang Karajgaonkar 
> wrote:
> >
> > Thanks for sharing Vineet. How is this different than
> > https://github.com/apache/hive/blob/master/dev-support/
> eclipse-styles.xml
> >
> > I was able to import this in IntelliJ just fine.
> >
> >
> > On Thu, Feb 1, 2018 at 4:42 PM, Vineet Garg 
> wrote:
> >
> >> Hi,
> >>
> >> If you would like to use Intellij’s checkstyle plugin and are unable to
> >> import current checkstyle.xml under HIVE_REPO/checkstyle.xml file use a
> >> modified version located at:
> >> https://github.com/vineetgarg02/misc/blob/master/
> hive_tools/checkstyle.xml
> >>
> >> Current format seems to be unsupported with checkstyle plugin.
> >>
> >> Thanks,
> >> Vineet
> >>
>
>

[jira] [Created] (HIVE-18626) Repl load "with" clause does not pass config to tasks

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-18626:
-

 Summary: Repl load "with" clause does not pass config to tasks
 Key: HIVE-18626
 URL: https://issues.apache.org/jira/browse/HIVE-18626
 Project: Hive
  Issue Type: Bug
  Components: repl
Reporter: Daniel Dai
Assignee: Daniel Dai


The "with" clause in repl load suppose to pass custom hive config entries to 
replication. However, the config is only effective in BootstrapEventsIterator, 
but not the generated tasks (such as MoveTask, DDLTask).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65431: HIVE-16605 : NOT NULL constraint enforcement

[Gopal V]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65431/#review196823
---




ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEnforceNotNullConstraint.java
Lines 45 (patched)


The vectorized expressions is picked up as-is, you need to remove them to 
prevent vectorization.

FuncAbs is not a replacement for enforce_constraint.

Needs an "explain vectorization detail" qtest to verify.


- Gopal V


On Feb. 4, 2018, 2:36 a.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65431/
> ---
> 
> (Updated Feb. 4, 2018, 2:36 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Jesús Camacho Rodríguez.
> 
> 
> Bugs: HIVE-16605
> https://issues.apache.org/jira/browse/HIVE-16605
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This patch introduces support for enable NOT NULL constraint enforcement for 
> following type of statements:
> 
> * INSERT INTO
> * INSERT as SELECT
> * INSERT OVERWRITE
> * UPDATE
> * MERGE
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 99e8457c7b 
>   itests/src/test/resources/testconfiguration.properties fed9394544 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 76e85636d1 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java b1e05dffe9 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 
> 372cfad866 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> c2e24999eb 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 
> 9fcde76d21 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java e3a9e62bdb 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEnforceNotNullConstraint.java
>  PRE-CREATION 
>   ql/src/test/queries/clientnegative/alter_notnull_constraint_violation.q 
> PRE-CREATION 
>   ql/src/test/queries/clientnegative/create_with_constraints_enable.q 
> 59ebb1e9b2 
>   ql/src/test/queries/clientnegative/create_with_constraints_enforced.q  
>   ql/src/test/queries/clientnegative/create_with_fk_constraints_enforced.q 
> PRE-CREATION 
>   
> ql/src/test/queries/clientnegative/create_with_unique_constraints_enforced.q 
> PRE-CREATION 
>   ql/src/test/queries/clientnegative/insert_into_acid_notnull.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/insert_into_notnull_constraint.q 
> PRE-CREATION 
>   ql/src/test/queries/clientnegative/insert_multi_into_notnull.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/insert_overwrite_notnull_constraint.q 
> PRE-CREATION 
>   ql/src/test/queries/clientnegative/merge_constraint_notnull.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/update_notnull_constraint.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/enforce_constraint_notnull.q 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/alter_notnull_constraint_violation.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/create_with_constraints_enable.q.out  
>   ql/src/test/results/clientnegative/create_with_constraints_enforced.q.out  
>   
> ql/src/test/results/clientnegative/create_with_fk_constraints_enforced.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/insert_into_acid_notnull.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/insert_into_notnull_constraint.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/insert_multi_into_notnull.q.out 
> PRE-CREATION 
>   
> ql/src/test/results/clientnegative/insert_overwrite_notnull_constraint.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/merge_constraint_notnull.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/update_notnull_constraint.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/enforce_constraint_notnull.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/show_functions.q.out 43e4a5de39 
> 
> 
> Diff: https://reviews.apache.org/r/65431/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vineet Garg
> 
>

Re: Review Request 65413: HIVE-18575 ACID properties usage in jobconf is ambiguous for MM tables

[Eugene Koifman]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65413/#review196819
---




ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java
Lines 1306 (patched)


nit: wouldn't it be better if this removed both props if 'isTxnTable' is 
false?



ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java
Line 862 (original), 863 (patched)


is 'job' cloned somewhere between invocations?  If not this seems like a bug



ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java
Line 202 (original), 202 (patched)


on the client we have isTransactionalTable() to mean MM or full acid and 
isAcidTable() to mean full acid.

Shouldn't this be isAcidRead() to be consistent?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java
Line 507 (original), 507 (patched)


stupid RB won't let me edit my own comment...
Previsouly this check was testing for full acid but now just for 
transactional.  Why?



ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java
Line 138 (original), 138 (patched)


AcidUtils.isAcidTable() is meant to test for "full acid".  Now this code 
looks confusing.  (and I think wrong for insert_only table - will set 
isTransactional=false.  )



ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
Lines 377 (patched)


this seems to be set in the caller of setColumnTypes


- Eugene Koifman


On Jan. 29, 2018, 11:12 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65413/
> ---
> 
> (Updated Jan. 29, 2018, 11:12 p.m.)
> 
> 
> Review request for hive and Eugene Koifman.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> .
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> b7d3e99e1a505f576a06c530080fc72dddcd85ba 
>   
> hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FosterStorageHandler.java
>  5ee8aadfa774a85a0bdbcaf78a636ff6593c43e2 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  5e12614cfe17030f8fcb56ef8c83b53b8b870c97 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/mutate/StreamingAssert.java
>  c98d22be2e6216e95d9c13f3a26540ca03e7405e 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  13059023516edbb58a9129ba9aa49de7e40129e6 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
>  d252279be973201227da52d8aecf83b3fcc4656b 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  68bb168bd23b84dd150cdc4da63d73657f1b33bb 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 
> a7dace955d6fb3dabc4c5e77ef68f83617eb48d1 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java 
> 270b576199c57c109195b85d43e216743a607955 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 
> abd42ec651927503e7c8c2d9a7d3d415cc9c4ac4 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 
> eb75308e8393cadf8e69e0e30b303474b89df03e 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 
> c3b846c4d2fee8691b4952b9f6cf4dd1d8bd632f 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 
> ff2cc0455c64ed210d8ff14a9f112cd91b7314be 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java 
> 61565ef0305006a57b7f608e60ddcdf2b6ff474d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
>  da200049bcbc8f2fe1d793acc7b84f8b99ae67cc 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcInputFormat.java 
> 7b157e648646c5a199aaebf04484b81ff1c12478 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 923372d5b6da42446997051d0758e9aab4881e2e 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> 190771ea6b1cbf4b669a8919271b25a689af941b 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 
> 661446df0b9fbb5cf248d76205e47dbaa113026f 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> 9152b4e08a7a1044fc7f844d47ae8e180162b78b 
>   ql/src/test/org/apache/hadoop/hive/ql/io/TestAcidUtils.java 
> 26a96a47f1935de8e985d382b40c8aae604a9880 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
> 92f005d1dc837ea5ba7d8579892b6e7325940120 
> 

Review Request 65507: HIVE-18580: Create tests to cover exchange partitions

[Marta Kuczora via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65507/
---

Review request for hive, Peter Vary and Adam Szita.


Bugs: HIVE-18580
https://issues.apache.org/jira/browse/HIVE-18580


Repository: hive-git


Description
---

The following methods of IMetaStoreClient are covered by this test.
- int Partition exchange_partition(Map, String, String, String, 
String)
- List Partition exchange_partition(Map, String, 
String, String, String)

The test covers not just the happy pathes, but the edge cases as well.


Diffs
-

  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestExchangePartitions.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/65507/diff/1/


Testing
---

Ran the tests


Thanks,

Marta Kuczora

Re: Review Request 65478: HIVE-18553 VectorizedParquetReader fails after adding a new column to table

[Vihang Karajgaonkar via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65478/#review196718
---




ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetDataColumnReaderFactory.java
Lines 24 (patched)


Do we need to override the methods for other reads as well? What is the 
criteria to identify the methods which need to be overriden for this and 
TypesFromInt64PageReader?



ql/src/test/queries/clientpositive/schema_evol_par_vec_table.q
Lines 9 (patched)


can you please confirm that the result of this q file matches with 
non-vectorized execution?



ql/src/test/results/clientpositive/schema_evol_par_vec_table.q.out
Lines 140 (patched)


this is interesting. Do you know why this row is returned first?


- Vihang Karajgaonkar


On Feb. 5, 2018, 8:46 a.m., cheng xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65478/
> ---
> 
> (Updated Feb. 5, 2018, 8:46 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> VectorizedParquetReader throws an exception when trying to reading from a 
> parquet table on which new columns are added.
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/BaseVectorizedColumnReader.java
>  907a9b8 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetDataColumnReaderFactory.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedDummyColumnReader.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java
>  08ac57b 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestVectorizedColumnReader.java
>  9e414dc 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/VectorizedColumnReaderTestBase.java
>  5d3ebd6 
>   ql/src/test/queries/clientpositive/schema_evol_par_vec_table.q PRE-CREATION 
>   ql/src/test/results/clientpositive/schema_evol_par_vec_table.q.out 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65478/diff/2/
> 
> 
> Testing
> ---
> 
> Newly added UT passed and qtest passed locally.
> 
> 
> Thanks,
> 
> cheng xu
> 
>

[jira] [Created] (HIVE-18624) Parsing time is very high in certain complex expressions

[Amruth S (JIRA)]

Amruth S created HIVE-18624:
---

 Summary: Parsing time is very high in certain complex expressions
 Key: HIVE-18624
 URL: https://issues.apache.org/jira/browse/HIVE-18624
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 2.3.2
Reporter: Amruth S
 Attachments: thread_dump

Explain of the same query takes

0.1 to 3 seconds in hive 2.1.0 &
10-15 min in hive 2.3.2

Sample expression below
{code:java}
EXPLAIN
SELECT DISTINCT


  IF(lower('a') <= lower('a')
  ,'a'
  ,IF(('a' IS NULL AND from_unixtime(UNIX_TIMESTAMP()) <= 'a')
  ,'a'
  ,IF(if('a' = 'a', TRUE, FALSE) = 1
  ,'a'
  ,IF(('a' = 1 and lower('a') NOT IN ('a', 'a')
   and lower(if('a' = 'a','a','a')) <= lower('a'))
  OR ('a' like 'a' OR 'a' like 'a')
  OR 'a' in ('a','a')
  ,'a'
  ,IF(if(lower('a') in ('a', 'a') and 'a'='a', TRUE, FALSE) = 1
  ,'a'
  ,IF('a'='a' and unix_timestamp(if('a' = 'a',cast('a' as 
string),coalesce('a',cast('a' as string),from_unixtime(unix_timestamp() <= 
unix_timestamp(concat_ws('a',cast(lower('a') as string),'00:00:00')) + 9*3600
  ,'a'

  ,If(lower('a') <= lower('a')
  and if(lower('a') in ('a', 'a') and 'a'<>'a', TRUE, FALSE) <> 1
  ,'a'
  ,IF('a'=1 AND 'a'=1
  ,'a'
  ,IF('a' = 1 and COALESCE(cast('a' as int),0) = 0
  ,'a'
  ,IF('a' = 'a'
  ,'a'

  ,If('a' = 'a' AND 
lower('a')>lower(if(lower('a')<1830,'a',cast(date_add('a',1) as timestamp)))
  ,'a'



  ,IF('a' = 1

  ,IF('a' in ('a', 'a') and ((unix_timestamp('a')-unix_timestamp('a')) / 60) > 
30 and 'a' = 1


  ,'a', 'a')


  ,IF(if('a' = 'a', FALSE, TRUE ) = 1 AND 'a' IS NULL
  ,'a'
  ,IF('a' = 1 and 'a'>0
  , 'a'

  ,IF('a' = 1 AND 'a' ='a'
  ,'a'
  ,IF('a' is not null and 'a' is not null and 'a' > 'a'
  ,'a'
  ,IF('a' = 1
  ,'a'

  ,IF('a' = 'a'
  ,'a'

  ,If('a' = 1
  ,'a'
  ,IF('a' = 1
  ,'a'
  ,IF('a' = 1
  ,'a'

  ,IF('a' ='a' and 'a' ='a' and cast(unix_timestamp('a') as  int) + 93600 < 
cast(unix_timestamp()  as int)
  ,'a'
  ,IF('a' = 'a'
  ,'a'
  ,IF('a' = 'a' and 'a' in ('a','a','a')
  ,'a'
  ,IF('a' = 'a'
  ,'a','a'))
  )))
AS test_comp_exp
{code}
 

Taking a look at [^thread_dump] shows a very large function stack getting 
created.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18623) Hive throws an exception "Renames across Mount points not supported" when running in a federated cluster

[yangfang (JIRA)]

yangfang created HIVE-18623:
---

 Summary: Hive throws an exception "Renames across Mount points not 
supported" when running in a federated cluster
 Key: HIVE-18623
 URL: https://issues.apache.org/jira/browse/HIVE-18623
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Affects Versions: 2.3.2, 2.3.1, 2.3.0, 2.2.0
 Environment: hadoop 2.7.5, HDFS Federation enabled
hive 3.0.0
Reporter: yangfang
Assignee: yangfang


 

I run a sql query in in a federated cluster and I have two namespaces: 
nameservice and nameservice1. I set hive.exec.stagingdir=/nameservice1/hive_tmp 
in hive-site.xml and my data tables are located in the directory of 
nameservice, then I got the exception as below:
hive> create external table test_par6(id int,name string) partitioned by(p int);
OK
Time taken: 1.527 seconds
hive> insert into table test_par6 partition(p = 1) values(1,'Jack');
Moving data to directory 
viewfs://nsX/nameservice1/hive_tmp_hive_2018-02-05_14-09-36_416_3075179128063595297-1/-ext-1
Loading data to table default.test_par6 partition (p=1)
Failed with exception java.io.IOException: Renames across Mount points not 
supported
FAILED: Execution Error, return code 1 from 
org.apache.hadoop.hive.ql.exec.MoveTask. java.io.IOException: Renames across 
Mount points not supported
MapReduce Jobs Launched: 
Stage-Stage-1: Map: 1 Cumulative CPU: 2.08 sec HDFS Read: 3930 HDFS Write: 7 
SUCCESS
Total MapReduce CPU Time Spent: 2 seconds 80 msec



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18622) Vectorization: IF statement, Comparisons, and more do not handle NULLs correctly

[Matt McCline (JIRA)]

Matt McCline created HIVE-18622:
---

 Summary: Vectorization: IF statement, Comparisons, and more do not 
handle NULLs correctly
 Key: HIVE-18622
 URL: https://issues.apache.org/jira/browse/HIVE-18622
 Project: Hive
  Issue Type: Bug
  Components: Hive
Reporter: Matt McCline
Assignee: Matt McCline
 Fix For: 3.0.0


 
Many vector expression classes are missing guards around setting noNulls among 
other things.

{code:java}
// Carefully update noNulls...
if (outputColVector.noNulls) {
  outputColVector.noNulls = inputColVector.noNulls;
}
 {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65478: HIVE-18553 VectorizedParquetReader fails after adding a new column to table

[cheng xu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65478/
---

(Updated Feb. 5, 2018, 4:46 p.m.)


Review request for hive.


Repository: hive-git


Description
---

VectorizedParquetReader throws an exception when trying to reading from a 
parquet table on which new columns are added.


Diffs (updated)
-

  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/BaseVectorizedColumnReader.java
 907a9b8 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetDataColumnReaderFactory.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedDummyColumnReader.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java
 08ac57b 
  
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestVectorizedColumnReader.java
 9e414dc 
  
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/VectorizedColumnReaderTestBase.java
 5d3ebd6 
  ql/src/test/queries/clientpositive/schema_evol_par_vec_table.q PRE-CREATION 
  ql/src/test/results/clientpositive/schema_evol_par_vec_table.q.out 
PRE-CREATION 


Diff: https://reviews.apache.org/r/65478/diff/2/

Changes: https://reviews.apache.org/r/65478/diff/1-2/


Testing
---

Newly added UT passed and qtest passed locally.


Thanks,

cheng xu