[jira] [Created] (HIVE-18067) Remove extraneous golden files

2017-11-14 Thread Ashutosh Chauhan (JIRA) 
Ashutosh Chauhan created HIVE-18067:
---

 Summary: Remove extraneous golden files
 Key: HIVE-18067
 URL: https://issues.apache.org/jira/browse/HIVE-18067
 Project: Hive
  Issue Type: Bug
  Components: Tests
Affects Versions: 3.0.0
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan


TestDanglingQouts makes sure that there are no unneeded files in repo. This is 
currently failing. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 63711: HIVE-17528 Add more q-tests for Hive-on-Spark with Parquet vectorized reader

2017-11-14 Thread cheng xu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63711/
---

(Updated Nov. 15, 2017, 9:34 a.m.)


Review request for hive and Vihang Karajgaonkar.


Repository: hive-git


Description
---

Most of the vectorization related q-tests operate on ORC tables using Tez. It 
would be good to add more coverage on a different combination of engine and 
file-format. We can model existing q-tests using parquet tables and run it 
using TestSparkCliDriver


Diffs (updated)
-

  data/scripts/q_test_cleanup.sql 4620dcd 
  data/scripts/q_test_init.sql f763c12 
  itests/src/test/resources/testconfiguration.properties 1d16b65 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java f1d90ff 
  pom.xml dfb29ce 
  ql/src/test/queries/clientpositive/parquet_read_backward_compatible_files.q 
0abbc2f 
  ql/src/test/queries/clientpositive/parquet_vectorization_0.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_1.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_10.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_11.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_12.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_13.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_14.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_15.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_16.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_17.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_2.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_3.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_4.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_5.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_6.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_7.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_8.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_9.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_decimal_date.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_div0.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_limit.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_nested_udf.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_not.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_offset_limit.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_part.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_part_project.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_part_varchar.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_pushdown.q 
PRE-CREATION 
  ql/src/test/results/clientpositive/add_part_exist.q.out f8d50ca 
  ql/src/test/results/clientpositive/alter1.q.out c2efbe5 
  ql/src/test/results/clientpositive/alter2.q.out 18032ac 
  ql/src/test/results/clientpositive/alter3.q.out 3bd7288 
  ql/src/test/results/clientpositive/alter4.q.out ddcb0ed 
  ql/src/test/results/clientpositive/alter5.q.out 1eb24c2 
  ql/src/test/results/clientpositive/alter_index.q.out bca4e12 
  ql/src/test/results/clientpositive/alter_rename_partition.q.out 5702d39 
  ql/src/test/results/clientpositive/authorization_9.q.out 39e0a56 
  ql/src/test/results/clientpositive/authorization_show_grant.q.out d0fed81 
  ql/src/test/results/clientpositive/cte_5.q.out ac0dac0 
  ql/src/test/results/clientpositive/cte_mat_4.q.out bc92f14 
  ql/src/test/results/clientpositive/cte_mat_5.q.out 3747cec 
  ql/src/test/results/clientpositive/describe_table_json.q.out ebf15ac 
  ql/src/test/results/clientpositive/drop_table_with_index.q.out 3175cdb 
  ql/src/test/results/clientpositive/index_creation.q.out ca113ef 
  ql/src/test/results/clientpositive/input2.q.out 4dac88a 
  ql/src/test/results/clientpositive/input3.q.out 8316a46 
  ql/src/test/results/clientpositive/llap/cte_5.q.out dad6cd8 
  ql/src/test/results/clientpositive/llap/cte_mat_4.q.out 67910c5 
  ql/src/test/results/clientpositive/llap/cte_mat_5.q.out 92d8601 
  ql/src/test/results/clientpositive/llap/jdbc_handler.q.out 6e0451f 
  ql/src/test/results/clientpositive/llap/resourceplan.q.out 5b5d8f1 
  ql/src/test/results/clientpositive/llap/sysdb.q.out 9813636 
  ql/src/test/results/clientpositive/llap/temp_table.q.out 4cf47b5 
  
ql/src/test/results/clientpositive/parquet_read_backward_compatible_files.q.out 
22a050a 
  ql/src/test/results/clientpositive/parquet_vectorization_0.q.out

[jira] [Created] (HIVE-18066) Unable to insert data into S3 backed table from Hive

2017-11-14 Thread Shreyas Joshi (JIRA) 
Shreyas Joshi created HIVE-18066:


 Summary: Unable to insert data into S3 backed table from Hive
 Key: HIVE-18066
 URL: https://issues.apache.org/jira/browse/HIVE-18066
 Project: Hive
  Issue Type: Improvement
  Components: Hive
 Environment: Hive 1.2.0
Hadoop 2.6.0
HDFS
AWS- S3A
Debian jessie
Reporter: Shreyas Joshi


I am trying to create a table and insert data into a table on S3 from Hive like:


{code:sql}
create table my_db.test_table (
userid int
) location   's3a://username:pass@bucket/s3-test/test-table'
{code}

This works, but when I try inserting data into this table like:

{code:sql}
INSERT INTO my_db.test_table VALUES (1);
{code}

It fails, and here's what I see:

{code}
hive> insert into test values (1);
-chgrp: '' does not match expected pattern for group
Usage: hadoop fs [generic options] -chgrp [-R] GROUP PATH...
Failed with exception Wrong FS: 
s3a://username:pass@bucket/s3-test/test-table/.hive-staging_hive_2017-11-13_17-37-21_376_4109060838187898328-1/-ext-10002,
 expected: hdfs://:
FAILED: Execution Error, return code 1 from 
org.apache.hadoop.hive.ql.exec.MoveTask
{code}

The {{hdfs://:}} in the error seems to be coming from the 
{{fs.default.name}} setting in {{core-site.xml}}

If I insert data into the table via other other means, I can successfully query 
the table from Hive. So, insertion not selection seems to be the problem. What 
might be happening here?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-18065) Exception when pushing postaggregates into Druid

2017-11-14 Thread Jesus Camacho Rodriguez (JIRA) 
Jesus Camacho Rodriguez created HIVE-18065:
--

 Summary: Exception when pushing postaggregates into Druid
 Key: HIVE-18065
 URL: https://issues.apache.org/jira/browse/HIVE-18065
 Project: Hive
  Issue Type: Bug
  Components: Druid integration
Affects Versions: 3.0.0
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez


After Calcite is upgraded to 1.14 and the rule to push post-aggregations to 
Druid is enabled, the following query will fail:
{code}
EXPLAIN
SELECT language, robot, sum(added) - sum(delta) AS a
FROM druid_table_1
WHERE extract (week from `__time`) IN (10,11)
  AND robot='Bird Call'
GROUP BY language, robot;
{code}

The error we get is the following:
{code}
Cannot add expression of different type to set:
set type is RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" COLLATE 
"ISO-8859-1$en_US$primary" language, VARCHAR(2147483647) CHARACTER SET 
"UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" robot, DOUBLE a) NOT NULL
expression type is RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" 
COLLATE "ISO-8859-1$en_US$primary" language, DOUBLE postagg#0) NOT NULL
set is 
rel#1507:HiveProject.HIVE.[](input=HepRelVertex#1514,language=$0,robot=CAST(_UTF-16LE'Bird
 Call'):VARCHAR(2147483647) CHARACTER SET "UTF-16LE" COLLATE 
"ISO-8859-1$en_US$primary",a=-($1, $2))
expression is DruidQuery#1516
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-18064) Hive on Tez parallel order by

2017-11-14 Thread Zhiyuan Yang (JIRA) 
Zhiyuan Yang created HIVE-18064:
---

 Summary: Hive on Tez parallel order by
 Key: HIVE-18064
 URL: https://issues.apache.org/jira/browse/HIVE-18064
 Project: Hive
  Issue Type: Bug
Reporter: Zhiyuan Yang
Assignee: Zhiyuan Yang


We've built parallel sorting in TEZ-3837. It does sampling as output is 
generated and figure out a range partitioner for shuffle edge. Each reducer 
output a sorted span. This is mainly for external consumption since output 
files need to be read in certain order.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 63782: HIVE-18002 add group support for pool mappings

2017-11-14 Thread j . prasanth . j 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63782/#review191004
---


Ship it!




Ship It!

- Prasanth_J


On Nov. 14, 2017, 1:22 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63782/
> ---
> 
> (Updated Nov. 14, 2017, 1:22 a.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java 3bb4f585ce 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/UserPoolMapping.java 
> 50cf4daeed 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 
> a73a24aef3 
> 
> 
> Diff: https://reviews.apache.org/r/63782/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Review Request 63806: HIVE-16756 : Vectorization: LongColModuloLongColumn throws java.lang.ArithmeticException: divide by zero

2017-11-14 Thread Vihang Karajgaonkar via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63806/
---

Review request for hive, Aihua Xu and Matt McCline.


Bugs: HIVE-16756
https://issues.apache.org/jira/browse/HIVE-16756


Repository: hive-git


Description
---

HIVE-16756 : Vectorization: LongColModuloLongColumn throws 
java.lang.ArithmeticException: divide by zero


Diffs
-

  ql/src/gen/vectorization/ExpressionTemplates/ColumnDivideColumn.txt 
8b586b1f00ce7d6081f973a5736100d8941f79bc 
  ql/src/gen/vectorization/ExpressionTemplates/ColumnDivideLong.txt 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/LongColDivideLongColumn.java
 446c033ed473adfcfa0db630ceb716aecb623612 
  ql/src/test/queries/clientpositive/vectorization_div0.q 
025d457807dd0642965a81c6b093e421c4acd0f8 
  ql/src/test/results/clientpositive/spark/vectorization_div0.q.out 
631b0723fb0d7ab011ad2bfd7be4b33d11d76b1c 
  ql/src/test/results/clientpositive/tez/vectorization_div0.q.out 
6c3354cb4a8cd439d86df7e6b0cf759ea4c04cd0 
  ql/src/test/results/clientpositive/vectorization_div0.q.out 
97f1687b85193e681f26c61107a6d9266c1d87a2 
  vector-code-gen/src/org/apache/hadoop/hive/tools/GenVectorCode.java 
e58d4e91938dc266111042fe98b05a3d9c6fc5e9 


Diff: https://reviews.apache.org/r/63806/diff/1/


Testing
---


Thanks,

Vihang Karajgaonkar

Review Request 63802: HIVE-14495

2017-11-14 Thread Jesús Camacho Rodríguez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63802/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-14495
https://issues.apache.org/jira/browse/HIVE-14495


Repository: hive-git


Description
---

HIVE-14495


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
0a34633fa44f6913c2af3703ae3dfeff5419170a 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
579f2df280fcb7e267bacab81253324698aaf34f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 
0bbd9be40689dd73ac9e7c563f62b3923047f622 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 
e704c731121f683c733ebc48d387ce5c9499c6ec 
  ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java 
1ce1c76f37a88a5accd1f205cad1517fa42dc61b 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveOperationType.java
 ba1d01f3c62e22f7c7bfd7e5ba7a0213ab17a662 
  ql/src/test/queries/clientpositive/show_materialized_views.q PRE-CREATION 
  ql/src/test/results/clientpositive/show_materialized_views.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/63802/diff/1/


Testing
---


Thanks,

Jesús Camacho Rodríguez

Re: Review Request 55816: HIVE-15680: Incorrect results when hive.optimize.index.filter=true and same ORC table is referenced twice in query

2017-11-14 Thread j . prasanth . j 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55816/#review190971
---




ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java
Lines 720 (patched)


what is only subset of aliases does not have filters and remaining subset 
has filters? this looks like if atleast one does not have filters, no filters 
are pushed.


- Prasanth_J


On Jan. 31, 2017, 2:43 a.m., Anthony Hsu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55816/
> ---
> 
> (Updated Jan. 31, 2017, 2:43 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-15680
> https://issues.apache.org/jira/browse/HIVE-15680
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-15680: Incorrect results when hive.optimize.index.filter=true and same 
> ORC table is referenced twice in query
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 
> 68dd5e7247415dec1e353010ea34481c4f2fc6cd 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 
> 51530ac16c92cc75d501bfcb573557754ba0c964 
>   ql/src/test/queries/clientpositive/orc_ppd_same_table_multiple_aliases.q 
> PRE-CREATION 
>   
> ql/src/test/results/clientpositive/orc_ppd_same_table_multiple_aliases.q.out 
> PRE-CREATION 
>   serde/src/java/org/apache/hadoop/hive/serde2/ColumnProjectionUtils.java 
> 1354680584305bc7ea928526160f08fc9cbfd73e 
> 
> 
> Diff: https://reviews.apache.org/r/55816/diff/6/
> 
> 
> Testing
> ---
> 
> Added qtest.
> 
> 
> Thanks,
> 
> Anthony Hsu
> 
>

[jira] [Created] (HIVE-18063) Make CommandProcessorResponse an exception instead of a return class

2017-11-14 Thread Zoltan Haindrich (JIRA) 
Zoltan Haindrich created HIVE-18063:
---

 Summary: Make CommandProcessorResponse an exception instead of a 
return class
 Key: HIVE-18063
 URL: https://issues.apache.org/jira/browse/HIVE-18063
 Project: Hive
  Issue Type: Sub-task
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich


the usage pattern of the {{CommandProcessorResponse}} class suggests that its 
current role is closer to Exceptions than to return values.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 63442: HIVE-17934 Merging Statistics are promoted to COMPLETE (most of the time)

2017-11-14 Thread Zoltan Haindrich 


> On Nov. 9, 2017, 7:51 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/llap/auto_sortmerge_join_12.q.out
> > Line 160 (original), 160 (patched)
> > 
> >
> > bucket_small has no stats gathered. This should be NONE.
> 
> Zoltan Haindrich wrote:
> `hive.stats.autogather` is enabled by default from `HiveConf`
> 
> Ashutosh Chauhan wrote:
> Those are load statements, not inserts. We don't gather stats with load 
> statements only with insets.
> 
> Zoltan Haindrich wrote:
> sorry, you are right: basic stats are not gathered in this case in any 
> way.
> 
> But the stat state is complete; because: there is logic which scans the 
> file sizes - to calculate the datasizes; and from there HIVE-16811 can guess 
> some row counts
> 
> 
> https://github.com/kgyrtkirk/hive/blob/9f67a878512117eb5c251794adc1a91bae62fea7/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L386-L393
> 
> Firts I would like to make the standalone table/partitioned table's 
> calculation-s are a bit more similar to eachother
> 
> I've tried to come up with some definitions for NONE/PARTIAL/COMPLETE; 
> currently I would say the following:
> 
> * NONE: not known
> * on table: no information (afaik currently this can't happen)
> * estimation tree: all nodes in the estimation tree were NONE
> * PARTIAL:
> * on table: the current information is estimated from data size
> * estimation tree: contains at least one NONE/PARTIAL
> * COMPLETE:
> * current information is correct (calculated by statstask-s)
> * estimation tree: the whole subtree has COMPLETE status
> 
> If I use these definitions; then I would say that the filesystem size 
> based estimation should be considered PARTIAL.
> 
> Ashutosh Chauhan wrote:
> Definitions sounds good. Lets use them to make sure our state calculation 
> logic is built on it.
> Can you also add this in code comments.

I've opened HIVE-18062 to address these problems


- Zoltan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63442/#review190633
---


On Nov. 9, 2017, 5:39 p.m., Zoltan Haindrich wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63442/
> ---
> 
> (Updated Nov. 9, 2017, 5:39 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-17934
> https://issues.apache.org/jira/browse/HIVE-17934
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> * remove the reactive stat state guessing method
> * make the guessing only work when a new object is created
> * change the way stat objects are merged
> 
> this patch will most probably break almost all qtest outputs
> 
> 
> Diffs
> -
> 
>   accumulo-handler/src/test/results/positive/accumulo_queries.q.out 
> b3adf4e504 
>   hbase-handler/src/test/results/positive/hbase_queries.q.out b2eda12e95 
>   hbase-handler/src/test/results/positive/hbasestats.q.out 29eefd43a9 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkMapJoinOptimizer.java
>  7a3fae65e8 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
>  a4f60accce 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java 8ffb4ce44b 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java ce7c96c639 
>   ql/src/test/queries/clientpositive/lateral_view_onview2.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/stats_empty_partition2.q PRE-CREATION 
>   ql/src/test/results/clientpositive/acid_table_stats.q.out 351ff0da0a 
>   ql/src/test/results/clientpositive/alterColumnStatsPart.q.out 858e16fe22 
>   ql/src/test/results/clientpositive/annotate_stats_part.q.out 3a94a6a4e3 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out 7875e9693a 
>   ql/src/test/results/clientpositive/cbo_const.q.out e9f885b363 
>   ql/src/test/results/clientpositive/cbo_input26.q.out 77fc194829 
>   ql/src/test/results/clientpositive/columnstats_partlvl_dp.q.out 414b715b7a 
>   ql/src/test/results/clientpositive/columnstats_quoting.q.out 683c1e274f 
>   ql/src/test/results/clientpositive/columnstats_tbllvl.q.out a2c6ead293 
>   ql/src/test/results/clientpositive/constGby.q.out c633624935 
>   ql/src/test/results/clientpositive/constant_prop_3.q.out cba4744866 
>   ql/src/test/results/clientpositive/constprog3.q.out f54168d0ee 
>   ql/src/test/results/clientpositive/correlationoptimizer10.q.out a03acd38a7 
>   ql/src/test/results/clientpositive/correlationoptimizer11.q.out cf2250790a 
>

[jira] [Created] (HIVE-18062) Revise basic stat states for estimations

2017-11-14 Thread Zoltan Haindrich (JIRA) 
Zoltan Haindrich created HIVE-18062:
---

 Summary: Revise basic stat states for estimations
 Key: HIVE-18062
 URL: https://issues.apache.org/jira/browse/HIVE-18062
 Project: Hive
  Issue Type: Sub-task
  Components: Statistics
Reporter: Zoltan Haindrich


basic stat states might be misleading; because currently estimations also get 
the *COMPLETE* qualifier in most cases..

proposed definitions for the states:

* {{NONE}}
   ** on table: no information (afaik currently this can't happen)
   ** estimation tree: all nodes in the estimation tree were NONE
* {{PARTIAL}}:
   ** on table: the current information is estimated from data size
   ** estimation tree: contains at least one NONE/PARTIAL; and at least 1 
PARTIAL
* {{COMPLETE}}:
  ** current information is correct (calculated by statstask-s)
   ** estimation tree: the whole subtree has COMPLETE status

document/change states to comply with the above definitions.

followup of HIVE-17934; 




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 63711: HIVE-17528 Add more q-tests for Hive-on-Spark with Parquet vectorized reader

2017-11-14 Thread cheng xu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63711/
---

(Updated Nov. 14, 2017, 11:30 p.m.)


Review request for hive and Vihang Karajgaonkar.


Repository: hive-git


Description
---

Most of the vectorization related q-tests operate on ORC tables using Tez. It 
would be good to add more coverage on a different combination of engine and 
file-format. We can model existing q-tests using parquet tables and run it 
using TestSparkCliDriver


Diffs (updated)
-

  data/scripts/q_test_cleanup.sql 4620dcd 
  data/scripts/q_test_init.sql f763c12 
  itests/src/test/resources/testconfiguration.properties 1d16b65 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java f1d90ff 
  pom.xml dfb29ce 
  ql/src/test/queries/clientpositive/parquet_read_backward_compatible_files.q 
0abbc2f 
  ql/src/test/queries/clientpositive/parquet_vectorization_0.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_1.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_10.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_11.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_12.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_13.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_14.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_15.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_16.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_17.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_2.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_3.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_4.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_5.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_6.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_7.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_8.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_9.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_decimal_date.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_div0.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_limit.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_nested_udf.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_not.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_offset_limit.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_part.q PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_part_project.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_part_varchar.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_vectorization_pushdown.q 
PRE-CREATION 
  ql/src/test/results/clientpositive/add_part_exist.q.out f8d50ca 
  ql/src/test/results/clientpositive/alter1.q.out c2efbe5 
  ql/src/test/results/clientpositive/alter2.q.out 18032ac 
  ql/src/test/results/clientpositive/alter3.q.out 3bd7288 
  ql/src/test/results/clientpositive/alter4.q.out ddcb0ed 
  ql/src/test/results/clientpositive/alter5.q.out 1eb24c2 
  ql/src/test/results/clientpositive/alter_index.q.out bca4e12 
  ql/src/test/results/clientpositive/alter_rename_partition.q.out 5702d39 
  ql/src/test/results/clientpositive/authorization_9.q.out 39e0a56 
  ql/src/test/results/clientpositive/authorization_show_grant.q.out d0fed81 
  ql/src/test/results/clientpositive/cte_5.q.out ac0dac0 
  ql/src/test/results/clientpositive/cte_mat_4.q.out bc92f14 
  ql/src/test/results/clientpositive/cte_mat_5.q.out 3747cec 
  ql/src/test/results/clientpositive/describe_table_json.q.out ebf15ac 
  ql/src/test/results/clientpositive/drop_table_with_index.q.out 3175cdb 
  ql/src/test/results/clientpositive/index_creation.q.out ca113ef 
  ql/src/test/results/clientpositive/input2.q.out 4dac88a 
  ql/src/test/results/clientpositive/input3.q.out 8316a46 
  ql/src/test/results/clientpositive/llap/cte_5.q.out dad6cd8 
  ql/src/test/results/clientpositive/llap/cte_mat_4.q.out 67910c5 
  ql/src/test/results/clientpositive/llap/cte_mat_5.q.out 92d8601 
  
ql/src/test/results/clientpositive/llap/insert_values_orig_table_use_metadata.q.out
 92c4eaf 
  ql/src/test/results/clientpositive/llap/jdbc_handler.q.out 6e0451f 
  ql/src/test/results/clientpositive/llap/resourceplan.q.out 5b5d8f1 
  ql/src/test/results/clientpositive/llap/sysdb.q.out 9813636 
  ql/src/test/results/clientpositive/llap/temp_table.q.out 4cf47b5

[jira] [Created] (HIVE-18061) q.outs: be more selective with masikng hdfs paths

2017-11-14 Thread Zoltan Haindrich (JIRA) 
Zoltan Haindrich created HIVE-18061:
---

 Summary: q.outs: be more selective with masikng hdfs paths
 Key: HIVE-18061
 URL: https://issues.apache.org/jira/browse/HIVE-18061
 Project: Hive
  Issue Type: Improvement
Reporter: Zoltan Haindrich



currently any line which contains a path which looks like an hdfs location is 
replaced with a "masked pattern was here"...

it might be releavant to record these messages; since even an exception message 
might contain an hdfs location

noticed in
HIVE-18012




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-18060) UpdateInputAccessTimeHook fails for non-current database

2017-11-14 Thread Oleksiy Sayankin (JIRA) 
Oleksiy Sayankin created HIVE-18060:
---

 Summary: UpdateInputAccessTimeHook fails for non-current database
 Key: HIVE-18060
 URL: https://issues.apache.org/jira/browse/HIVE-18060
 Project: Hive
  Issue Type: Bug
Reporter: Oleksiy Sayankin
Assignee: Oleksiy Sayankin


Steps to reproduce:

*STEP 1. Create DBs and tables*

{code}
hive> create database temp;
hive> use temp;
hive> create table test(id int);
hive> create database temp2;
hive> use temp2;
hive> create table test2(id int);
{code}

*STEP 2. Set {{hive.exec.pre.hooks}}*

{code}
hive> set 
hive.exec.pre.hooks=org.apache.hadoop.hive.ql.hooks.UpdateInputAccessTimeHook$PreExec;
{code}

*STEP 3. Use {{desc}}*

{code}
hive> use temp;
hive> desc temp2.test2;
{code}

*EXPECTED RESULT*

Code works fine and shows table info

*ACTUAL RESULT*

{code}
FAILED: Hive Internal Error: 
org.apache.hadoop.hive.ql.metadata.InvalidTableException(Table not found test2)
org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found test2
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1258)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1209)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1196)
at 
org.apache.hadoop.hive.ql.hooks.UpdateInputAccessTimeHook$PreExec.run(UpdateInputAccessTimeHook.java:61)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1688)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1454)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1172)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1162)
at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:234)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:185)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:401)
at 
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:791)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:729)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:652)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:647)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-18059) remove unused hiveconf variables

2017-11-14 Thread Zoltan Haindrich (JIRA) 
Zoltan Haindrich created HIVE-18059:
---

 Summary: remove unused hiveconf variables
 Key: HIVE-18059
 URL: https://issues.apache.org/jira/browse/HIVE-18059
 Project: Hive
  Issue Type: Improvement
Reporter: Zoltan Haindrich


* for example hive.debug.localtask is there...but it seems like its not used 
anywhere
* there might be more conf variables which are just hanging there without 
purpose



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-18058) remove old UDF / UDAF

2017-11-14 Thread Zoltan Haindrich (JIRA) 
Zoltan Haindrich created HIVE-18058:
---

 Summary: remove old UDF / UDAF
 Key: HIVE-18058
 URL: https://issues.apache.org/jira/browse/HIVE-18058
 Project: Hive
  Issue Type: Improvement
Reporter: Zoltan Haindrich


* I think these classes are in the shadow zone since 2014
* the preferred way is to use genericudf instead of them



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-18057) remove PostExecute / PreExecute hook support

2017-11-14 Thread Zoltan Haindrich (JIRA) 
Zoltan Haindrich created HIVE-18057:
---

 Summary: remove PostExecute / PreExecute hook support
 Key: HIVE-18057
 URL: https://issues.apache.org/jira/browse/HIVE-18057
 Project: Hive
  Issue Type: Improvement
Reporter: Zoltan Haindrich


* deprecated since 2010
* they are needlessly complicate the dispatch logic
* the current dispatch logic just silently accepts pre/post hooks if they don't 
implement neither interface



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)