[jira] [Created] (HIVE-19919) HiveServer2 - expose queryable data dictionary (ie Oracles' ALL_TAB_COLUMNS)

2018-06-15 Thread t oo (JIRA) 
t oo created HIVE-19919:
---

 Summary: HiveServer2 - expose queryable data dictionary (ie 
Oracles' ALL_TAB_COLUMNS)
 Key: HIVE-19919
 URL: https://issues.apache.org/jira/browse/HIVE-19919
 Project: Hive
  Issue Type: New Feature
  Components: SQL
Affects Versions: 2.3.2, 3.0.0
Reporter: t oo


All major db vendors have a table like information_schema.columns, 
all_tab_columns or syscolumns containing table_name,column_name, data_type, 
col_order. Adding this feature to HiveServer2 would be very convenient for 
users.

This information is currently only available in the mysql metastore ie TBLS, 
COLS but should be exposed up into the HiveServer2 1 port connection. Thus 
saving users from having 2 connections (1 to see data, 1 to see metadata). For 
security reason too, mysql can be firewalled from end-users.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19920) Schematool fails in embedded mode when auth is on

2018-06-15 Thread Daniel Dai (JIRA) 
Daniel Dai created HIVE-19920:
-

 Summary: Schematool fails in embedded mode when auth is on
 Key: HIVE-19920
 URL: https://issues.apache.org/jira/browse/HIVE-19920
 Project: Hive
  Issue Type: Bug
  Components: Standalone Metastore
Reporter: Daniel Dai
Assignee: Daniel Dai


This is a follow up of HIVE-19775. We need to override more properties in 
embedded hs2.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19910) hive.spark.log.dir isn't honored for TestSparkCliDriver

2018-06-15 Thread Sahil Takiar (JIRA) 
Sahil Takiar created HIVE-19910:
---

 Summary: hive.spark.log.dir isn't honored for TestSparkCliDriver
 Key: HIVE-19910
 URL: https://issues.apache.org/jira/browse/HIVE-19910
 Project: Hive
  Issue Type: Sub-task
  Components: Spark
Reporter: Sahil Takiar


I haven't actually confirmed this, but I don't think {{hive.spark.log.dir}} is 
getting honored for any Spark test that sets {{spark.master}} to 
{{local-cluster}} because it adds {{hive.spark.log.dir}} as a system property 
via the {{spark.driver.extraJavaOptions}} configuration, but according to the 
Spark docs passing system properties via this parameter doesn't work in client 
mode, users have to use {{--driver-java-options}} instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19909) qtests: retire hadoop_major version specific tests; and logics

2018-06-15 Thread Zoltan Haindrich (JIRA) 
Zoltan Haindrich created HIVE-19909:
---

 Summary: qtests: retire hadoop_major version specific tests; and 
logics
 Key: HIVE-19909
 URL: https://issues.apache.org/jira/browse/HIVE-19909
 Project: Hive
  Issue Type: Sub-task
  Components: Tests
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich


these are just hanging around in QTestUtil...they don't even in use...since 
they are looking for hadoop versions like  "0.23" and "0.20" ...



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 67468: HIVE-18118: provide supportability support for Erasure Coding Update number of Erasure Coded Files in a directory as part of Basic (aka Quick) Stats This information is then

2018-06-15 Thread Sahil Takiar 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67468/#review204836
---



Why not show the #of EC files for regular explain plans too? To decrease the # 
of q file updates, it can be omitted if the # of EC files = 0


standalone-metastore/src/main/java/org/apache/hadoop/hive/common/StatsSetupConst.java
Line 116 (original), 122 (patched)


why change this from an array to a list?


- Sahil Takiar


On June 6, 2018, 12:46 a.m., Andrew Sherman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67468/
> ---
> 
> (Updated June 6, 2018, 12:46 a.m.)
> 
> 
> Review request for hive and Sahil Takiar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18118: provide supportability support for Erasure Coding 
> [NOTE THIS REVIEW INITIALLY OMITS 200+ .q.out changes]
> Update number of Erasure Coded Files in a directory as part of Basic (aka 
> Quick) Stats 
> This information is then (mostly) available through 'EXPLAIN EXTENDED' and 
> 'DESCRIBE EXTENDED' 
> Extend the MiniHS2 Builder to allow configuring the number of datanodes. 
> Add a jdbc MiniHS2/Spark test that uses Erasure Coding. 
> There are some change to StatsSetupConst to make checkstyle happy.
> 
> 
> Diffs
> -
> 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
> d7d7097336fc6be4c2f7a35cd6897e0375486e81 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2ErasureCoding.java
>  PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties 
> 463fda1913f6d5b928fcee038f19e124b0239e96 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 2365fb76bd08f3a310e81ac3a19ca64971aeec8e 
>   itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 
> 1700c08d3f37285de43b5d4fe5c77ef55c170235 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
> e06949928d179cfd9a4dcb7176203b885509 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
>  209fdfb287cabc5bb7cab2117d771f7907deb2b9 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStatsNoJobTask.java 
> d4d46a3671efdaaed32f63b7262b963cce00b94e 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStatsTask.java 
> 8c238871765b0d5312a459a0e7f68c81f3837c13 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> 982b18076180ba300094f30a7f87f025f993b265 
>   ql/src/test/queries/clientpositive/erasure_explain.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/erasure_simple.q 
> c08409c17787417b986d90a43104f5ddd456e600 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_explain.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_simple.q.out 
> 01f6015a346c1e4283fd6a8cf1eaa3b670450e20 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/common/StatsSetupConst.java
>  78ea01d9687fe043d63441430c46b30c25cd9756 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  77ed2b4de4569fa8aca23b16f2b362b187c7c4fc 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
>  9b36d09eb9fb332e913d442bb476628eca334b6e 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/utils/TestMetaStoreUtils.java
>  55ff1502d415dea52095cfdd523d01f1e49ce084 
> 
> 
> Diff: https://reviews.apache.org/r/67468/diff/1/
> 
> 
> Testing
> ---
> 
> Ran driver tests and new jdbc test
> 
> 
> Thanks,
> 
> Andrew Sherman
> 
>

Re: Review Request 65480: HIVE-18545 Add UDF to parse complex types from json

2018-06-15 Thread Zoltan Haindrich 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65480/
---

(Updated June 15, 2018, 2:12 p.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

use the same in serde/udf; some refactor


Bugs: HIVE-18545
https://issues.apache.org/jira/browse/HIVE-18545


Repository: hive-git


Description (updated)
---

* add a generalized reader for json2hive struct parsing
* add json_read udf ; tests
* rename the new "copied" the new jsonserde "serde2.JsonSerde" to 
"serde2.json.JsonSerde2" - I think it would be bad to have 2 classes named the 
same ; in the same project, it would even confuse users - because only the 
package path was different
* make the hcat jsonserde use the "serde2" version as a back end - so that only 
1 real jsonserde exists in hive right now...


Diffs (updated)
-

  hcatalog/core/src/main/java/org/apache/hive/hcatalog/data/JsonSerDe.java 
114c205c83 
  hcatalog/core/src/test/java/org/apache/hive/hcatalog/data/TestJsonSerDe.java 
8aeb4f4f10 
  
itests/hive-jmh/src/main/java/org/apache/hive/benchmark/udf/json_read/JsonReadBench.java
 PRE-CREATION 
  
itests/hive-jmh/src/main/resources/org/apache/hive/benchmark/udf/json_read/val1.json
 PRE-CREATION 
  
itests/hive-jmh/src/main/resources/org/apache/hive/benchmark/udf/json_read/val1.type
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java e77fe18129 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFJsonRead.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFJsonRead.java 
PRE-CREATION 
  ql/src/test/queries/clientpositive/json_serde2.q PRE-CREATION 
  ql/src/test/queries/clientpositive/udf_json_read.q PRE-CREATION 
  ql/src/test/results/clientpositive/json_serde2.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/show_functions.q.out 91d3660aa0 
  ql/src/test/results/clientpositive/udf_json_read.q.out PRE-CREATION 
  serde/src/java/org/apache/hadoop/hive/serde2/JsonSerDe.java 1119fa2af5 
  serde/src/java/org/apache/hadoop/hive/serde2/json/HiveJsonStructReader.java 
PRE-CREATION 
  serde/src/java/org/apache/hadoop/hive/serde2/json/JsonSerDe2.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StrictJsonWriter.java cabb64cdcc 


Diff: https://reviews.apache.org/r/65480/diff/2/

Changes: https://reviews.apache.org/r/65480/diff/1-2/


Testing (updated)
---

* added some new tests for the udf


Thanks,

Zoltan Haindrich

[jira] [Created] (HIVE-19911) Hive delete queries fail with Invalid table alias or column reference

2018-06-15 Thread Mykhailo Kysliuk (JIRA) 
Mykhailo Kysliuk created HIVE-19911:
---

 Summary: Hive delete queries fail with Invalid table alias or 
column reference
 Key: HIVE-19911
 URL: https://issues.apache.org/jira/browse/HIVE-19911
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.3.3
Reporter: Mykhailo Kysliuk


Env:
hadoop-2.7.0
hive-2.3.3

OS:
centos-release-7-5.1804.el7.centos.x86_64

Steps to reproduce (at hive cli):

{code}
set hive.support.concurrency=true;
set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
DROP TABLE IF EXISTS detaillineitem_all;
DROP TABLE IF EXISTS detaillineitem_all_delete_1526330755128;

CREATE TABLE `detaillineitem_all`(
  `detailid` decimal(20,0),
  `branchnumber` varchar(3)
) PARTITIONED BY (
  `branchnumber_p` varchar(3))
CLUSTERED BY (
  detailid)
INTO 25 BUCKETS
STORED AS ORC
TBLPROPERTIES (
  'orc.compress'='NONE',
  'transactional'='true');

CREATE TABLE `detaillineitem_all_delete_1526330755128`(
  `detailid` decimal(20,0),
  `branchnumber` varchar(3),
  `branchnumber_p` varchar(3));

DELETE from detaillineitem_all WHERE EXISTS (
SELECT
1
FROM
detaillineitem_all_delete_1526330755128 AS t1
WHERE
(detaillineitem_all.detailid = t1.detailid)
  AND
(detaillineitem_all.branchnumber = CAST(t1.branchnumber AS STRING)));
{code}

Exception:

{code}
2018-06-15T16:51:48,625 ERROR [f6bd86a7-04e5-4284-9031-3b9a0ccc80f3 main] 
ql.Driver: FAILED: SemanticException Line 0:-1 Invalid table alias or column 
reference 'sq_1': (possible column names are: mber)) sq_corr_1)) (tok_where (= 
1 1), (. (tok_table_or_col sq_1) sq_corr_1))
org.apache.hadoop.hive.ql.parse.SemanticException: Line 0:-1 Invalid table 
alias or column reference 'sq_1': (possible column names are: mber)) 
sq_corr_1)) (tok_where (= 1 1), (. (tok_table_or_col sq_1) sq_corr_1))
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:11620)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:11568)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:11536)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:11514)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genMapGroupByForSemijoin(SemanticAnalyzer.java:8416)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genJoinOperator(SemanticAnalyzer.java:8305)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFilterPlan(SemanticAnalyzer.java:3278)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:9592)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:10549)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:10427)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:11125)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11138)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10807)
at 
org.apache.hadoop.hive.ql.parse.UpdateDeleteSemanticAnalyzer.analyzeInternal(UpdateDeleteSemanticAnalyzer.java:73)
at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
at 
org.apache.hadoop.hive.ql.parse.UpdateDeleteSemanticAnalyzer.reparseAndSuperAnalyze(UpdateDeleteSemanticAnalyzer.java:462)
at 
org.apache.hadoop.hive.ql.parse.UpdateDeleteSemanticAnalyzer.analyzeDelete(UpdateDeleteSemanticAnalyzer.java:111)
at 
org.apache.hadoop.hive.ql.parse.UpdateDeleteSemanticAnalyzer.analyzeInternal(UpdateDeleteSemanticAnalyzer.java:81)
at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:512)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
at 
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at

[jira] [Created] (HIVE-19912) Schema evolution checks prints a log line in INFO mode for each vectorized rowbatch, impacts performance

2018-06-15 Thread Nita Dembla (JIRA) 
Nita Dembla created HIVE-19912:
--

 Summary: Schema evolution checks prints a log line in INFO mode 
for each vectorized rowbatch, impacts performance
 Key: HIVE-19912
 URL: https://issues.apache.org/jira/browse/HIVE-19912
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 3.0.0
Reporter: Nita Dembla
 Fix For: 3.0.1


While benchmarking query96, noticed 17K log lines printed for each vector 
rowbactch

 

In file ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java

 
{code:java}
@@ -2554,8 +2554,8 @@ public static TypeDescription 
getDesiredRowTypeDescr(Configuration conf,
 }

 if (haveSchemaEvolutionProperties) {
-  if (LOG.isInfoEnabled()) {
-    LOG.info("Using schema evolution configuration variables 
schema.evolution.columns " +
+  if (LOG.isDebugEnabled()) {
+    LOG.debug("Using schema evolution configuration variables 
schema.evolution.columns " +
 schemaEvolutionColumnNames.toString() +
 " / schema.evolution.columns.types " +
 schemaEvolutionTypeDescrs.toString() +{code}
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 67468: HIVE-18118: provide supportability support for Erasure Coding Update number of Erasure Coded Files in a directory as part of Basic (aka Quick) Stats This information is then

2018-06-15 Thread Andrew Sherman via Review Board 


> On June 15, 2018, 1:37 p.m., Sahil Takiar wrote:
> > Why not show the #of EC files for regular explain plans too? To decrease 
> > the # of q file updates, it can be omitted if the # of EC files = 0

I saw that regular explain did not report numFiles so I did not report 
numEcFiles there.
I think you are saying that IF EC files are in a dir then instead of 
"Statistics: Num rows: 1 Data size: 15812" I would print  "Statistics: Num 
rows: 1 Data size: 15812 Erasure files: 2". Is that right?
Are you also suggesting that in extended explain I should also not report 
erasure coded file count unless it is positive? (This would require some small 
code changes as we currently I think just rely on dumping the properties )


> On June 15, 2018, 1:37 p.m., Sahil Takiar wrote:
> > standalone-metastore/src/main/java/org/apache/hadoop/hive/common/StatsSetupConst.java
> > Line 116 (original), 122 (patched)
> > 
> >
> > why change this from an array to a list?

When I changed some of the arrays to have new members, checkstyle didn't like 
it and suggested Lists. It was ugly to have some Lists and some arrays so I 
changed all the similar ones to be Lists.


- Andrew


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67468/#review204836
---


On June 6, 2018, 12:46 a.m., Andrew Sherman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67468/
> ---
> 
> (Updated June 6, 2018, 12:46 a.m.)
> 
> 
> Review request for hive and Sahil Takiar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18118: provide supportability support for Erasure Coding 
> [NOTE THIS REVIEW INITIALLY OMITS 200+ .q.out changes]
> Update number of Erasure Coded Files in a directory as part of Basic (aka 
> Quick) Stats 
> This information is then (mostly) available through 'EXPLAIN EXTENDED' and 
> 'DESCRIBE EXTENDED' 
> Extend the MiniHS2 Builder to allow configuring the number of datanodes. 
> Add a jdbc MiniHS2/Spark test that uses Erasure Coding. 
> There are some change to StatsSetupConst to make checkstyle happy.
> 
> 
> Diffs
> -
> 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
> d7d7097336fc6be4c2f7a35cd6897e0375486e81 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2ErasureCoding.java
>  PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties 
> 463fda1913f6d5b928fcee038f19e124b0239e96 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 2365fb76bd08f3a310e81ac3a19ca64971aeec8e 
>   itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 
> 1700c08d3f37285de43b5d4fe5c77ef55c170235 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
> e06949928d179cfd9a4dcb7176203b885509 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
>  209fdfb287cabc5bb7cab2117d771f7907deb2b9 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStatsNoJobTask.java 
> d4d46a3671efdaaed32f63b7262b963cce00b94e 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStatsTask.java 
> 8c238871765b0d5312a459a0e7f68c81f3837c13 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> 982b18076180ba300094f30a7f87f025f993b265 
>   ql/src/test/queries/clientpositive/erasure_explain.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/erasure_simple.q 
> c08409c17787417b986d90a43104f5ddd456e600 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_explain.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_simple.q.out 
> 01f6015a346c1e4283fd6a8cf1eaa3b670450e20 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/common/StatsSetupConst.java
>  78ea01d9687fe043d63441430c46b30c25cd9756 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  77ed2b4de4569fa8aca23b16f2b362b187c7c4fc 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
>  9b36d09eb9fb332e913d442bb476628eca334b6e 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/utils/TestMetaStoreUtils.java
>  55ff1502d415dea52095cfdd523d01f1e49ce084 
> 
> 
> Diff: https://reviews.apache.org/r/67468/diff/1/
> 
> 
> Testing
> ---
> 
> Ran driver tests and new jdbc test
> 
> 
> Thanks,
> 
> Andrew Sherman
> 
>

[jira] [Created] (HIVE-19913) OWNER_TYPE is missing in some metastore upgrade script

2018-06-15 Thread Daniel Dai (JIRA) 
Daniel Dai created HIVE-19913:
-

 Summary: OWNER_TYPE is missing in some metastore upgrade script
 Key: HIVE-19913
 URL: https://issues.apache.org/jira/browse/HIVE-19913
 Project: Hive
  Issue Type: Bug
  Components: Standalone Metastore
Reporter: Daniel Dai
Assignee: Daniel Dai


OWNER_TYPE introduced in HIVE-19372 is missing in upgrade-2.3.0-to-3.0.0.*.sql 
except derby.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 67612: HIVE-19904 : Load data rewrite into Tez job fails for ACID

2018-06-15 Thread Deepak Jaiswal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67612/
---

(Updated June 15, 2018, 9:46 p.m.)


Review request for hive, Eugene Koifman and Sankar Hariappan.


Changes
---

Use lower case for temp table name prefix.


Bugs: HIVE-19904
https://issues.apache.org/jira/browse/HIVE-19904


Repository: hive-git


Description
---

Load data rewrite into IAS fails for ACID as there is some code which does not 
take into account the table name could be in upper case, specifically 
ValidTxnWriteIdList


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
189975ed2e 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java fb88f25285 
  ql/src/test/results/clientpositive/llap/load_data_using_job.q.out 7a62be23dd 


Diff: https://reviews.apache.org/r/67612/diff/2/

Changes: https://reviews.apache.org/r/67612/diff/1-2/


Testing
---


Thanks,

Deepak Jaiswal

[jira] [Created] (HIVE-19915) master-txnstats branch - support CachedStore

2018-06-15 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-19915:
---

 Summary: master-txnstats branch - support CachedStore
 Key: HIVE-19915
 URL: https://issues.apache.org/jira/browse/HIVE-19915
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergey Shelukhin
Assignee: Steve Yeom






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19916) master-txnstats branch - integrate with HIVE-19382

2018-06-15 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-19916:
---

 Summary: master-txnstats branch - integrate with HIVE-19382
 Key: HIVE-19916
 URL: https://issues.apache.org/jira/browse/HIVE-19916
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergey Shelukhin
Assignee: Steve Yeom


There's some code in original txn stats patch that may go to metastore to get 
write Ids. This code should not go to metastore, it should fail instead. 
HIVE-19382 should ensure that we have correct IDs already present during 
optimizer - they are using by e.g. materialized view optimizer, so they should 
be there; if they are not present, some integration might be needed so that txn 
stats optimizations also have access to those write Ids.

cc [~jcamachorodriguez]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19914) master-txnstats branch - make sure SQL changes are in correct upgrade scripts

2018-06-15 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-19914:
---

 Summary: master-txnstats branch - make sure SQL changes are in 
correct upgrade scripts
 Key: HIVE-19914
 URL: https://issues.apache.org/jira/browse/HIVE-19914
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergey Shelukhin


The initial commit changed multiple files e.g.
{noformat}
standalone-metastore/src/main/sql/mysql/hive-schema-3.0.0.mysql.sql
standalone-metastore/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
standalone-metastore/src/main/sql/mysql/upgrade-2.3.0-to-3.0.0.mysql.sql
standalone-metastore/src/main/sql/mysql/upgrade-3.1.0-to-4.0.0.mysql.sql
{noformat}

The target version is currently 4.0 (or 3.1? cc [~hagleitn]), so all the 
changes should be in the scripts upgrading to 4.0
cc [~vgarg]




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19917) Import of full CRUD transactional table fails if table is not in default database

2018-06-15 Thread Eugene Koifman (JIRA) 
Eugene Koifman created HIVE-19917:
-

 Summary: Import of full CRUD transactional table fails if table is 
not in default database
 Key: HIVE-19917
 URL: https://issues.apache.org/jira/browse/HIVE-19917
 Project: Hive
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman


The actual issues is fixed by HIVE-19861.
This is a follow up to add a test case.

Issue:
{noformat}
org.apache.hadoop.hive.ql.metadata.HiveException: 
java.lang.IllegalArgumentException: Can not create a Path from a null string
at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:940) 
~[hive-exec-3.0.0.3.0.0.0-1485.jar:3.0.0.3.0.0.0-1485]
at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:945) 
~[hive-exec-3.0.0.3.0.0.0-1485.jar:3.0.0.3.0.0.0-1485]
at 
org.apache.hadoop.hive.ql.exec.DDLTask.createTableLike(DDLTask.java:5099) 
~[hive-exec-3.0.0.3.0.0.0-1485.jar:3.0.0.3.0.0.0-1485]
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:433) 
~[hive-exec-3.0.0.3.0.0.0-1485.jar:3.0.0.3.0.0.0-1485]
at 
org.apache.hadoop.hive.ql.parse.UpdateDeleteSemanticAnalyzer.analyzeAcidExport(UpdateDeleteSemanticAnalyzer.java:195)
 ~[hive-exec-3.0.0.3.0.0.0-1485.jar:3.0.0.3.0.0.0-1485]
at 
org.apache.hadoop.hive.ql.parse.UpdateDeleteSemanticAnalyzer.analyzeInternal(UpdateDeleteSemanticAnalyzer.java:106)
 ~[hive-exec-3.0.0.3.0.0.0-1485.jar:3.0.0.3.0.0.0-1485]
at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:288)
 ~[hive-exec-3.0.0.3.0.0.0-1485.jar:3.0.0.3.0.0.0-1485]
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:658) 
~[hive-exec-3.0.0.3.0.0.0-1485.jar:3.0.0.3.0.0.0-1485]
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1813) 
~[hive-exec-3.0.0.3.0.0.0-1485.jar:3.0.0.3.0.0.0-1485]
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1760) 
~[hive-exec-3.0.0.3.0.0.0-1485.jar:3.0.0.3.0.0.0-1485]
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1755) 
~[hive-exec-3.0.0.3.0.0.0-1485.jar:3.0.0.3.0.0.0-1485]
at 
org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
 ~[hive-exec-3.0.0.3.0.0.0-1485.jar:3.0.0.3.0.0.0-1485]
at 
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:194)
 ~[hive-service-3.0.0.3.0.0.0-1485.jar:3.0.0.3.0.0.0-1485]
at 
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:257)
 ~[hive-service-3.0.0.3.0.0.0-1485.jar:3.0.0.3.0.0.0-1485]
at 
org.apache.hive.service.cli.operation.Operation.run(Operation.java:243) 
~[hive-service-3.0.0.3.0.0.0-1485.jar:3.0.0.3.0.0.0-1485]
at 
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:541)
 ~[hive-service-3.0.0.3.0.0.0-1485.jar:3.0.0.3.0.0.0-1485]
at 
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:527)
 ~[hive-service-3.0.0.3.0.0.0-1485.jar:3.0.0.3.0.0.0-1485]
at 
org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:312)
 ~[hive-service-3.0.0.3.0.0.0-1485.jar:3.0.0.3.0.0.0-1485]
at 
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:562)
 ~[hive-service-3.0.0.3.0.0.0-1485.jar:3.0.0.3.0.0.0-1485]
at 
org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1557)
 ~[hive-exec-3.0.0.3.0.0.0-1485.jar:3.0.0.3.0.0.0-1485]
at 
org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1542)
 ~[hive-exec-3.0.0.3.0.0.0-1485.jar:3.0.0.3.0.0.0-1485]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) 
~[hive-exec-3.0.0.3.0.0.0-1485.jar:3.0.0.3.0.0.0-1485]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) 
~[hive-exec-3.0.0.3.0.0.0-1485.jar:3.0.0.3.0.0.0-1485]
at 
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:647)
 ~[hive-exec-3.0.0.3.0.0.0-1485.jar:3.0.0.3.0.0.0-1485]
at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
 ~[hive-exec-3.0.0.3.0.0.0-1485.jar:3.0.0.3.0.0.0-1485]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
~[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
~[?:1.8.0_112]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
Caused by: java.lang.IllegalArgumentException: Can not create a Path from a 
null string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:164) 
~[hadoop-common-3.0.0.3.0.0.0-1485.jar:?]

[jira] [Created] (HIVE-19918) IllegalArgumentException when getContext().getVertexTaskResource().getMemory() returns -1

2018-06-15 Thread Jaume M (JIRA) 
Jaume M created HIVE-19918:
--

 Summary: IllegalArgumentException when 
getContext().getVertexTaskResource().getMemory() returns -1
 Key: HIVE-19918
 URL: https://issues.apache.org/jira/browse/HIVE-19918
 Project: Hive
  Issue Type: Bug
Reporter: Jaume M


{code}
Beeline version 4.0.0-SNAPSHOT by Apache Hive
;: jdbc:hive2://hs2.example.com:1/> CREATE TABLE pokes (foo INT, bar STRING)
No rows affected (3.834 seconds)
G) PARTITIONED BY (ds STRING);m:1/> CREATE TABLE invites (foo INT, bar STRIN
No rows affected (0.175 seconds)
0: jdbc:hive2://hs2.example.com:1/> SELECT * FROM pokes, invites;
Error: Error while processing statement: FAILED: Execution Error, return code 2 
from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 
1, vertexId=vertex_1529099908701_0003_1_01, diagnostics=[Vertex 
vertex_1529099908701_0003_1_01 [Map 1] killed/failed due 
to:ROOT_INPUT_INIT_FAILURE, Vertex Input: pokes initializer failed, 
vertex=vertex_1529099908701_0003_1_01 [Map 1], 
java.lang.IllegalArgumentException: Illegal Capacity: -10444
at java.util.ArrayList.(ArrayList.java:157)
at 
org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:339)
at 
org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:519)
at 
org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:768)
at 
org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:211)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
at 
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
at 
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
at 
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0 
(state=08S01,code=2)
{code}

This is happening because 
[here|https://github.com/apache/hive/blob/bceb3dd82cbeb0334c3c33da7a21a74ad631810b/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HiveSplitGenerator.java#L188]
 the return is -1 and afterwards we request a negative number of splits.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)