[jira] [Created] (HIVE-23277) HiveProtoLogger should carry out JSON conversion in its own thread

2020-04-22 Thread Rajesh Balamohan (Jira) 
Rajesh Balamohan created HIVE-23277:
---

 Summary: HiveProtoLogger should carry out JSON conversion in its 
own thread
 Key: HIVE-23277
 URL: https://issues.apache.org/jira/browse/HIVE-23277
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan
 Attachments: Screenshot 2020-04-23 at 11.27.42 AM.png

!Screenshot 2020-04-23 at 11.27.42 AM.png|width=623,height=423!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23276) remove HttpException NPE possibility

2020-04-22 Thread Eugene Chung (Jira) 
Eugene Chung created HIVE-23276:
---

 Summary: remove HttpException NPE possibility 
 Key: HIVE-23276
 URL: https://issues.apache.org/jira/browse/HIVE-23276
 Project: Hive
  Issue Type: Bug
  Components: JDBC
Reporter: Eugene Chung
Assignee: Eugene Chung


{code:java}
java.lang.NullPointerException: null
  at org.apache.http.HttpException.clean(HttpException.java:48) 
~[httpcore-4.4.10.jar:4.4.10]
  at org.apache.http.HttpException.(HttpException.java:105) 
~[httpcore-4.4.10.jar:4.4.10]
  at 
org.apache.hive.jdbc.HttpKerberosRequestInterceptor.addHttpAuthHeader(HttpKerberosRequestInterceptor.java:67)
 ~[hive-jdbc-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
  at 
org.apache.hive.jdbc.HttpRequestInterceptorBase.process(HttpRequestInterceptorBase.java:74)
 ~[hive-jdbc-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
  at 
org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:133)
 ~[httpcore-4.4.10.jar:4.4.10]
  at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:183) 
~[httpclient-4.5.6.jar:4.5.6]
  at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) 
~[httpclient-4.5.6.jar:4.5.6]
  at 
org.apache.http.impl.execchain.ServiceUnavailableRetryExec.execute(ServiceUnavailableRetryExec.java:85)
 ~[httpclient-4.5.6.jar:4.5.6]
  at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) 
~[httpclient-4.5.6.jar:4.5.6]
  at 
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
 ~[httpclient-4.5.6.jar:4.5.6]
  at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:118)
 ~[httpclient-4.5.6.jar:4.5.6]
  at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
 ~[httpclient-4.5.6.jar:4.5.6]
  at 
org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:251)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
  at org.apache.thrift.transport.THttpClient.flush(THttpClient.java:313) 
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
  at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73) 
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
  at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62) 
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
  at 
org.apache.hive.service.rpc.thrift.TCLIService$Client.send_OpenSession(TCLIService.java:170)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
  at 
org.apache.hive.service.rpc.thrift.TCLIService$Client.OpenSession(TCLIService.java:162)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
  at org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:862) 
~[hive-jdbc-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
  at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:320) 
[hive-jdbc-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
  at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:94) 
[hive-jdbc-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
  at java.sql.DriverManager.getConnection(DriverManager.java:664) [?:1.8.0_191]
  at java.sql.DriverManager.getConnection(DriverManager.java:208) [?:1.8.0_191]
  at 
org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:145) 
[hive-beeline-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
  at 
org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:209)
 [hive-beeline-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
  at org.apache.hive.beeline.Commands.connect(Commands.java:1679) 
[hive-beeline-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
  at org.apache.hive.beeline.Commands.connect(Commands.java:1574) 
[hive-beeline-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_191]
  at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
~[?:1.8.0_191]
  at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.8.0_191]
  at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_191]
  at 
org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:57)
 [hive-beeline-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
  at org.apache.hive.beeline.BeeLine.execCommandWithPrefix(BeeLine.java:1494) 
[hive-beeline-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
  at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1533) 
[hive-beeline-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
  at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:1393) 
[hive-beeline-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
  at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1163) 
[hive-beeline-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
  at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:) 
[hive-beeline-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
  at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:555) 
[hive-beeline-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
  at org.apache.hive.beeline.BeeLine.main(BeeLine.java:537)

[jira] [Created] (HIVE-23275) Represent UNBOUNDED in window functions in CBO correctly

2020-04-22 Thread Jesus Camacho Rodriguez (Jira) 
Jesus Camacho Rodriguez created HIVE-23275:
--

 Summary: Represent UNBOUNDED in window functions in CBO correctly
 Key: HIVE-23275
 URL: https://issues.apache.org/jira/browse/HIVE-23275
 Project: Hive
  Issue Type: Improvement
  Components: CBO
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez


Currently we use a bounded representation with bound set to Integer.MAX_VALUE, 
which works correctly since that is the Hive implementation. However, Calcite 
has a specific boundary class {{RexWindowBoundUnbounded}} that we should be 
using instead.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Review Request 72421: Move q tests to TestMiniLlapLocal from TestCliDriver where the output is different, batch 1

2020-04-22 Thread Miklos Gergely 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72421/
---

Review request for hive.


Bugs: HIVE-23274
https://issues.apache.org/jira/browse/HIVE-23274


Repository: hive-git


Description
---

Move q tests to TestMiniLlapLocal from TestCliDriver where the output is 
different, batch 1


Diffs
-

  ql/src/test/results/clientpositive/llap/acid_mapjoin.q.out f66e978a01 
  ql/src/test/results/clientpositive/llap/acid_nullscan.q.out 0e5c2418d0 
  ql/src/test/results/clientpositive/llap/acid_stats2.q.out 45c903fb87 
  ql/src/test/results/clientpositive/llap/acid_stats3.q.out f9b0c57522 
  ql/src/test/results/clientpositive/llap/acid_stats4.q.out 279701d786 
  ql/src/test/results/clientpositive/llap/acid_stats5.q.out 4ae023fc4e 
  ql/src/test/results/clientpositive/llap/acid_table_stats.q.out c4bc80ee12 
  ql/src/test/results/clientpositive/llap/acid_view_delete.q.out a1de850a17 
  ql/src/test/results/clientpositive/llap/alias_casted_column.q.out 7a6620c799 
  ql/src/test/results/clientpositive/llap/allcolref_in_udf.q.out 87713544f5 
  ql/src/test/results/clientpositive/llap/alterColumnStatsPart.q.out fe17d00211 
  ql/src/test/results/clientpositive/llap/alter_change_db_location.q.out 
4c2115334b 
  ql/src/test/results/clientpositive/llap/alter_db_owner.q.out a6118f279a 
  ql/src/test/results/clientpositive/llap/alter_partition_coltype.q.out 
183cc4f8be 
  ql/src/test/results/clientpositive/llap/ambiguitycheck.q.out 3de3d68118 
  ql/src/test/results/clientpositive/llap/ambiguous_col.q.out bcc36a936b 
  ql/src/test/results/clientpositive/llap/analyze_table_null_partition.q.out 
ab05bd9f1a 
  ql/src/test/results/clientpositive/llap/analyze_tbl_date.q.out 9ce5f589ce 
  ql/src/test/results/clientpositive/llap/annotate_stats_deep_filters.q.out 
e721eacdcc 
  ql/src/test/results/clientpositive/llap/annotate_stats_filter.q.out 
6f8fad3971 
  ql/src/test/results/clientpositive/llap/annotate_stats_groupby.q.out 
4443a0e079 
  ql/src/test/results/clientpositive/llap/annotate_stats_groupby2.q.out 
e604e13ee8 
  ql/src/test/results/clientpositive/llap/annotate_stats_join.q.out f054af2a24 
  ql/src/test/results/clientpositive/llap/annotate_stats_join_pkfk.q.out 
21a1b12feb 
  ql/src/test/results/clientpositive/llap/annotate_stats_limit.q.out ec1dc8bb25 
  ql/src/test/results/clientpositive/llap/annotate_stats_part.q.out 31fa1426a2 
  ql/src/test/results/clientpositive/llap/annotate_stats_select.q.out 
dba27998a4 
  ql/src/test/results/clientpositive/llap/annotate_stats_table.q.out 9b9e31b214 
  ql/src/test/results/clientpositive/llap/annotate_stats_udtf.q.out c094fc1fea 
  ql/src/test/results/clientpositive/llap/annotate_stats_union.q.out 92108f21b3 
  ql/src/test/results/clientpositive/llap/ansi_sql_arithmetic.q.out 4760e986e1 
  ql/src/test/results/clientpositive/llap/array_map_access_nonconstant.q.out 
528e98ffe4 
  ql/src/test/results/clientpositive/llap/array_size_estimation.q.out 
00166721ef 
  ql/src/test/results/clientpositive/llap/authorization_9.q.out 5a22a8879a 
  ql/src/test/results/clientpositive/llap/authorization_explain.q.out 
61551f4ecb 
  ql/src/test/results/clientpositive/llap/authorization_owner_actions_db.q.out 
bc4805143f 
  ql/src/test/results/clientpositive/llap/authorization_view_1.q.out eac3fdcedd 
  
ql/src/test/results/clientpositive/llap/authorization_view_disable_cbo_1.q.out 
0b50d93acf 
  ql/src/test/results/clientpositive/llap/autoColumnStats_11.q.out c7a97865e8 
  ql/src/test/results/clientpositive/llap/autoColumnStats_4.q.out a9b5ad0c48 
  ql/src/test/results/clientpositive/llap/autoColumnStats_5.q.out 01793458a5 
  ql/src/test/results/clientpositive/llap/autoColumnStats_5a.q.out 4bc9df4b80 
  ql/src/test/results/clientpositive/llap/autoColumnStats_7.q.out 902a48fb4e 
  ql/src/test/results/clientpositive/llap/autoColumnStats_8.q.out de7352c36d 
  ql/src/test/results/clientpositive/llap/autoColumnStats_9.q.out 0d3eb2cf7c 
  ql/src/test/results/clientpositive/llap/auto_join10.q.out 3f750166ef 
  ql/src/test/results/clientpositive/llap/auto_join11.q.out 6cdfa34d5e 
  ql/src/test/results/clientpositive/llap/auto_join12.q.out 60a8f4732f 
  ql/src/test/results/clientpositive/llap/auto_join13.q.out 5f83b393fe 
  ql/src/test/results/clientpositive/llap/auto_join14.q.out 238befeb5c 
  ql/src/test/results/clientpositive/llap/auto_join15.q.out 15fa1d263d 
  ql/src/test/results/clientpositive/llap/auto_join16.q.out f13105665c 
  ql/src/test/results/clientpositive/llap/auto_join17.q.out 88185e790c 
  ql/src/test/results/clientpositive/llap/auto_join18.q.out 0473494c2a 
  ql/src/test/results/clientpositive/llap/auto_join18_multi_distinct.q.out 
961a320982 
  ql/src/test/results/clientpositive/llap/auto_join19.q.out cad3e0971e 
  ql/src/test/results/clientpositive/llap/auto_join19_inclause.q.out f42743a39f

Re: Review Request 72417: Move q tests to TestMiniLlapLocal from TestCliDriver where the output is different, batch 1

2020-04-22 Thread Miklos Gergely 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72417/
---

(Updated April 22, 2020, 8:22 p.m.)


Review request for hive.


Bugs: HIVE-23274
https://issues.apache.org/jira/browse/HIVE-23274


Repository: hive-git


Description
---

Move q tests to TestMiniLlapLocal from TestCliDriver where the output is 
different, batch 1


Diffs (updated)
-

  ql/src/test/results/clientpositive/acid_mapjoin.q.out f66e978a01 
  ql/src/test/results/clientpositive/acid_nullscan.q.out 0e5c2418d0 
  ql/src/test/results/clientpositive/acid_stats2.q.out 45c903fb87 
  ql/src/test/results/clientpositive/acid_stats3.q.out f9b0c57522 
  ql/src/test/results/clientpositive/acid_stats4.q.out 279701d786 
  ql/src/test/results/clientpositive/acid_stats5.q.out 4ae023fc4e 
  ql/src/test/results/clientpositive/acid_table_stats.q.out c4bc80ee12 
  ql/src/test/results/clientpositive/acid_view_delete.q.out a1de850a17 
  ql/src/test/results/clientpositive/alias_casted_column.q.out 7a6620c799 
  ql/src/test/results/clientpositive/allcolref_in_udf.q.out 87713544f5 
  ql/src/test/results/clientpositive/alterColumnStatsPart.q.out fe17d00211 
  ql/src/test/results/clientpositive/alter_change_db_location.q.out 4c2115334b 
  ql/src/test/results/clientpositive/alter_db_owner.q.out a6118f279a 
  ql/src/test/results/clientpositive/alter_partition_coltype.q.out 183cc4f8be 
  ql/src/test/results/clientpositive/ambiguitycheck.q.out 3de3d68118 
  ql/src/test/results/clientpositive/ambiguous_col.q.out bcc36a936b 
  ql/src/test/results/clientpositive/analyze_table_null_partition.q.out 
ab05bd9f1a 
  ql/src/test/results/clientpositive/analyze_tbl_date.q.out 9ce5f589ce 
  ql/src/test/results/clientpositive/annotate_stats_deep_filters.q.out 
e721eacdcc 
  ql/src/test/results/clientpositive/annotate_stats_filter.q.out 6f8fad3971 
  ql/src/test/results/clientpositive/annotate_stats_groupby.q.out 4443a0e079 
  ql/src/test/results/clientpositive/annotate_stats_groupby2.q.out e604e13ee8 
  ql/src/test/results/clientpositive/annotate_stats_join.q.out f054af2a24 
  ql/src/test/results/clientpositive/annotate_stats_join_pkfk.q.out 21a1b12feb 
  ql/src/test/results/clientpositive/annotate_stats_limit.q.out ec1dc8bb25 
  ql/src/test/results/clientpositive/annotate_stats_part.q.out 31fa1426a2 
  ql/src/test/results/clientpositive/annotate_stats_select.q.out dba27998a4 
  ql/src/test/results/clientpositive/annotate_stats_table.q.out 9b9e31b214 
  ql/src/test/results/clientpositive/annotate_stats_udtf.q.out c094fc1fea 
  ql/src/test/results/clientpositive/annotate_stats_union.q.out 92108f21b3 
  ql/src/test/results/clientpositive/ansi_sql_arithmetic.q.out 4760e986e1 
  ql/src/test/results/clientpositive/array_map_access_nonconstant.q.out 
528e98ffe4 
  ql/src/test/results/clientpositive/array_size_estimation.q.out 00166721ef 
  ql/src/test/results/clientpositive/authorization_9.q.out 5a22a8879a 
  ql/src/test/results/clientpositive/authorization_explain.q.out 61551f4ecb 
  ql/src/test/results/clientpositive/authorization_owner_actions_db.q.out 
bc4805143f 
  ql/src/test/results/clientpositive/authorization_view_1.q.out eac3fdcedd 
  ql/src/test/results/clientpositive/authorization_view_disable_cbo_1.q.out 
0b50d93acf 
  ql/src/test/results/clientpositive/autoColumnStats_11.q.out c7a97865e8 
  ql/src/test/results/clientpositive/autoColumnStats_4.q.out a9b5ad0c48 
  ql/src/test/results/clientpositive/autoColumnStats_5.q.out 01793458a5 
  ql/src/test/results/clientpositive/autoColumnStats_5a.q.out 4bc9df4b80 
  ql/src/test/results/clientpositive/autoColumnStats_7.q.out 902a48fb4e 
  ql/src/test/results/clientpositive/autoColumnStats_8.q.out de7352c36d 
  ql/src/test/results/clientpositive/autoColumnStats_9.q.out 0d3eb2cf7c 
  ql/src/test/results/clientpositive/auto_join10.q.out 3f750166ef 
  ql/src/test/results/clientpositive/auto_join11.q.out 6cdfa34d5e 
  ql/src/test/results/clientpositive/auto_join12.q.out 60a8f4732f 
  ql/src/test/results/clientpositive/auto_join13.q.out 5f83b393fe 
  ql/src/test/results/clientpositive/auto_join14.q.out 238befeb5c 
  ql/src/test/results/clientpositive/auto_join15.q.out 15fa1d263d 
  ql/src/test/results/clientpositive/auto_join16.q.out f13105665c 
  ql/src/test/results/clientpositive/auto_join17.q.out 88185e790c 
  ql/src/test/results/clientpositive/auto_join18.q.out 0473494c2a 
  ql/src/test/results/clientpositive/auto_join18_multi_distinct.q.out 
961a320982 
  ql/src/test/results/clientpositive/auto_join19.q.out cad3e0971e 
  ql/src/test/results/clientpositive/auto_join19_inclause.q.out f42743a39f 
  ql/src/test/results/clientpositive/auto_join2.q.out c694679e6f 
  ql/src/test/results/clientpositive/auto_join20.q.out 6475ad2ee0 
  ql/src/test/results/clientpositive/auto_join22.q.out 5a98716fed 
  ql/src/test/results/clientpositive/auto_join23.q.out e91161ad5c

Re: Review Request 72420: Move q tests to TestMiniLlapLocal from TestCliDriver where the output is different, batch 1

2020-04-22 Thread Miklos Gergely 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72420/
---

(Updated April 22, 2020, 8:22 p.m.)


Review request for hive.


Bugs: HIVE-23274
https://issues.apache.org/jira/browse/HIVE-23274


Repository: hive-git


Description
---

Move q tests to TestMiniLlapLocal from TestCliDriver where the output is 
different, batch 1


Diffs (updated)
-

  ql/src/test/results/clientpositive/acid_mapjoin.q.out f66e978a01 
  ql/src/test/results/clientpositive/acid_nullscan.q.out 0e5c2418d0 
  ql/src/test/results/clientpositive/acid_stats2.q.out 45c903fb87 
  ql/src/test/results/clientpositive/acid_stats3.q.out f9b0c57522 
  ql/src/test/results/clientpositive/acid_stats4.q.out 279701d786 
  ql/src/test/results/clientpositive/acid_stats5.q.out 4ae023fc4e 
  ql/src/test/results/clientpositive/acid_table_stats.q.out c4bc80ee12 
  ql/src/test/results/clientpositive/acid_view_delete.q.out a1de850a17 
  ql/src/test/results/clientpositive/alias_casted_column.q.out 7a6620c799 
  ql/src/test/results/clientpositive/allcolref_in_udf.q.out 87713544f5 
  ql/src/test/results/clientpositive/alterColumnStatsPart.q.out fe17d00211 
  ql/src/test/results/clientpositive/alter_change_db_location.q.out 4c2115334b 
  ql/src/test/results/clientpositive/alter_db_owner.q.out a6118f279a 
  ql/src/test/results/clientpositive/alter_partition_coltype.q.out 183cc4f8be 
  ql/src/test/results/clientpositive/ambiguitycheck.q.out 3de3d68118 
  ql/src/test/results/clientpositive/ambiguous_col.q.out bcc36a936b 
  ql/src/test/results/clientpositive/analyze_table_null_partition.q.out 
ab05bd9f1a 
  ql/src/test/results/clientpositive/analyze_tbl_date.q.out 9ce5f589ce 
  ql/src/test/results/clientpositive/annotate_stats_deep_filters.q.out 
e721eacdcc 
  ql/src/test/results/clientpositive/annotate_stats_filter.q.out 6f8fad3971 
  ql/src/test/results/clientpositive/annotate_stats_groupby.q.out 4443a0e079 
  ql/src/test/results/clientpositive/annotate_stats_groupby2.q.out e604e13ee8 
  ql/src/test/results/clientpositive/annotate_stats_join.q.out f054af2a24 
  ql/src/test/results/clientpositive/annotate_stats_join_pkfk.q.out 21a1b12feb 
  ql/src/test/results/clientpositive/annotate_stats_limit.q.out ec1dc8bb25 
  ql/src/test/results/clientpositive/annotate_stats_part.q.out 31fa1426a2 
  ql/src/test/results/clientpositive/annotate_stats_select.q.out dba27998a4 
  ql/src/test/results/clientpositive/annotate_stats_table.q.out 9b9e31b214 
  ql/src/test/results/clientpositive/annotate_stats_udtf.q.out c094fc1fea 
  ql/src/test/results/clientpositive/annotate_stats_union.q.out 92108f21b3 
  ql/src/test/results/clientpositive/ansi_sql_arithmetic.q.out 4760e986e1 
  ql/src/test/results/clientpositive/array_map_access_nonconstant.q.out 
528e98ffe4 
  ql/src/test/results/clientpositive/array_size_estimation.q.out 00166721ef 
  ql/src/test/results/clientpositive/authorization_9.q.out 5a22a8879a 
  ql/src/test/results/clientpositive/authorization_explain.q.out 61551f4ecb 
  ql/src/test/results/clientpositive/authorization_owner_actions_db.q.out 
bc4805143f 
  ql/src/test/results/clientpositive/authorization_view_1.q.out eac3fdcedd 
  ql/src/test/results/clientpositive/authorization_view_disable_cbo_1.q.out 
0b50d93acf 
  ql/src/test/results/clientpositive/autoColumnStats_11.q.out c7a97865e8 
  ql/src/test/results/clientpositive/autoColumnStats_4.q.out a9b5ad0c48 
  ql/src/test/results/clientpositive/autoColumnStats_5.q.out 01793458a5 
  ql/src/test/results/clientpositive/autoColumnStats_5a.q.out 4bc9df4b80 
  ql/src/test/results/clientpositive/autoColumnStats_7.q.out 902a48fb4e 
  ql/src/test/results/clientpositive/autoColumnStats_8.q.out de7352c36d 
  ql/src/test/results/clientpositive/autoColumnStats_9.q.out 0d3eb2cf7c 
  ql/src/test/results/clientpositive/auto_join10.q.out 3f750166ef 
  ql/src/test/results/clientpositive/auto_join11.q.out 6cdfa34d5e 
  ql/src/test/results/clientpositive/auto_join12.q.out 60a8f4732f 
  ql/src/test/results/clientpositive/auto_join13.q.out 5f83b393fe 
  ql/src/test/results/clientpositive/auto_join14.q.out 238befeb5c 
  ql/src/test/results/clientpositive/auto_join15.q.out 15fa1d263d 
  ql/src/test/results/clientpositive/auto_join16.q.out f13105665c 
  ql/src/test/results/clientpositive/auto_join17.q.out 88185e790c 
  ql/src/test/results/clientpositive/auto_join18.q.out 0473494c2a 
  ql/src/test/results/clientpositive/auto_join18_multi_distinct.q.out 
961a320982 
  ql/src/test/results/clientpositive/auto_join19.q.out cad3e0971e 
  ql/src/test/results/clientpositive/auto_join19_inclause.q.out f42743a39f 
  ql/src/test/results/clientpositive/auto_join2.q.out c694679e6f 
  ql/src/test/results/clientpositive/auto_join20.q.out 6475ad2ee0 
  ql/src/test/results/clientpositive/auto_join22.q.out 5a98716fed 
  ql/src/test/results/clientpositive/auto_join23.q.out e91161ad5c

Review Request 72420: Move q tests to TestMiniLlapLocal from TestCliDriver where the output is different, batch 1

2020-04-22 Thread Miklos Gergely 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72420/
---

Review request for hive.


Bugs: HIVE-23274
https://issues.apache.org/jira/browse/HIVE-23274


Repository: hive-git


Description
---

Move q tests to TestMiniLlapLocal from TestCliDriver where the output is 
different, batch 1


Diffs
-

  ql/src/test/results/clientpositive/acid_mapjoin.q.out  
  ql/src/test/results/clientpositive/acid_nullscan.q.out  
  ql/src/test/results/clientpositive/acid_stats2.q.out  
  ql/src/test/results/clientpositive/acid_stats3.q.out  
  ql/src/test/results/clientpositive/acid_stats4.q.out  
  ql/src/test/results/clientpositive/acid_stats5.q.out  
  ql/src/test/results/clientpositive/acid_table_stats.q.out  
  ql/src/test/results/clientpositive/acid_view_delete.q.out  
  ql/src/test/results/clientpositive/alias_casted_column.q.out  
  ql/src/test/results/clientpositive/allcolref_in_udf.q.out  
  ql/src/test/results/clientpositive/alterColumnStatsPart.q.out  
  ql/src/test/results/clientpositive/alter_change_db_location.q.out  
  ql/src/test/results/clientpositive/alter_db_owner.q.out  
  ql/src/test/results/clientpositive/alter_partition_coltype.q.out  
  ql/src/test/results/clientpositive/ambiguitycheck.q.out  
  ql/src/test/results/clientpositive/ambiguous_col.q.out  
  ql/src/test/results/clientpositive/analyze_table_null_partition.q.out  
  ql/src/test/results/clientpositive/analyze_tbl_date.q.out  
  ql/src/test/results/clientpositive/annotate_stats_deep_filters.q.out  
  ql/src/test/results/clientpositive/annotate_stats_filter.q.out  
  ql/src/test/results/clientpositive/annotate_stats_groupby.q.out  
  ql/src/test/results/clientpositive/annotate_stats_groupby2.q.out  
  ql/src/test/results/clientpositive/annotate_stats_join.q.out  
  ql/src/test/results/clientpositive/annotate_stats_join_pkfk.q.out  
  ql/src/test/results/clientpositive/annotate_stats_limit.q.out  
  ql/src/test/results/clientpositive/annotate_stats_part.q.out  
  ql/src/test/results/clientpositive/annotate_stats_select.q.out  
  ql/src/test/results/clientpositive/annotate_stats_table.q.out  
  ql/src/test/results/clientpositive/annotate_stats_udtf.q.out  
  ql/src/test/results/clientpositive/annotate_stats_union.q.out  
  ql/src/test/results/clientpositive/ansi_sql_arithmetic.q.out  
  ql/src/test/results/clientpositive/array_map_access_nonconstant.q.out  
  ql/src/test/results/clientpositive/array_size_estimation.q.out  
  ql/src/test/results/clientpositive/authorization_9.q.out  
  ql/src/test/results/clientpositive/authorization_explain.q.out  
  ql/src/test/results/clientpositive/authorization_owner_actions_db.q.out  
  ql/src/test/results/clientpositive/authorization_view_1.q.out  
  ql/src/test/results/clientpositive/authorization_view_disable_cbo_1.q.out  
  ql/src/test/results/clientpositive/autoColumnStats_11.q.out  
  ql/src/test/results/clientpositive/autoColumnStats_4.q.out  
  ql/src/test/results/clientpositive/autoColumnStats_5.q.out  
  ql/src/test/results/clientpositive/autoColumnStats_5a.q.out  
  ql/src/test/results/clientpositive/autoColumnStats_7.q.out  
  ql/src/test/results/clientpositive/autoColumnStats_8.q.out  
  ql/src/test/results/clientpositive/autoColumnStats_9.q.out  
  ql/src/test/results/clientpositive/auto_join10.q.out  
  ql/src/test/results/clientpositive/auto_join11.q.out  
  ql/src/test/results/clientpositive/auto_join12.q.out  
  ql/src/test/results/clientpositive/auto_join13.q.out  
  ql/src/test/results/clientpositive/auto_join14.q.out  
  ql/src/test/results/clientpositive/auto_join15.q.out  
  ql/src/test/results/clientpositive/auto_join16.q.out  
  ql/src/test/results/clientpositive/auto_join17.q.out  
  ql/src/test/results/clientpositive/auto_join18.q.out  
  ql/src/test/results/clientpositive/auto_join18_multi_distinct.q.out  
  ql/src/test/results/clientpositive/auto_join19.q.out  
  ql/src/test/results/clientpositive/auto_join19_inclause.q.out  
  ql/src/test/results/clientpositive/auto_join2.q.out  
  ql/src/test/results/clientpositive/auto_join20.q.out  
  ql/src/test/results/clientpositive/auto_join22.q.out  
  ql/src/test/results/clientpositive/auto_join23.q.out  
  ql/src/test/results/clientpositive/auto_join24.q.out  
  ql/src/test/results/clientpositive/auto_join25.q.out  
  ql/src/test/results/clientpositive/auto_join26.q.out  
  ql/src/test/results/clientpositive/auto_join27.q.out  
  ql/src/test/results/clientpositive/auto_join28.q.out  
  ql/src/test/results/clientpositive/auto_join3.q.out  
  ql/src/test/results/clientpositive/auto_join31.q.out  
  ql/src/test/results/clientpositive/auto_join32.q.out  
  ql/src/test/results/clientpositive/auto_join33.q.out  
  ql/src/test/results/clientpositive/auto_join4.q.out  
  ql/src/test/results/clientpositive/auto_join5.q.out

Re: Hive Grammar Parser Visualization

2020-04-22 Thread Driesprong, Fokko 
Thanks for sharing David,

It looks great. I think this would give great insights into how Hive parses
queries.

Cheers, Fokko

Op do 16 apr. 2020 om 16:49 schreef David Mollitor :

> Hello Gang,
>
> https://issues.apache.org/jira/browse/HIVE-23171
>
> I just wanted to share with you a tool that I put into the master branch of
> Hive.  It is a small command-line tool for building visualizations of the
> output of the Hive ANTLR parser.
>
> I hope you find it useful.
>
> Thanks.
>

Review Request 72417: Move q tests to TestMiniLlapLocal from TestCliDriver where the output is different, batch 1

2020-04-22 Thread Miklos Gergely 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72417/
---

Review request for hive.


Bugs: HIVE-23274
https://issues.apache.org/jira/browse/HIVE-23274


Repository: hive-git


Description
---

Move q tests to TestMiniLlapLocal from TestCliDriver where the output is 
different, batch 1


Diffs
-

  ql/src/test/results/clientpositive/acid_mapjoin.q.out  
  ql/src/test/results/clientpositive/acid_nullscan.q.out  
  ql/src/test/results/clientpositive/acid_stats2.q.out  
  ql/src/test/results/clientpositive/acid_stats3.q.out  
  ql/src/test/results/clientpositive/acid_stats4.q.out  
  ql/src/test/results/clientpositive/acid_stats5.q.out  
  ql/src/test/results/clientpositive/acid_table_stats.q.out  
  ql/src/test/results/clientpositive/acid_view_delete.q.out  
  ql/src/test/results/clientpositive/alias_casted_column.q.out  
  ql/src/test/results/clientpositive/allcolref_in_udf.q.out  
  ql/src/test/results/clientpositive/alterColumnStatsPart.q.out  
  ql/src/test/results/clientpositive/alter_change_db_location.q.out  
  ql/src/test/results/clientpositive/alter_db_owner.q.out  
  ql/src/test/results/clientpositive/alter_partition_coltype.q.out  
  ql/src/test/results/clientpositive/ambiguitycheck.q.out  
  ql/src/test/results/clientpositive/ambiguous_col.q.out  
  ql/src/test/results/clientpositive/analyze_table_null_partition.q.out  
  ql/src/test/results/clientpositive/analyze_tbl_date.q.out  
  ql/src/test/results/clientpositive/annotate_stats_deep_filters.q.out  
  ql/src/test/results/clientpositive/annotate_stats_filter.q.out  
  ql/src/test/results/clientpositive/annotate_stats_groupby.q.out  
  ql/src/test/results/clientpositive/annotate_stats_groupby2.q.out  
  ql/src/test/results/clientpositive/annotate_stats_join.q.out  
  ql/src/test/results/clientpositive/annotate_stats_join_pkfk.q.out  
  ql/src/test/results/clientpositive/annotate_stats_limit.q.out  
  ql/src/test/results/clientpositive/annotate_stats_part.q.out  
  ql/src/test/results/clientpositive/annotate_stats_select.q.out  
  ql/src/test/results/clientpositive/annotate_stats_table.q.out  
  ql/src/test/results/clientpositive/annotate_stats_udtf.q.out  
  ql/src/test/results/clientpositive/annotate_stats_union.q.out  
  ql/src/test/results/clientpositive/ansi_sql_arithmetic.q.out  
  ql/src/test/results/clientpositive/array_map_access_nonconstant.q.out  
  ql/src/test/results/clientpositive/array_size_estimation.q.out  
  ql/src/test/results/clientpositive/authorization_9.q.out  
  ql/src/test/results/clientpositive/authorization_explain.q.out  
  ql/src/test/results/clientpositive/authorization_owner_actions_db.q.out  
  ql/src/test/results/clientpositive/authorization_view_1.q.out  
  ql/src/test/results/clientpositive/authorization_view_disable_cbo_1.q.out  
  ql/src/test/results/clientpositive/autoColumnStats_11.q.out  
  ql/src/test/results/clientpositive/autoColumnStats_4.q.out  
  ql/src/test/results/clientpositive/autoColumnStats_5.q.out  
  ql/src/test/results/clientpositive/autoColumnStats_5a.q.out  
  ql/src/test/results/clientpositive/autoColumnStats_7.q.out  
  ql/src/test/results/clientpositive/autoColumnStats_8.q.out  
  ql/src/test/results/clientpositive/autoColumnStats_9.q.out  
  ql/src/test/results/clientpositive/auto_join10.q.out  
  ql/src/test/results/clientpositive/auto_join11.q.out  
  ql/src/test/results/clientpositive/auto_join12.q.out  
  ql/src/test/results/clientpositive/auto_join13.q.out  
  ql/src/test/results/clientpositive/auto_join14.q.out  
  ql/src/test/results/clientpositive/auto_join15.q.out  
  ql/src/test/results/clientpositive/auto_join16.q.out  
  ql/src/test/results/clientpositive/auto_join17.q.out  
  ql/src/test/results/clientpositive/auto_join18.q.out  
  ql/src/test/results/clientpositive/auto_join18_multi_distinct.q.out  
  ql/src/test/results/clientpositive/auto_join19.q.out  
  ql/src/test/results/clientpositive/auto_join19_inclause.q.out  
  ql/src/test/results/clientpositive/auto_join2.q.out  
  ql/src/test/results/clientpositive/auto_join20.q.out  
  ql/src/test/results/clientpositive/auto_join22.q.out  
  ql/src/test/results/clientpositive/auto_join23.q.out  
  ql/src/test/results/clientpositive/auto_join24.q.out  
  ql/src/test/results/clientpositive/auto_join25.q.out  
  ql/src/test/results/clientpositive/auto_join26.q.out  
  ql/src/test/results/clientpositive/auto_join27.q.out  
  ql/src/test/results/clientpositive/auto_join28.q.out  
  ql/src/test/results/clientpositive/auto_join3.q.out  
  ql/src/test/results/clientpositive/auto_join31.q.out  
  ql/src/test/results/clientpositive/auto_join32.q.out  
  ql/src/test/results/clientpositive/auto_join33.q.out  
  ql/src/test/results/clientpositive/auto_join4.q.out  
  ql/src/test/results/clientpositive/auto_join5.q.out

[jira] [Created] (HIVE-23274) Move q tests to TestMiniLlapLocal from TestCliDriver where the output is different, batch 1

2020-04-22 Thread Miklos Gergely (Jira) 
Miklos Gergely created HIVE-23274:
-

 Summary: Move q tests to TestMiniLlapLocal from TestCliDriver 
where the output is different, batch 1
 Key: HIVE-23274
 URL: https://issues.apache.org/jira/browse/HIVE-23274
 Project: Hive
  Issue Type: Sub-task
Reporter: Miklos Gergely
Assignee: Miklos Gergely






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23273) Add fix order to cbo_limit.q queries + improve readability

2020-04-22 Thread Miklos Gergely (Jira) 
Miklos Gergely created HIVE-23273:
-

 Summary: Add fix order to cbo_limit.q queries + improve readability
 Key: HIVE-23273
 URL: https://issues.apache.org/jira/browse/HIVE-23273
 Project: Hive
  Issue Type: Bug
  Components: Hive
Reporter: Miklos Gergely
Assignee: Miklos Gergely
 Fix For: 4.0.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72387: Locks: Add new lock implementations for always zero-wait readers

2020-04-22 Thread Denys Kuzmenko via Review Board 


> On April 22, 2020, 3:22 p.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/ValidTxnManager.java
> > Line 113 (original), 112 (patched)
> > 
> >
> > Isn't reference level check is risky here?

good catch, need to reorder :)


- Denys


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72387/#review220426
---


On April 22, 2020, 2:29 p.m., Denys Kuzmenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72387/
> ---
> 
> (Updated April 22, 2020, 2:29 p.m.)
> 
> 
> Review request for hive, Marton Bod and Peter Vary.
> 
> 
> Bugs: HIVE-19369
> https://issues.apache.org/jira/browse/HIVE-19369
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Hive Locking with Micro-managed and full-ACID tables needs a better locking 
> implementation which allows for no-wait readers always.
> 
> EXCL_DROP
> EXCL_WRITE
> SHARED_WRITE
> SHARED_READ
> 
> Short write-up
> 
> EXCL_DROP is a "drop partition" or "drop table" and waits for all others to 
> exit
> EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to 
> exit.
> SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an 
> EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different 
> threads).
> 
> SHARED_READ does not wait for any lock - it fails fast for a pending 
> EXCL_DROP, because even if there is an EXCL_WRITE or SHARED_WRITE pending, 
> there's no semantic reason to wait for them to succeed before going ahead 
> with a SHARED_WRITE.
> 
> a select * => SHARED_READ
> an insert into => SHARED_WRITE
> an insert overwrite or MERGE => EXCL_WRITE
> a drop table => EXCL_DROP
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 16bae920df 
>   ql/src/java/org/apache/hadoop/hive/ql/ValidTxnManager.java 4885e437aa 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 77878ca40b 
>   ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 
> 1d211857bf 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
> 73d3b91585 
>   ql/src/test/queries/clientpositive/explain_locks.q 3c11560c5f 
>   ql/src/test/results/clientpositive/llap/explain_locks.q.out 3183533478 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockRequestBuilder.java
>  22902a9c20 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockTypeComparator.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  d080df417b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtil.java
>  f928bf781b 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtilTest.java
>  3d69a6e7dc 
> 
> 
> Diff: https://reviews.apache.org/r/72387/diff/5/
> 
> 
> Testing
> ---
> 
> Added number of tests under TestDbTxnManager2, TestTxnHandler & extended 
> explain_locks.q test.
> 
> 
> Thanks,
> 
> Denys Kuzmenko
> 
>

Re: Review Request 72387: Locks: Add new lock implementations for always zero-wait readers

2020-04-22 Thread Denys Kuzmenko via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72387/
---

(Updated April 22, 2020, 3:27 p.m.)


Review request for hive, Marton Bod and Peter Vary.


Bugs: HIVE-19369
https://issues.apache.org/jira/browse/HIVE-19369


Repository: hive-git


Description
---

Hive Locking with Micro-managed and full-ACID tables needs a better locking 
implementation which allows for no-wait readers always.

EXCL_DROP
EXCL_WRITE
SHARED_WRITE
SHARED_READ

Short write-up

EXCL_DROP is a "drop partition" or "drop table" and waits for all others to exit
EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to 
exit.
SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an 
EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different 
threads).

SHARED_READ does not wait for any lock - it fails fast for a pending EXCL_DROP, 
because even if there is an EXCL_WRITE or SHARED_WRITE pending, there's no 
semantic reason to wait for them to succeed before going ahead with a 
SHARED_WRITE.

a select * => SHARED_READ
an insert into => SHARED_WRITE
an insert overwrite or MERGE => EXCL_WRITE
a drop table => EXCL_DROP


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 16bae920df 
  ql/src/java/org/apache/hadoop/hive/ql/ValidTxnManager.java 4885e437aa 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 77878ca40b 
  ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 
1d211857bf 
  ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
73d3b91585 
  ql/src/test/queries/clientpositive/explain_locks.q 3c11560c5f 
  ql/src/test/results/clientpositive/llap/explain_locks.q.out 3183533478 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockRequestBuilder.java
 22902a9c20 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockTypeComparator.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
 d080df417b 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtil.java
 f928bf781b 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtilTest.java
 3d69a6e7dc 


Diff: https://reviews.apache.org/r/72387/diff/6/

Changes: https://reviews.apache.org/r/72387/diff/5-6/


Testing
---

Added number of tests under TestDbTxnManager2, TestTxnHandler & extended 
explain_locks.q test.


Thanks,

Denys Kuzmenko

Re: Review Request 72387: Locks: Add new lock implementations for always zero-wait readers

2020-04-22 Thread Peter Vary via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72387/#review220426
---


Ship it!





ql/src/java/org/apache/hadoop/hive/ql/ValidTxnManager.java
Lines 96 (patched)


nit: new line?



ql/src/java/org/apache/hadoop/hive/ql/ValidTxnManager.java
Line 113 (original), 112 (patched)


Isn't reference level check is risky here?


- Peter Vary


On ápr. 22, 2020, 2:29 du, Denys Kuzmenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72387/
> ---
> 
> (Updated ápr. 22, 2020, 2:29 du)
> 
> 
> Review request for hive, Marton Bod and Peter Vary.
> 
> 
> Bugs: HIVE-19369
> https://issues.apache.org/jira/browse/HIVE-19369
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Hive Locking with Micro-managed and full-ACID tables needs a better locking 
> implementation which allows for no-wait readers always.
> 
> EXCL_DROP
> EXCL_WRITE
> SHARED_WRITE
> SHARED_READ
> 
> Short write-up
> 
> EXCL_DROP is a "drop partition" or "drop table" and waits for all others to 
> exit
> EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to 
> exit.
> SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an 
> EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different 
> threads).
> 
> SHARED_READ does not wait for any lock - it fails fast for a pending 
> EXCL_DROP, because even if there is an EXCL_WRITE or SHARED_WRITE pending, 
> there's no semantic reason to wait for them to succeed before going ahead 
> with a SHARED_WRITE.
> 
> a select * => SHARED_READ
> an insert into => SHARED_WRITE
> an insert overwrite or MERGE => EXCL_WRITE
> a drop table => EXCL_DROP
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 16bae920df 
>   ql/src/java/org/apache/hadoop/hive/ql/ValidTxnManager.java 4885e437aa 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 77878ca40b 
>   ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 
> 1d211857bf 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
> 73d3b91585 
>   ql/src/test/queries/clientpositive/explain_locks.q 3c11560c5f 
>   ql/src/test/results/clientpositive/llap/explain_locks.q.out 3183533478 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockRequestBuilder.java
>  22902a9c20 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockTypeComparator.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  d080df417b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtil.java
>  f928bf781b 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtilTest.java
>  3d69a6e7dc 
> 
> 
> Diff: https://reviews.apache.org/r/72387/diff/5/
> 
> 
> Testing
> ---
> 
> Added number of tests under TestDbTxnManager2, TestTxnHandler & extended 
> explain_locks.q test.
> 
> 
> Thanks,
> 
> Denys Kuzmenko
> 
>

[jira] [Created] (HIVE-23272) Fix and reenable timestamptz_2.q

2020-04-22 Thread Peter Vary (Jira) 
Peter Vary created HIVE-23272:
-

 Summary: Fix and reenable timestamptz_2.q
 Key: HIVE-23272
 URL: https://issues.apache.org/jira/browse/HIVE-23272
 Project: Hive
  Issue Type: Test
Reporter: Peter Vary






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23271) Can't start Hive Interactive Server in HDP 3.1.4 Cluster

2020-04-22 Thread Jira 
Gerardo Adrián Aguirre Vivar created HIVE-23271:
---

 Summary: Can't start Hive Interactive Server in HDP 3.1.4 Cluster
 Key: HIVE-23271
 URL: https://issues.apache.org/jira/browse/HIVE-23271
 Project: Hive
  Issue Type: Bug
  Components: Configuration
Affects Versions: 3.1.0
 Environment: All nodes have CentOS 7

Cluster HDP 3.1.4
Reporter: Gerardo Adrián Aguirre Vivar
Assignee: Gerardo Adrián Aguirre Vivar


Hive interactive server is not working.  The installation guide has  been 
followed 
([https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.4/performance-tuning/content/hive_prepare_to_tune_performance.html])
 but when the server try to start, an errors appears. 

 

LOGS:

 

2020-04-22T16:43:48,271 INFO [main]: impl.YarnClientImpl 
(YarnClientImpl.java:submitApplication(306)) - Submitted application 
application_1587555843754_0015
2020-04-22T16:43:48,275 INFO [main]: client.TezClient 
(TezClient.java:start(404)) - The url to track the Tez Session: 
http://:8088/proxy/application_1587555843754_0015/
2020-04-22T16:43:53,435 INFO [main]: client.TezClient 
(TezClient.java:getAppMasterStatus(881)) - *{color:#0747a6}Failed to retrieve 
AM Status via proxy{color}*
com.google.protobuf.ServiceException: java.io.EOFException: End of File 
Exception between local host is: "/10.22.39.12"; 
destination host is: "":33889; : java.io.EOFException; For more 
details see: http://wiki.apache.org/hadoop/EOFException
 at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:242)
 ~[hadoop-common-3.1.1.3.1.4.0-315.jar:?]
 at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
 ~[hadoop-common-3.1.1.3.1.4.0-315.jar:?]
 at com.sun.proxy.$Proxy75.getAMStatus(Unknown Source) ~[?:?]
 at org.apache.tez.client.TezClient.getAppMasterStatus(TezClient.java:874) 
[tez-api-0.9.1.3.1.4.0-315.jar:0.9.1.3.1.4.0-315]
 at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:1011) 
[tez-api-0.9.1.3.1.4.0-315.jar:0.9.1.3.1.4.0-315]
 at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:982) 
[tez-api-0.9.1.3.1.4.0-315.jar:0.9.1.3.1.4.0-315]
 at 
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:536)
 [hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
 at 
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:451)
 [hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
 at 
org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolSession.openInternal(TezSessionPoolSession.java:124)
 [hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
 at 
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:373)
 [hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
 at 
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:236)
 [hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
 at 
org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.startInitialSession(TezSessionPool.java:354)
 [hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
 at 
org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.startUnderInitLock(TezSessionPool.java:166)
 [hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
 at 
org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.start(TezSessionPool.java:123)
 [hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
 at 
org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.startPool(TezSessionPoolManager.java:112)
 [hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
 at 
org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:855)
 [hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
 at 
org.apache.hive.service.server.HiveServer2.startOrReconnectTezSessions(HiveServer2.java:828)
 [hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
 at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:752) 
[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
 at 
org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1078)
 [hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
 at 
org.apache.hive.service.server.HiveServer2.access$1700(HiveServer2.java:136) 
[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
 at 
org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1346)
 [hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
 at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1190) 
[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_232]
 at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
~[?:1.8.0_232]
 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.8.0_232]
 at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_232]
 at

[jira] [Created] (HIVE-23270) Optimize isValidTxnListState to reduce the numbers of HMS calls

2020-04-22 Thread Peter Vary (Jira) 
Peter Vary created HIVE-23270:
-

 Summary: Optimize isValidTxnListState to reduce the numbers of HMS 
calls
 Key: HIVE-23270
 URL: https://issues.apache.org/jira/browse/HIVE-23270
 Project: Hive
  Issue Type: Improvement
Reporter: Peter Vary
Assignee: Peter Vary


There are several checks which does not need a HMS call, and they can already 
define the return value. Move them forward potentially preventing an extra HMS 
call



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72387: Locks: Add new lock implementations for always zero-wait readers

2020-04-22 Thread Denys Kuzmenko via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72387/
---

(Updated April 22, 2020, 2:29 p.m.)


Review request for hive, Marton Bod and Peter Vary.


Changes
---

reorder checks & fixed isValidTxnListState


Bugs: HIVE-19369
https://issues.apache.org/jira/browse/HIVE-19369


Repository: hive-git


Description
---

Hive Locking with Micro-managed and full-ACID tables needs a better locking 
implementation which allows for no-wait readers always.

EXCL_DROP
EXCL_WRITE
SHARED_WRITE
SHARED_READ

Short write-up

EXCL_DROP is a "drop partition" or "drop table" and waits for all others to exit
EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to 
exit.
SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an 
EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different 
threads).

SHARED_READ does not wait for any lock - it fails fast for a pending EXCL_DROP, 
because even if there is an EXCL_WRITE or SHARED_WRITE pending, there's no 
semantic reason to wait for them to succeed before going ahead with a 
SHARED_WRITE.

a select * => SHARED_READ
an insert into => SHARED_WRITE
an insert overwrite or MERGE => EXCL_WRITE
a drop table => EXCL_DROP


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 16bae920df 
  ql/src/java/org/apache/hadoop/hive/ql/ValidTxnManager.java 4885e437aa 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 77878ca40b 
  ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 
1d211857bf 
  ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
73d3b91585 
  ql/src/test/queries/clientpositive/explain_locks.q 3c11560c5f 
  ql/src/test/results/clientpositive/llap/explain_locks.q.out 3183533478 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockRequestBuilder.java
 22902a9c20 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockTypeComparator.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
 d080df417b 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtil.java
 f928bf781b 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtilTest.java
 3d69a6e7dc 


Diff: https://reviews.apache.org/r/72387/diff/5/

Changes: https://reviews.apache.org/r/72387/diff/4-5/


Testing
---

Added number of tests under TestDbTxnManager2, TestTxnHandler & extended 
explain_locks.q test.


Thanks,

Denys Kuzmenko

[jira] [Created] (HIVE-23269) Unsafe compares bigints and chars

2020-04-22 Thread Zhihua Deng (Jira) 
Zhihua Deng created HIVE-23269:
--

 Summary: Unsafe compares bigints and chars
 Key: HIVE-23269
 URL: https://issues.apache.org/jira/browse/HIVE-23269
 Project: Hive
  Issue Type: Improvement
Reporter: Zhihua Deng


Comparing bigints and varchars or chars may result to wrong result,  for 
example:

CREATE TABLE test_a (appid1 varchar(256),  appid2 char(20));
INSERT INTO  test_a VALUES ('2882303761517473127', '2882303761517473127'), 
('2882303761517473276','2882303761517473276');

SET hive.strict.checks.type.safety=false;
SELECT appid1 FROM test_a WHERE appid1 = 2882303761517473127;
SELECT appid2 FROM test_a WHERE appid2 = 2882303761517473127;​

Both queries will output the row: ('2882303761517473276','2882303761517473276')




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72378: HIVE-23201: Improve logging in locking

2020-04-22 Thread Denys Kuzmenko via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72378/#review220419
---



Looks good, however I don't really like change with REQ_LOCKS_ALIASED_FIELDS as 
it generats much bigger query (keep in mind there 3 of them joined by union, 
will be 4 for excl_write) + we are transfering more data that we are not very 
interested in. To get more information about the lock we can simply run show 
locks.

- Denys Kuzmenko


On April 17, 2020, 11:30 a.m., Marton Bod wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72378/
> ---
> 
> (Updated April 17, 2020, 11:30 a.m.)
> 
> 
> Review request for hive, Denys Kuzmenko and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-23201: Improve logging in locking
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbLockManager.java 4b6bc3e1e3 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  e7910c1c5d 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java
>  b3a1f826bb 
> 
> 
> Diff: https://reviews.apache.org/r/72378/diff/3/
> 
> 
> Testing
> ---
> 
> Green build: https://builds.apache.org/job/PreCommit-HIVE-Build/21717/
> 
> 
> Thanks,
> 
> Marton Bod
> 
>

Re: Review Request 72378: HIVE-23201: Improve logging in locking

2020-04-22 Thread Marton Bod 


> On April 21, 2020, 10:38 a.m., Denys Kuzmenko wrote:
> > LGTM, just number of comments

Thanks for the review Denys! 
As per your comments, I have elevated the log levels, removed the less useful 
log lines and changed the original statements to prepared statements.


- Marton


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72378/#review220389
---


On April 17, 2020, 11:30 a.m., Marton Bod wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72378/
> ---
> 
> (Updated April 17, 2020, 11:30 a.m.)
> 
> 
> Review request for hive, Denys Kuzmenko and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-23201: Improve logging in locking
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbLockManager.java 4b6bc3e1e3 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  d080df417b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java
>  b3a1f826bb 
> 
> 
> Diff: https://reviews.apache.org/r/72378/diff/2/
> 
> 
> Testing
> ---
> 
> Green build: https://builds.apache.org/job/PreCommit-HIVE-Build/21717/
> 
> 
> Thanks,
> 
> Marton Bod
> 
>

[jira] [Created] (HIVE-23268) Eliminate beanutils transitive dependency

2020-04-22 Thread Ashutosh Chauhan (Jira) 
Ashutosh Chauhan created HIVE-23268:
---

 Summary: Eliminate beanutils transitive dependency
 Key: HIVE-23268
 URL: https://issues.apache.org/jira/browse/HIVE-23268
 Project: Hive
  Issue Type: Improvement
  Components: Hive
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan


Transitively retrieved from hadoop-commons



--
This message was sent by Atlassian Jira
(v8.3.4#803005)