Re: Review Request 72499: HIVE-23446:LLAP: Reduce IPC connection misses to AM for short queries

2020-05-13 Thread Ashutosh Chauhan


> On May 14, 2020, 4:31 a.m., Ashutosh Chauhan wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
> > Lines 610 (patched)
> > 
> >
> > What's the reason for time based expiry? Is it because UGI expires 
> > after 24 hrs?
> > Else, I would have expected long living cache with blocking queue of 
> > bounded size.
> > 
> > Queue should be bounded by number of executors anyways, since having 
> > more connections than executors probably won't be needed.
> 
> Rajesh Balamohan wrote:
> Since this is based on the AM. So if AM dies after sometime (due to 
> inactivity, as in no-DAG submissions), these UGIs will be auto purged after 
> 10 minutes.

In LLAP, AM doesn't die due to inactivity, its long living. It may die because 
of crash though. So, then should this expiry be longer. 3 hrs?


> On May 14, 2020, 4:31 a.m., Ashutosh Chauhan wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
> > Lines 646-651 (patched)
> > 
> >
> > Is this logic needed? You already have valueloader in get() which must 
> > return a ugi, so it cant be null.
> 
> Rajesh Balamohan wrote:
> Yes, value loader is for initial miss. This is to avoid single connection 
> becoming a contention for AM communication. 
> https://issues.apache.org/jira/browse/HIVE-16634

Not sure I follow. Can you add comments in code to explain the need for this?


> On May 14, 2020, 4:31 a.m., Ashutosh Chauhan wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
> > Lines 663 (patched)
> > 
> >
> > if its null, then its programming error. Better to not do this null 
> > check and offer without checking for null.

better to throw NPE then to leak ugi failing to return to pool.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72499/#review220748
---


On May 12, 2020, 12:06 p.m., Rajesh Balamohan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72499/
> ---
> 
> (Updated May 12, 2020, 12:06 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Gopal V.
> 
> 
> Bugs: HIVE-23446
> https://issues.apache.org/jira/browse/HIVE-23446
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Currently UGI pool is maintained at QueryInfo level. However, when short 
> queries and lots of AMs are there, it ends missing IPC connection cache. Too 
> many connections are are also established. Patch tries to avoid that by 
> maintaining this at ContainerRunner level. It retains the current behaviour 
> of having multiple connection to same AM (otherwise can get bottlenecked on 
> single connection)
> 
> 
> Diffs
> -
> 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
>  6a13b55e69 
>   llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryInfo.java 
> 00fed15d2b 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java
>  eae8e08540 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
>  50dec4759e 
> 
> 
> Diff: https://reviews.apache.org/r/72499/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Rajesh Balamohan
> 
>



Re: Review Request 72499: HIVE-23446:LLAP: Reduce IPC connection misses to AM for short queries

2020-05-13 Thread Rajesh Balamohan


> On May 14, 2020, 4:31 a.m., Ashutosh Chauhan wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
> > Lines 638 (patched)
> > 
> >
> > Bound this queue by number of executors?

Will fix this in next patch.


- Rajesh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72499/#review220748
---


On May 12, 2020, 12:06 p.m., Rajesh Balamohan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72499/
> ---
> 
> (Updated May 12, 2020, 12:06 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Gopal V.
> 
> 
> Bugs: HIVE-23446
> https://issues.apache.org/jira/browse/HIVE-23446
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Currently UGI pool is maintained at QueryInfo level. However, when short 
> queries and lots of AMs are there, it ends missing IPC connection cache. Too 
> many connections are are also established. Patch tries to avoid that by 
> maintaining this at ContainerRunner level. It retains the current behaviour 
> of having multiple connection to same AM (otherwise can get bottlenecked on 
> single connection)
> 
> 
> Diffs
> -
> 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
>  6a13b55e69 
>   llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryInfo.java 
> 00fed15d2b 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java
>  eae8e08540 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
>  50dec4759e 
> 
> 
> Diff: https://reviews.apache.org/r/72499/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Rajesh Balamohan
> 
>



Re: Review Request 72499: HIVE-23446:LLAP: Reduce IPC connection misses to AM for short queries

2020-05-13 Thread Rajesh Balamohan


> On May 14, 2020, 4:31 a.m., Ashutosh Chauhan wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
> > Lines 610 (patched)
> > 
> >
> > What's the reason for time based expiry? Is it because UGI expires 
> > after 24 hrs?
> > Else, I would have expected long living cache with blocking queue of 
> > bounded size.
> > 
> > Queue should be bounded by number of executors anyways, since having 
> > more connections than executors probably won't be needed.

Since this is based on the AM. So if AM dies after sometime (due to inactivity, 
as in no-DAG submissions), these UGIs will be auto purged after 10 minutes.


> On May 14, 2020, 4:31 a.m., Ashutosh Chauhan wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
> > Lines 646-651 (patched)
> > 
> >
> > Is this logic needed? You already have valueloader in get() which must 
> > return a ugi, so it cant be null.

Yes, value loader is for initial miss. This is to avoid single connection 
becoming a contention for AM communication. 
https://issues.apache.org/jira/browse/HIVE-16634


- Rajesh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72499/#review220748
---


On May 12, 2020, 12:06 p.m., Rajesh Balamohan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72499/
> ---
> 
> (Updated May 12, 2020, 12:06 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Gopal V.
> 
> 
> Bugs: HIVE-23446
> https://issues.apache.org/jira/browse/HIVE-23446
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Currently UGI pool is maintained at QueryInfo level. However, when short 
> queries and lots of AMs are there, it ends missing IPC connection cache. Too 
> many connections are are also established. Patch tries to avoid that by 
> maintaining this at ContainerRunner level. It retains the current behaviour 
> of having multiple connection to same AM (otherwise can get bottlenecked on 
> single connection)
> 
> 
> Diffs
> -
> 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
>  6a13b55e69 
>   llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryInfo.java 
> 00fed15d2b 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java
>  eae8e08540 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
>  50dec4759e 
> 
> 
> Diff: https://reviews.apache.org/r/72499/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Rajesh Balamohan
> 
>



Re: Review Request 72499: HIVE-23446:LLAP: Reduce IPC connection misses to AM for short queries

2020-05-13 Thread Ashutosh Chauhan

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72499/#review220748
---




llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
Lines 610 (patched)


What's the reason for time based expiry? Is it because UGI expires after 24 
hrs?
Else, I would have expected long living cache with blocking queue of 
bounded size.

Queue should be bounded by number of executors anyways, since having more 
connections than executors probably won't be needed.



llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
Lines 617 (patched)


LOG.debug



llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
Lines 638 (patched)


Bound this queue by number of executors?



llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
Lines 640 (patched)


LOG.debug



llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
Lines 646-651 (patched)


Is this logic needed? You already have valueloader in get() which must 
return a ugi, so it cant be null.



llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
Lines 663 (patched)


if its null, then its programming error. Better to not do this null check 
and offer without checking for null.


- Ashutosh Chauhan


On May 12, 2020, 12:06 p.m., Rajesh Balamohan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72499/
> ---
> 
> (Updated May 12, 2020, 12:06 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Gopal V.
> 
> 
> Bugs: HIVE-23446
> https://issues.apache.org/jira/browse/HIVE-23446
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Currently UGI pool is maintained at QueryInfo level. However, when short 
> queries and lots of AMs are there, it ends missing IPC connection cache. Too 
> many connections are are also established. Patch tries to avoid that by 
> maintaining this at ContainerRunner level. It retains the current behaviour 
> of having multiple connection to same AM (otherwise can get bottlenecked on 
> single connection)
> 
> 
> Diffs
> -
> 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
>  6a13b55e69 
>   llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryInfo.java 
> 00fed15d2b 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java
>  eae8e08540 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
>  50dec4759e 
> 
> 
> Diff: https://reviews.apache.org/r/72499/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Rajesh Balamohan
> 
>



[jira] [Created] (HIVE-23467) Add a skip.trash config for HMS to skip trash when deleting external table data

2020-05-13 Thread Sam An (Jira)
Sam An created HIVE-23467:
-

 Summary: Add a skip.trash config for HMS to skip trash when 
deleting external table data
 Key: HIVE-23467
 URL: https://issues.apache.org/jira/browse/HIVE-23467
 Project: Hive
  Issue Type: Improvement
  Components: Hive
Reporter: Sam An
Assignee: Sam An


We have an auto.purge flag, which means skip trash. It can be confusing as we 
have 'external.table.purge'='true' to indicate delete table data when this 
tblproperties is set. 

 

We should make the meaning clearer by introducing a skip trash alias/option. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (HIVE-23465) Investigate why uniquejoin.q is not producing the proper results

2020-05-13 Thread Miklos Gergely (Jira)
Miklos Gergely created HIVE-23465:
-

 Summary: Investigate why uniquejoin.q is not producing the proper 
results
 Key: HIVE-23465
 URL: https://issues.apache.org/jira/browse/HIVE-23465
 Project: Hive
  Issue Type: Sub-task
Reporter: Miklos Gergely


Check [https://reviews.apache.org/r/72491/] for details.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (HIVE-23466) ZK registry base should remove only specific instance instead of host

2020-05-13 Thread Prasanth Jayachandran (Jira)
Prasanth Jayachandran created HIVE-23466:


 Summary: ZK registry base should remove only specific instance 
instead of host
 Key: HIVE-23466
 URL: https://issues.apache.org/jira/browse/HIVE-23466
 Project: Hive
  Issue Type: Bug
Affects Versions: 4.0.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran


When ZKRegistryBase detects new ZK nodes it maintains path based cache and host 
based cache. The host based cached already handles multiple instances running 
in same host. But even if single instance is removed all instances belonging to 
the host are removed. 

Another issue is that, if single host has multiple instances it returns a Set 
with no ordering. Ideally, we want the newest instance to be top of the set 
(use TreeSet maybe?). 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (HIVE-23464) Investigate why udaf_percentile_approx_23.q output has changed

2020-05-13 Thread Miklos Gergely (Jira)
Miklos Gergely created HIVE-23464:
-

 Summary: Investigate why udaf_percentile_approx_23.q output has 
changed
 Key: HIVE-23464
 URL: https://issues.apache.org/jira/browse/HIVE-23464
 Project: Hive
  Issue Type: Sub-task
Reporter: Miklos Gergely


Check [https://reviews.apache.org/r/72491/] for details.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (HIVE-23463) Investigate why the results have changed at udf_mask_show_last_n.q, udf_mask_show_first_n.q, udf_mask_last_n.q, udf_mask_first_n.q, udf_mask.q

2020-05-13 Thread Miklos Gergely (Jira)
Miklos Gergely created HIVE-23463:
-

 Summary: Investigate why the results have changed at 
udf_mask_show_last_n.q, udf_mask_show_first_n.q, udf_mask_last_n.q, 
udf_mask_first_n.q, udf_mask.q
 Key: HIVE-23463
 URL: https://issues.apache.org/jira/browse/HIVE-23463
 Project: Hive
  Issue Type: Sub-task
Reporter: Miklos Gergely






--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (HIVE-23462) Add option to rewrite NTILE to sketch functions

2020-05-13 Thread Zoltan Haindrich (Jira)
Zoltan Haindrich created HIVE-23462:
---

 Summary: Add option to rewrite NTILE to sketch functions
 Key: HIVE-23462
 URL: https://issues.apache.org/jira/browse/HIVE-23462
 Project: Hive
  Issue Type: Sub-task
Reporter: Zoltan Haindrich






--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (HIVE-23461) Needs to capture input/output entities in explainRewrite

2020-05-13 Thread Wenchao Li (Jira)
Wenchao Li created HIVE-23461:
-

 Summary: Needs to capture input/output entities in explainRewrite
 Key: HIVE-23461
 URL: https://issues.apache.org/jira/browse/HIVE-23461
 Project: Hive
  Issue Type: Improvement
Reporter: Wenchao Li


HIVE-18778(CVE-2018-1314) capture input/output entitles in explain semantic 
analyzer so when a query is disallowed by Ranger, Sentry or Sqlstd 
authorizizer, the corresponding explain statement will be disallowed either.

However, ExplainSQRewriteSemanticAnalyzer also uses an instance of 
DDLSemanticAnalyzer to analyze the explain rewrite query.

 
{code:java}
SemanticAnalyzer sem = (SemanticAnalyzer)
 SemanticAnalyzerFactory.get(queryState, input);
sem.analyze(input, ctx);
sem.validate();

The inputs/outputs entities for this query are never set on the instance of 
ExplainSQRewriteSemanticAnalyzer itself and thus is not propagated into the 
HookContext in the calling Driver code. It is a similar issue to 
HIVE-18778.{code}
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Review Request 72506: HIVE-23434 Add option to rewrite PERCENTILE_CONT to sketch functions

2020-05-13 Thread Zoltan Haindrich

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72506/
---

Review request for hive.


Bugs: HIVE-23434
https://issues.apache.org/jira/browse/HIVE-23434


Repository: hive-git


Description
---

d


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 4f2ea9aa7a6 
  itests/src/test/resources/testconfiguration.properties 39e78d6cc8f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DataSketchesFunctions.java 
88653805aa5 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRewriteCountDistinctToDataSketches.java
 c23e2c4938c 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRewriteToDataSketchesRule.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 085de48056b 
  ql/src/test/queries/clientpositive/sketches_rewrite.q  
  ql/src/test/queries/clientpositive/sketches_rewrite_percentile_cont.q 
PRE-CREATION 
  ql/src/test/results/clientpositive/llap/sketches_rewrite.q.out dedcff94529 
  
ql/src/test/results/clientpositive/llap/sketches_rewrite_percentile_cont.q.out 
PRE-CREATION 


Diff: https://reviews.apache.org/r/72506/diff/1/


Testing
---


Thanks,

Zoltan Haindrich



Re: Review Request 72491: Move q tests to TestMiniLlapLocal from TestCliDriver where the output is different, batch 4

2020-05-13 Thread Zoltan Haindrich

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72491/#review220738
---




ql/src/test/results/clientpositive/llap/union35.q.out
Line 18 (original)


resultset order change



ql/src/test/results/clientpositive/llap/union_pos_alias.q.out
Line 461 (original), 462 (patched)


huge resultset change



ql/src/test/results/clientpositive/llap/union_pos_alias.q.out
Lines 1895 (patched)


huge resultset change


- Zoltan Haindrich


On May 11, 2020, 9:05 p.m., Miklos Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72491/
> ---
> 
> (Updated May 11, 2020, 9:05 p.m.)
> 
> 
> Review request for hive, Jesús Camacho Rodríguez, John Sherman, Zoltan 
> Haindrich, Krisztian Kasa, Steve Carlin, and Vineet Garg.
> 
> 
> Bugs: HIVE-23440
> https://issues.apache.org/jira/browse/HIVE-23440
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Move q tests to TestMiniLlapLocal from TestCliDriver where the output is 
> different, batch 4
> 
> 
> Diffs
> -
> 
>   ql/src/test/queries/clientpositive/union.q 3f40a25d49 
>   
> ql/src/test/results/clientpositive/llap/temp_table_merge_dynamic_partition.q.out
>  8b1cfad5ab 
>   
> ql/src/test/results/clientpositive/llap/temp_table_merge_dynamic_partition2.q.out
>  413a3f2a63 
>   
> ql/src/test/results/clientpositive/llap/temp_table_merge_dynamic_partition3.q.out
>  12d5d59fb1 
>   
> ql/src/test/results/clientpositive/llap/temp_table_merge_dynamic_partition4.q.out
>  8ddbb96fca 
>   
> ql/src/test/results/clientpositive/llap/temp_table_merge_dynamic_partition5.q.out
>  7dbf56cf39 
>   ql/src/test/results/clientpositive/llap/temp_table_options1.q.out 
> be31a5a289 
>   
> ql/src/test/results/clientpositive/llap/temp_table_parquet_mixed_partition_formats2.q.out
>  23bb41edfe 
>   ql/src/test/results/clientpositive/llap/temp_table_partition_boolexpr.q.out 
> d4af83b320 
>   
> ql/src/test/results/clientpositive/llap/temp_table_partition_condition_remover.q.out
>  18f5348f0f 
>   ql/src/test/results/clientpositive/llap/temp_table_partition_ctas.q.out 
> bd3574f03f 
>   
> ql/src/test/results/clientpositive/llap/temp_table_partition_multilevels.q.out
>  2ea8bf8631 
>   ql/src/test/results/clientpositive/llap/temp_table_partition_pruning.q.out 
> f6fdd61928 
>   
> ql/src/test/results/clientpositive/llap/temp_table_windowing_expressions.q.out
>  c45f36e988 
>   ql/src/test/results/clientpositive/llap/test_teradatabinaryfile.q.out 
> 75584e9ba2 
>   ql/src/test/results/clientpositive/llap/timestamp.q.out 90a46f58f4 
>   ql/src/test/results/clientpositive/llap/timestamp_comparison3.q.out 
> 3977be77f7 
>   ql/src/test/results/clientpositive/llap/timestamp_ints_casts.q.out 
> 572c49ea72 
>   ql/src/test/results/clientpositive/llap/timestamp_literal.q.out cfcd06f907 
>   ql/src/test/results/clientpositive/llap/timestamptz.q.out 09c50ddf10 
>   ql/src/test/results/clientpositive/llap/truncate_column_list_bucket.q.out 
> c8e40bd447 
>   ql/src/test/results/clientpositive/llap/type_cast_1.q.out 22dad1a0f2 
>   ql/src/test/results/clientpositive/llap/type_widening.q.out f295e66ee9 
>   ql/src/test/results/clientpositive/llap/udaf_binarysetfunctions.q.out 
> 86dbcf6f57 
>   
> ql/src/test/results/clientpositive/llap/udaf_binarysetfunctions_no_cbo.q.out 
> 6857ca9739 
>   ql/src/test/results/clientpositive/llap/udaf_number_format.q.out 822ea784ba 
>   ql/src/test/results/clientpositive/llap/udaf_percentile_approx_23.q.out 
> c200ecf75a 
>   ql/src/test/results/clientpositive/llap/udaf_percentile_cont.q.out 
> 509ae7bfe6 
>   ql/src/test/results/clientpositive/llap/udaf_percentile_disc.q.out 
> e7efcf9302 
>   ql/src/test/results/clientpositive/llap/udf1.q.out 9647770bcd 
>   ql/src/test/results/clientpositive/llap/udf2.q.out bcc2faa16a 
>   ql/src/test/results/clientpositive/llap/udf3.q.out 18abd9560c 
>   ql/src/test/results/clientpositive/llap/udf4.q.out d9b841aab9 
>   ql/src/test/results/clientpositive/llap/udf5.q.out 58a1dab60b 
>   ql/src/test/results/clientpositive/llap/udf6.q.out e6d58324c7 
>   ql/src/test/results/clientpositive/llap/udf7.q.out 44b282f82e 
>   ql/src/test/results/clientpositive/llap/udf8.q.out 8e8ca424b4 
>   ql/src/test/results/clientpositive/llap/udf9.q.out a55b3cdb34 
>   ql/src/test/results/clientpositive/llap/udf_10_trims.q.out 41eefa3e8c 
>   ql/src/test/results/clientpositive/llap/udf_E.q.out 469f396a85 
>   ql/src/test/results/clientpositive/llap/udf_PI.q.out a9ec8c1e06 
>   ql/src/test/results/clientpositive/llap/udf_abs.q.out fee7592ec9 
>   

[jira] [Created] (HIVE-23460) Add qoption to disable qtests

2020-05-13 Thread Zoltan Haindrich (Jira)
Zoltan Haindrich created HIVE-23460:
---

 Summary: Add qoption to disable qtests
 Key: HIVE-23460
 URL: https://issues.apache.org/jira/browse/HIVE-23460
 Project: Hive
  Issue Type: Sub-task
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich
 Attachments: HIVE-23460.01.patch

instead other ways to exclude them... (testconfiguration.properties; 
CliConfig#excludeQuery)

{code}
--! qt:disabled:reason
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (HIVE-23459) Reduce number of listPath calls in AcidUtils::getAcidState

2020-05-13 Thread Rajesh Balamohan (Jira)
Rajesh Balamohan created HIVE-23459:
---

 Summary: Reduce number of listPath calls in AcidUtils::getAcidState
 Key: HIVE-23459
 URL: https://issues.apache.org/jira/browse/HIVE-23459
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan
 Attachments: image-2020-05-13-13-57-27-270.png

There are atleast 3 places where listPaths is invoked for FS (highlighted in 
the follow profile).

!image-2020-05-13-13-57-27-270.png|width=869,height=626!

 

Dir caching works mainly for BI strategy and when there are no-delta files. It 
would be good to consider reducing number of NN calls to reduce getSplits time.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (HIVE-23458) Introduce unified thread pool for scheduled jobs

2020-05-13 Thread Eugene Chung (Jira)
Eugene Chung created HIVE-23458:
---

 Summary: Introduce unified thread pool for scheduled jobs
 Key: HIVE-23458
 URL: https://issues.apache.org/jira/browse/HIVE-23458
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Reporter: Eugene Chung
Assignee: Eugene Chung


As I mentioned in [the comment of 
HIVE-23164|https://issues.apache.org/jira/browse/HIVE-23164?focusedCommentId=17089506=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17089506],
 I've made the unified scheduled executor service like 
org.apache.hadoop.hive.metastore.ThreadPool.

I think it could help
1. to minimize the possibility of making non-daemon threads when developers 
need ScheduledExecutorService
2. to achieve the utilization of server resources because the current situation 
is all of the modules make its own ScheduledExecutorService and all of the 
threads are just using for one job. 
3. an administrator of Hive server by providing hive.exec.scheduler.num.threads 
configuration.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)