[jira] [Commented] (HIVE-21787) Metastore table cache LRU eviction

2019-06-20 Thread Hive QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/HIVE-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16868453#comment-16868453
 ] 

Hive QA commented on HIVE-21787:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972295/HIVE-21787.15.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1183 failed/errored test(s), 15334 tests 
executed
*Failed tests:*
{noformat}
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed 
out) (batchId=184)

[vector_case_when_1.q,tez_nway_join.q,escape2.q,bucket_map_join_tez1.q,insert_update_delete.q,correlationoptimizer3.q,schema_evol_orc_nonvec_part_all_primitive_llap_io.q,external_jdbc_table_partition.q,runtime_stats_merge.q,autoColumnStats_2.q,schema_evol_orc_acid_part_llap_io.q,semijoin6.q,materialized_view_rebuild.q,vectorization_0.q,orc_merge8.q,orc_merge_incompat2.q,vector_outer_join4.q,materialized_view_partitioned.q,orc_merge7.q,vectorized_casts.q,vector_grouping_sets.q,vector_outer_join5.q,schema_evol_orc_acidvec_part_update_llap_io.q,groupby_groupingset_bug.q,bucketmapjoin1.q,truncate_external_force.q,load_dyn_part1.q,groupby_rollup_empty2.q,external_jdbc_table_perf.q,udf_coalesce.q]
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.org.apache.hadoop.hive.cli.TestAccumuloCliDriver
 (batchId=279)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[buckets] 
(batchId=290)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[create_like] 
(batchId=290)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_blobstore]
 (batchId=290)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_local]
 (batchId=290)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_warehouse]
 (batchId=290)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_local_to_blobstore]
 (batchId=290)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_blobstore]
 (batchId=290)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_local]
 (batchId=290)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_warehouse]
 (batchId=290)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_local_to_blobstore]
 (batchId=290)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_directory]
 (batchId=290)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_buckets] 
(batchId=290)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_format_part]
 (batchId=290)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_nonstd_partitions_loc]
 (batchId=290)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[parquet_buckets]
 (batchId=290)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[parquet_format_part]
 (batchId=290)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[parquet_nonstd_partitions_loc]
 (batchId=290)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[rcfile_buckets] 
(batchId=290)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[rcfile_format_part]
 (batchId=290)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[rcfile_nonstd_partitions_loc]
 (batchId=290)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_stats3] (batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_stats4] (batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats] 
(batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[add_part_exist] 
(batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[add_part_multiple] 
(batchId=77)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter2] (batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_file_format] 
(batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_numbuckets_partitioned_table2_h23]
 (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_partition_clusterby_sortby]
 (batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_partition_coltype] 
(batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_partition_update_status]
 (batchId=99)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_rename_table] 
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_skewed_table] 
(batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_cascade] 
(batchId=97)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_partition_drop]
 (batchId=16)

[jira] [Commented] (HIVE-21787) Metastore table cache LRU eviction

2019-06-20 Thread Hive QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/HIVE-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16868412#comment-16868412
 ] 

Hive QA commented on HIVE-21787:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
14s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
45s{color} | {color:blue} itests/util in master has 44 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
31s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
14s{color} | {color:red} itests/util: The patch generated 2 new + 18 unchanged 
- 0 fixed = 20 total (was 18) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
16s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 52s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17669/dev-support/hive-personality.sh
 |
| git revision | master / 6b62a96 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17669/yetus/diff-checkstyle-itests_util.txt
 |
| modules | C: itests/util U: itests/util |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17669/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Metastore table cache LRU eviction
> --
>
> Key: HIVE-21787
> URL: https://issues.apache.org/jira/browse/HIVE-21787
> Project: Hive
>  Issue Type: New Feature
>  Components: Standalone Metastore
>Reporter: Sam An
>Assignee: Sam An
>Priority: Major
> Attachments: HIVE-21787.1.patch, HIVE-21787.10.patch, 
> HIVE-21787.11.patch, HIVE-21787.13.patch, HIVE-21787.14.patch, 
> HIVE-21787.15.patch, HIVE-21787.2.patch, HIVE-21787.3.patch, 
> HIVE-21787.4.patch, HIVE-21787.5.patch, HIVE-21787.6.patch, 
> HIVE-21787.7.patch, HIVE-21787.8.patch, HIVE-21787.9.patch
>
>
> Metastore currently uses black/white list to specify patterns of tables to 
> load into the cache. Cache is loaded in one shot "prewarm", and updated by a 
> background thread. This is not a very efficient design. 
> In this feature, we try to enhance the cache for Tables with LRU to improve 
> cache utilization.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21787) Metastore table cache LRU eviction

2019-06-20 Thread Daniel Dai (JIRA) 


[ 
https://issues.apache.org/jira/browse/HIVE-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16868286#comment-16868286
 ] 

Daniel Dai commented on HIVE-21787:
---

Patch committed to master. Also attach HIVE-21787.15.patch to kick a ptest run 
with cache enabled.

> Metastore table cache LRU eviction
> --
>
> Key: HIVE-21787
> URL: https://issues.apache.org/jira/browse/HIVE-21787
> Project: Hive
>  Issue Type: New Feature
>  Components: Standalone Metastore
>Reporter: Sam An
>Assignee: Sam An
>Priority: Major
> Attachments: HIVE-21787.1.patch, HIVE-21787.10.patch, 
> HIVE-21787.11.patch, HIVE-21787.13.patch, HIVE-21787.14.patch, 
> HIVE-21787.15.patch, HIVE-21787.2.patch, HIVE-21787.3.patch, 
> HIVE-21787.4.patch, HIVE-21787.5.patch, HIVE-21787.6.patch, 
> HIVE-21787.7.patch, HIVE-21787.8.patch, HIVE-21787.9.patch
>
>
> Metastore currently uses black/white list to specify patterns of tables to 
> load into the cache. Cache is loaded in one shot "prewarm", and updated by a 
> background thread. This is not a very efficient design. 
> In this feature, we try to enhance the cache for Tables with LRU to improve 
> cache utilization.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21787) Metastore table cache LRU eviction

2019-06-19 Thread Hive QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/HIVE-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16868258#comment-16868258
 ] 

Hive QA commented on HIVE-21787:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972279/HIVE-21787.14.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 16170 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/17665/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17665/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17665/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12972279 - PreCommit-HIVE-Build

> Metastore table cache LRU eviction
> --
>
> Key: HIVE-21787
> URL: https://issues.apache.org/jira/browse/HIVE-21787
> Project: Hive
>  Issue Type: New Feature
>  Components: Standalone Metastore
>Reporter: Sam An
>Assignee: Sam An
>Priority: Major
> Attachments: HIVE-21787.1.patch, HIVE-21787.10.patch, 
> HIVE-21787.11.patch, HIVE-21787.13.patch, HIVE-21787.14.patch, 
> HIVE-21787.2.patch, HIVE-21787.3.patch, HIVE-21787.4.patch, 
> HIVE-21787.5.patch, HIVE-21787.6.patch, HIVE-21787.7.patch, 
> HIVE-21787.8.patch, HIVE-21787.9.patch
>
>
> Metastore currently uses black/white list to specify patterns of tables to 
> load into the cache. Cache is loaded in one shot "prewarm", and updated by a 
> background thread. This is not a very efficient design. 
> In this feature, we try to enhance the cache for Tables with LRU to improve 
> cache utilization.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21787) Metastore table cache LRU eviction

2019-06-19 Thread Hive QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/HIVE-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16868219#comment-16868219
 ] 

Hive QA commented on HIVE-21787:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  1m 
13s{color} | {color:blue} standalone-metastore/metastore-server in master has 
184 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
42s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
50s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
45s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m 
41s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 41s{color} 
| {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
17s{color} | {color:red} standalone-metastore/metastore-server: The patch 
generated 6 new + 44 unchanged - 165 fixed = 50 total (was 209) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
20s{color} | {color:green} standalone-metastore/metastore-server generated 0 
new + 179 unchanged - 5 fixed = 179 total (was 184) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
42s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 14s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17665/dev-support/hive-personality.sh
 |
| git revision | master / cd42db4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17665/yetus/patch-mvninstall-itests_hive-unit.txt
 |
| compile | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17665/yetus/patch-compile-itests_hive-unit.txt
 |
| javac | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17665/yetus/patch-compile-itests_hive-unit.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17665/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
 |
| findbugs | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17665/yetus/patch-findbugs-itests_hive-unit.txt
 |
| modules | C: standalone-metastore/metastore-server itests/hive-unit U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17665/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Metastore table cache LRU eviction
> --
>
> Key: HIVE-21787
> URL: https://issues.apache.org/jira/browse/HIVE-21787
> Project: Hive
>  Issue Type: New Feature
>

[jira] [Commented] (HIVE-21787) Metastore table cache LRU eviction

2019-06-19 Thread Hive QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/HIVE-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16868161#comment-16868161
 ] 

Hive QA commented on HIVE-21787:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972274/HIVE-21787.13.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/17663/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17663/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17663/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL 
https://issues.apache.org/jira/secure/attachment/12972274/HIVE-21787.13.patch 
was found in seen patch url's cache and a test was probably run already on it. 
Aborting...
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12972274 - PreCommit-HIVE-Build

> Metastore table cache LRU eviction
> --
>
> Key: HIVE-21787
> URL: https://issues.apache.org/jira/browse/HIVE-21787
> Project: Hive
>  Issue Type: New Feature
>  Components: Standalone Metastore
>Reporter: Sam An
>Assignee: Sam An
>Priority: Major
> Attachments: HIVE-21787.1.patch, HIVE-21787.10.patch, 
> HIVE-21787.11.patch, HIVE-21787.13.patch, HIVE-21787.2.patch, 
> HIVE-21787.3.patch, HIVE-21787.4.patch, HIVE-21787.5.patch, 
> HIVE-21787.6.patch, HIVE-21787.7.patch, HIVE-21787.8.patch, HIVE-21787.9.patch
>
>
> Metastore currently uses black/white list to specify patterns of tables to 
> load into the cache. Cache is loaded in one shot "prewarm", and updated by a 
> background thread. This is not a very efficient design. 
> In this feature, we try to enhance the cache for Tables with LRU to improve 
> cache utilization.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21787) Metastore table cache LRU eviction

2019-06-19 Thread Hive QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/HIVE-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16868160#comment-16868160
 ] 

Hive QA commented on HIVE-21787:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972274/HIVE-21787.13.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 16170 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/17662/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17662/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17662/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12972274 - PreCommit-HIVE-Build

> Metastore table cache LRU eviction
> --
>
> Key: HIVE-21787
> URL: https://issues.apache.org/jira/browse/HIVE-21787
> Project: Hive
>  Issue Type: New Feature
>  Components: Standalone Metastore
>Reporter: Sam An
>Assignee: Sam An
>Priority: Major
> Attachments: HIVE-21787.1.patch, HIVE-21787.10.patch, 
> HIVE-21787.11.patch, HIVE-21787.13.patch, HIVE-21787.2.patch, 
> HIVE-21787.3.patch, HIVE-21787.4.patch, HIVE-21787.5.patch, 
> HIVE-21787.6.patch, HIVE-21787.7.patch, HIVE-21787.8.patch, HIVE-21787.9.patch
>
>
> Metastore currently uses black/white list to specify patterns of tables to 
> load into the cache. Cache is loaded in one shot "prewarm", and updated by a 
> background thread. This is not a very efficient design. 
> In this feature, we try to enhance the cache for Tables with LRU to improve 
> cache utilization.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21787) Metastore table cache LRU eviction

2019-06-19 Thread Hive QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/HIVE-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16868143#comment-16868143
 ] 

Hive QA commented on HIVE-21787:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  2m 
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  1m 
11s{color} | {color:blue} standalone-metastore/metastore-server in master has 
184 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
42s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
46s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
41s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m 
43s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 43s{color} 
| {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
18s{color} | {color:red} standalone-metastore/metastore-server: The patch 
generated 6 new + 44 unchanged - 165 fixed = 50 total (was 209) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 
22s{color} | {color:red} standalone-metastore/metastore-server generated 1 new 
+ 179 unchanged - 5 fixed = 180 total (was 184) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
35s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
15s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 25s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
|  |  Invocation of toString on SharedCache$TableWrapper.getSdHash() in 
org.apache.hadoop.hive.metastore.cache.SharedCache$1.onRemoval(RemovalNotification)
  At SharedCache.java:in 
org.apache.hadoop.hive.metastore.cache.SharedCache$1.onRemoval(RemovalNotification)
  At SharedCache.java:[line 207] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17662/dev-support/hive-personality.sh
 |
| git revision | master / cd42db4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17662/yetus/patch-mvninstall-itests_hive-unit.txt
 |
| compile | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17662/yetus/patch-compile-itests_hive-unit.txt
 |
| javac | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17662/yetus/patch-compile-itests_hive-unit.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17662/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
 |
| findbugs | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17662/yetus/new-findbugs-standalone-metastore_metastore-server.html
 |
| findbugs | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17662/yetus/patch-findbugs-itests_hive-unit.txt
 |
|

[jira] [Commented] (HIVE-21787) Metastore table cache LRU eviction

2019-06-19 Thread Hive QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/HIVE-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16868005#comment-16868005
 ] 

Hive QA commented on HIVE-21787:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972237/HIVE-21787.11.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 16170 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/17657/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17657/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17657/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12972237 - PreCommit-HIVE-Build

> Metastore table cache LRU eviction
> --
>
> Key: HIVE-21787
> URL: https://issues.apache.org/jira/browse/HIVE-21787
> Project: Hive
>  Issue Type: New Feature
>  Components: Standalone Metastore
>Reporter: Sam An
>Assignee: Sam An
>Priority: Major
> Attachments: HIVE-21787.1.patch, HIVE-21787.10.patch, 
> HIVE-21787.11.patch, HIVE-21787.2.patch, HIVE-21787.3.patch, 
> HIVE-21787.4.patch, HIVE-21787.5.patch, HIVE-21787.6.patch, 
> HIVE-21787.7.patch, HIVE-21787.8.patch, HIVE-21787.9.patch
>
>
> Metastore currently uses black/white list to specify patterns of tables to 
> load into the cache. Cache is loaded in one shot "prewarm", and updated by a 
> background thread. This is not a very efficient design. 
> In this feature, we try to enhance the cache for Tables with LRU to improve 
> cache utilization.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21787) Metastore table cache LRU eviction

2019-06-19 Thread Hive QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/HIVE-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16867956#comment-16867956
 ] 

Hive QA commented on HIVE-21787:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red}  0m 16s{color} 
| {color:red} 
/data/hiveptest/logs/PreCommit-HIVE-Build-17657/patches/PreCommit-HIVE-Build-17657.patch
 does not apply to master. Rebase required? Wrong Branch? See 
http://cwiki.apache.org/confluence/display/Hive/HowToContribute for help. 
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17657/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Metastore table cache LRU eviction
> --
>
> Key: HIVE-21787
> URL: https://issues.apache.org/jira/browse/HIVE-21787
> Project: Hive
>  Issue Type: New Feature
>  Components: Standalone Metastore
>Reporter: Sam An
>Assignee: Sam An
>Priority: Major
> Attachments: HIVE-21787.1.patch, HIVE-21787.10.patch, 
> HIVE-21787.11.patch, HIVE-21787.2.patch, HIVE-21787.3.patch, 
> HIVE-21787.4.patch, HIVE-21787.5.patch, HIVE-21787.6.patch, 
> HIVE-21787.7.patch, HIVE-21787.8.patch, HIVE-21787.9.patch
>
>
> Metastore currently uses black/white list to specify patterns of tables to 
> load into the cache. Cache is loaded in one shot "prewarm", and updated by a 
> background thread. This is not a very efficient design. 
> In this feature, we try to enhance the cache for Tables with LRU to improve 
> cache utilization.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21787) Metastore table cache LRU eviction

2019-06-19 Thread Hive QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/HIVE-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16867287#comment-16867287
 ] 

Hive QA commented on HIVE-21787:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972151/HIVE-21787.10.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16170 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.cache.TestCachedStore.testSharedStoreTable 
(batchId=232)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/17643/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17643/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17643/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12972151 - PreCommit-HIVE-Build

> Metastore table cache LRU eviction
> --
>
> Key: HIVE-21787
> URL: https://issues.apache.org/jira/browse/HIVE-21787
> Project: Hive
>  Issue Type: New Feature
>  Components: Standalone Metastore
>Reporter: Sam An
>Assignee: Sam An
>Priority: Major
> Attachments: HIVE-21787.1.patch, HIVE-21787.10.patch, 
> HIVE-21787.2.patch, HIVE-21787.3.patch, HIVE-21787.4.patch, 
> HIVE-21787.5.patch, HIVE-21787.6.patch, HIVE-21787.7.patch, 
> HIVE-21787.8.patch, HIVE-21787.9.patch
>
>
> Metastore currently uses black/white list to specify patterns of tables to 
> load into the cache. Cache is loaded in one shot "prewarm", and updated by a 
> background thread. This is not a very efficient design. 
> In this feature, we try to enhance the cache for Tables with LRU to improve 
> cache utilization.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21787) Metastore table cache LRU eviction

2019-06-18 Thread Hive QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/HIVE-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16867246#comment-16867246
 ] 

Hive QA commented on HIVE-21787:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red}  0m 16s{color} 
| {color:red} 
/data/hiveptest/logs/PreCommit-HIVE-Build-17643/patches/PreCommit-HIVE-Build-17643.patch
 does not apply to master. Rebase required? Wrong Branch? See 
http://cwiki.apache.org/confluence/display/Hive/HowToContribute for help. 
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17643/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Metastore table cache LRU eviction
> --
>
> Key: HIVE-21787
> URL: https://issues.apache.org/jira/browse/HIVE-21787
> Project: Hive
>  Issue Type: New Feature
>  Components: Standalone Metastore
>Reporter: Sam An
>Assignee: Sam An
>Priority: Major
> Attachments: HIVE-21787.1.patch, HIVE-21787.10.patch, 
> HIVE-21787.2.patch, HIVE-21787.3.patch, HIVE-21787.4.patch, 
> HIVE-21787.5.patch, HIVE-21787.6.patch, HIVE-21787.7.patch, 
> HIVE-21787.8.patch, HIVE-21787.9.patch
>
>
> Metastore currently uses black/white list to specify patterns of tables to 
> load into the cache. Cache is loaded in one shot "prewarm", and updated by a 
> background thread. This is not a very efficient design. 
> In this feature, we try to enhance the cache for Tables with LRU to improve 
> cache utilization.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)