[jira] [Updated] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22736: --- Attachment: HIVE-22736.patch Status: Patch Available (was: In Progress) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22736: --- Status: In Progress (was: Patch Available) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024122#comment-17024122 ] Hive QA commented on HIVE-22736: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12991869/HIVE-22736.patch {color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 17959 tests executed *Failed tests:* {noformat} org.apache.hive.jdbc.TestJdbcWithMiniLlapVectorArrow.testComplexQuery (batchId=291) org.apache.hive.jdbc.TestJdbcWithMiniLlapVectorArrow.testDataTypes (batchId=291) org.apache.hive.jdbc.TestJdbcWithMiniLlapVectorArrow.testEscapedStrings (batchId=291) org.apache.hive.jdbc.TestJdbcWithMiniLlapVectorArrow.testKillQuery (batchId=291) org.apache.hive.jdbc.TestJdbcWithMiniLlapVectorArrow.testLlapInputFormatEndToEnd (batchId=291) org.apache.hive.jdbc.TestJdbcWithMiniLlapVectorArrow.testNonAsciiStrings (batchId=291) org.apache.hive.jdbc.TestJdbcWithMiniLlapVectorArrow.testTypesNestedInListWithLimitAndFilters (batchId=291) org.apache.hive.jdbc.TestJdbcWithMiniLlapVectorArrow.testTypesNestedInMapWithLimitAndFilters (batchId=291) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/20349/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20349/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20349/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 8 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12991869 - PreCommit-HIVE-Build > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024109#comment-17024109 ] Hive QA commented on HIVE-22736: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 29s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 18s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 10s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 58s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 16s{color} | {color:blue} shims/common in master has 6 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 20s{color} | {color:blue} shims/0.23 in master has 7 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m 11s{color} | {color:blue} standalone-metastore/metastore-common in master has 37 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 32s{color} | {color:blue} common in master has 63 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m 8s{color} | {color:blue} standalone-metastore/metastore-server in master has 181 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 50s{color} | {color:blue} ql in master has 1533 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 38s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 2s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 9s{color} | {color:green} shims/common: The patch generated 0 new + 93 unchanged - 1 fixed = 93 total (was 94) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 9s{color} | {color:green} The patch 0.23 passed checkstyle {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 11s{color} | {color:red} standalone-metastore/metastore-common: The patch generated 4 new + 98 unchanged - 0 fixed = 102 total (was 98) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 14s{color} | {color:green} The patch common passed checkstyle {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 19s{color} | {color:green} The patch metastore-server passed checkstyle {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 37s{color} | {color:green} The patch ql passed checkstyle {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 16s{color} | {color:green} The patch hive-unit passed checkstyle {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 0s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch
[jira] [Commented] (HIVE-22366) Multiple metastore calls for same table and constraints during planning
[ https://issues.apache.org/jira/browse/HIVE-22366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024099#comment-17024099 ] Jesus Camacho Rodriguez commented on HIVE-22366: Left a couple of small comments. +1 > Multiple metastore calls for same table and constraints during planning > --- > > Key: HIVE-22366 > URL: https://issues.apache.org/jira/browse/HIVE-22366 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22366.1.patch, HIVE-22366.2.patch, > HIVE-22366.3.patch, HIVE-22366.4.patch, HIVE-22366.5.patch, > HIVE-22366.6.patch, HIVE-22366.7.patch > > Time Spent: 20m > Remaining Estimate: 0h > > e.g. for a simple query like > {code:sql} > explain select count(i) from test2 group by j limit 3; > {code} > planner makes the following calls: > {code} > 2019-10-17T22:44:49,892 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr cmd=get_table : > tbl=hive.default.test2 > 2019-10-17T22:44:49,908 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr > cmd=get_primary_keys : tbl=hive.default.test2 > 2019-10-17T22:44:49,910 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr > cmd=get_foreign_keys : parentdb=null parenttbl=null foreigndb=default > foreigntbl=test2 > 2019-10-17T22:44:49,911 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr > cmd=get_unique_constraints : tbl=hive.default.test2 > 2019-10-17T22:44:49,913 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr > cmd=get_not_null_constraints : tbl=hive.default.test2 > 2019-10-17T22:44:49,979 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr > cmd=get_partitions : tbl=hive.default.test2 > 2019-10-17T22:44:49,997 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr > cmd=get_aggr_stats_for: table=hive.default.test2 > 2019-10-17T22:44:50,014 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr cmd=get_table : > tbl=hive.default.test2 > 2019-10-17T22:44:50,023 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr > cmd=get_primary_keys : tbl=hive.default.test2 > 2019-10-17T22:44:50,024 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr > cmd=get_foreign_keys : parentdb=null parenttbl=null foreigndb=default > foreigntbl=test2 > 2019-10-17T22:44:50,025 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr > cmd=get_unique_constraints : tbl=hive.default.test2 > 2019-10-17T22:44:50,026 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr > cmd=get_not_null_constraints : tbl=hive.default.test2 > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22366) Multiple metastore calls for same table and constraints during planning
[ https://issues.apache.org/jira/browse/HIVE-22366?focusedWorklogId=377479=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377479 ] ASF GitHub Bot logged work on HIVE-22366: - Author: ASF GitHub Bot Created on: 27/Jan/20 05:41 Start Date: 27/Jan/20 05:41 Worklog Time Spent: 10m Work Description: jcamachor commented on pull request #888: HIVE-22366: Multiple metastore calls for same table and constraints d… URL: https://github.com/apache/hive/pull/888#discussion_r371064490 ## File path: ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java ## @@ -121,6 +121,16 @@ private transient DefaultConstraint dc; private transient CheckConstraint cc; + /** Constraint related flags + * This is to track if constraints are retrieved from metastore or not + */ + private boolean isPKFetched=false; Review comment: These can be transient. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377479) Time Spent: 20m (was: 10m) > Multiple metastore calls for same table and constraints during planning > --- > > Key: HIVE-22366 > URL: https://issues.apache.org/jira/browse/HIVE-22366 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22366.1.patch, HIVE-22366.2.patch, > HIVE-22366.3.patch, HIVE-22366.4.patch, HIVE-22366.5.patch, > HIVE-22366.6.patch, HIVE-22366.7.patch > > Time Spent: 20m > Remaining Estimate: 0h > > e.g. for a simple query like > {code:sql} > explain select count(i) from test2 group by j limit 3; > {code} > planner makes the following calls: > {code} > 2019-10-17T22:44:49,892 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr cmd=get_table : > tbl=hive.default.test2 > 2019-10-17T22:44:49,908 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr > cmd=get_primary_keys : tbl=hive.default.test2 > 2019-10-17T22:44:49,910 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr > cmd=get_foreign_keys : parentdb=null parenttbl=null foreigndb=default > foreigntbl=test2 > 2019-10-17T22:44:49,911 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr > cmd=get_unique_constraints : tbl=hive.default.test2 > 2019-10-17T22:44:49,913 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr > cmd=get_not_null_constraints : tbl=hive.default.test2 > 2019-10-17T22:44:49,979 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr > cmd=get_partitions : tbl=hive.default.test2 > 2019-10-17T22:44:49,997 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr > cmd=get_aggr_stats_for: table=hive.default.test2 > 2019-10-17T22:44:50,014 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr cmd=get_table : > tbl=hive.default.test2 > 2019-10-17T22:44:50,023 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr > cmd=get_primary_keys : tbl=hive.default.test2 > 2019-10-17T22:44:50,024 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr > cmd=get_foreign_keys : parentdb=null parenttbl=null foreigndb=default > foreigntbl=test2 > 2019-10-17T22:44:50,025 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr > cmd=get_unique_constraints : tbl=hive.default.test2 > 2019-10-17T22:44:50,026 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr > cmd=get_not_null_constraints : tbl=hive.default.test2 > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22366) Multiple metastore calls for same table and constraints during planning
[ https://issues.apache.org/jira/browse/HIVE-22366?focusedWorklogId=377480=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377480 ] ASF GitHub Bot logged work on HIVE-22366: - Author: ASF GitHub Bot Created on: 27/Jan/20 05:41 Start Date: 27/Jan/20 05:41 Worklog Time Spent: 10m Work Description: jcamachor commented on pull request #888: HIVE-22366: Multiple metastore calls for same table and constraints d… URL: https://github.com/apache/hive/pull/888#discussion_r371065177 ## File path: ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java ## @@ -1126,71 +1149,167 @@ public Boolean isOutdatedForRewriting() { return outdatedForRewritingMaterializedView; } - /* These are only populated during optimization and describing */ + public ColumnStatistics getColStats() { +return tTable.isSetColStats() ? tTable.getColStats() : null; + } + + /** + * Setup the table level stats as if the table is new. Used when setting up Table for a new + * table or during replication. + */ + public void setStatsStateLikeNewTable() { +if (isPartitioned()) { + StatsSetupConst.setStatsStateForCreateTable(getParameters(), null, + StatsSetupConst.FALSE); +} else { + StatsSetupConst.setStatsStateForCreateTable(getParameters(), + MetaStoreUtils.getColumnNames(getCols()), StatsSetupConst.TRUE); +} + } + + /** Constraints related methods + * Note that set apis are used by DESCRIBE only, although get apis return RELY or ENABLE + * constraints DESCRIBE could set all type of constraints + * */ + + /* This only return PK with RELY */ public PrimaryKeyInfo getPrimaryKeyInfo() { +if(!this.isPKFetched) { + try { +pki = Hive.get().getReliablePrimaryKeys(this.getDbName(), this.getTableName()); +this.isPKFetched = true; + } catch (HiveException e) { +LOG.warn("Cannot retrieve PK info for table : " + this.getTableName() ++ " ignoring exception: " + e); + } +} return pki; } public void setPrimaryKeyInfo(PrimaryKeyInfo pki) { this.pki = pki; +this.isPKFetched = true; } + /* This only return FK with RELY */ public ForeignKeyInfo getForeignKeyInfo() { Review comment: This solution will still make different calls to retrieve each type of constraints. Should we create a follow-up for that? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377480) Time Spent: 20m (was: 10m) > Multiple metastore calls for same table and constraints during planning > --- > > Key: HIVE-22366 > URL: https://issues.apache.org/jira/browse/HIVE-22366 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22366.1.patch, HIVE-22366.2.patch, > HIVE-22366.3.patch, HIVE-22366.4.patch, HIVE-22366.5.patch, > HIVE-22366.6.patch, HIVE-22366.7.patch > > Time Spent: 20m > Remaining Estimate: 0h > > e.g. for a simple query like > {code:sql} > explain select count(i) from test2 group by j limit 3; > {code} > planner makes the following calls: > {code} > 2019-10-17T22:44:49,892 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr cmd=get_table : > tbl=hive.default.test2 > 2019-10-17T22:44:49,908 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr > cmd=get_primary_keys : tbl=hive.default.test2 > 2019-10-17T22:44:49,910 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr > cmd=get_foreign_keys : parentdb=null parenttbl=null foreigndb=default > foreigntbl=test2 > 2019-10-17T22:44:49,911 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr > cmd=get_unique_constraints : tbl=hive.default.test2 > 2019-10-17T22:44:49,913 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr > cmd=get_not_null_constraints : tbl=hive.default.test2 > 2019-10-17T22:44:49,979 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr > cmd=get_partitions :
[jira] [Updated] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22736: --- Attachment: HIVE-22736.patch Status: Patch Available (was: In Progress) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22736: --- Status: In Progress (was: Patch Available) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22774) Add toggle to disable scheduled query service in HMS
[ https://issues.apache.org/jira/browse/HIVE-22774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024080#comment-17024080 ] Jesus Camacho Rodriguez commented on HIVE-22774: I see this patch fixes some other issues too. Can we link them all to this one? A small comment that can be addressed in follow-up patch: L163 is commented out ({{Thread.currentThread().interrupt()}}), but I think that should go into the the while condition? Other than that, patch LGTM. +1 > Add toggle to disable scheduled query service in HMS > > > Key: HIVE-22774 > URL: https://issues.apache.org/jira/browse/HIVE-22774 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-22774.01.patch, HIVE-22774.02.patch, > HIVE-22774.03.patch > > > * currently it's enabled by default -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22538) RS deduplication does not always enforce hive.optimize.reducededuplication.min.reducer
[ https://issues.apache.org/jira/browse/HIVE-22538?focusedWorklogId=377468=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377468 ] ASF GitHub Bot logged work on HIVE-22538: - Author: ASF GitHub Bot Created on: 27/Jan/20 04:37 Start Date: 27/Jan/20 04:37 Worklog Time Spent: 10m Work Description: jcamachor commented on pull request #877: HIVE-22538: RS deduplication does not always enforce hive.optimize.reducededuplication.min.reducer URL: https://github.com/apache/hive/pull/877#discussion_r371061780 ## File path: ql/src/test/results/clientpositive/autoColumnStats_4.q.out ## @@ -128,16 +127,20 @@ STAGE PLANS: Statistics: Num rows: 10 Data size: 1728 Basic stats: COMPLETE Column stats: COMPLETE Group By Operator aggregations: compute_stats(a, 'hll'), compute_stats(b, 'hll') - minReductionHashAggr: 0.99 - mode: hash + mode: complete Review comment: Why is the GBy operator changing to complete mode here? That could potentially slow down the insert. Does it have anything to do with the size (numrows) of the insert? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377468) Time Spent: 0.5h (was: 20m) > RS deduplication does not always enforce > hive.optimize.reducededuplication.min.reducer > -- > > Key: HIVE-22538 > URL: https://issues.apache.org/jira/browse/HIVE-22538 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22538.2.patch, HIVE-22538.3.patch, > HIVE-22538.4.patch, HIVE-22538.5.patch, HIVE-22538.6.patch, > HIVE-22538.6.patch, HIVE-22538.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > For transactional tables, that property might be overriden to 1, which can > lead to merging final aggregation into a single stage (hence leading to > performance degradation). For instance, when autogather column stats is > enabled, this can happen for the following query: > {code} > set hive.support.concurrency=true; > set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; > EXPLAIN > CREATE TABLE x STORED AS ORC TBLPROPERTIES('transactional'='true') AS > SELECT * FROM SRC x CLUSTER BY x.key; > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22538) RS deduplication does not always enforce hive.optimize.reducededuplication.min.reducer
[ https://issues.apache.org/jira/browse/HIVE-22538?focusedWorklogId=377469=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377469 ] ASF GitHub Bot logged work on HIVE-22538: - Author: ASF GitHub Bot Created on: 27/Jan/20 04:37 Start Date: 27/Jan/20 04:37 Worklog Time Spent: 10m Work Description: jcamachor commented on pull request #877: HIVE-22538: RS deduplication does not always enforce hive.optimize.reducededuplication.min.reducer URL: https://github.com/apache/hive/pull/877#discussion_r371061414 ## File path: ql/src/test/results/clientpositive/acid_table_directories_test.q.out ## @@ -154,6 +154,7 @@ POSTHOOK: Input: default@acidparttbl@p=200 ### ACID BASE DIR ### ### ACID BASE DIR ### ### ACID BASE DIR ### +### ACID BASE DIR ### Review comment: Is this expected? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377469) Time Spent: 0.5h (was: 20m) > RS deduplication does not always enforce > hive.optimize.reducededuplication.min.reducer > -- > > Key: HIVE-22538 > URL: https://issues.apache.org/jira/browse/HIVE-22538 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22538.2.patch, HIVE-22538.3.patch, > HIVE-22538.4.patch, HIVE-22538.5.patch, HIVE-22538.6.patch, > HIVE-22538.6.patch, HIVE-22538.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > For transactional tables, that property might be overriden to 1, which can > lead to merging final aggregation into a single stage (hence leading to > performance degradation). For instance, when autogather column stats is > enabled, this can happen for the following query: > {code} > set hive.support.concurrency=true; > set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; > EXPLAIN > CREATE TABLE x STORED AS ORC TBLPROPERTIES('transactional'='true') AS > SELECT * FROM SRC x CLUSTER BY x.key; > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22538) RS deduplication does not always enforce hive.optimize.reducededuplication.min.reducer
[ https://issues.apache.org/jira/browse/HIVE-22538?focusedWorklogId=377472=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377472 ] ASF GitHub Bot logged work on HIVE-22538: - Author: ASF GitHub Bot Created on: 27/Jan/20 04:37 Start Date: 27/Jan/20 04:37 Worklog Time Spent: 10m Work Description: jcamachor commented on pull request #877: HIVE-22538: RS deduplication does not always enforce hive.optimize.reducededuplication.min.reducer URL: https://github.com/apache/hive/pull/877#discussion_r371062078 ## File path: ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction_3.q.out ## @@ -261,31 +260,21 @@ STAGE PLANS: Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: compute_stats(a, 'hll'), compute_stats(b, 'hll') -minReductionHashAggr: 0.99 -mode: hash +mode: complete Review comment: Same as above... complete here would imply a single instance to write data to the table, which could slow down execution. What is the root cause for this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377472) Time Spent: 40m (was: 0.5h) > RS deduplication does not always enforce > hive.optimize.reducededuplication.min.reducer > -- > > Key: HIVE-22538 > URL: https://issues.apache.org/jira/browse/HIVE-22538 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22538.2.patch, HIVE-22538.3.patch, > HIVE-22538.4.patch, HIVE-22538.5.patch, HIVE-22538.6.patch, > HIVE-22538.6.patch, HIVE-22538.patch > > Time Spent: 40m > Remaining Estimate: 0h > > For transactional tables, that property might be overriden to 1, which can > lead to merging final aggregation into a single stage (hence leading to > performance degradation). For instance, when autogather column stats is > enabled, this can happen for the following query: > {code} > set hive.support.concurrency=true; > set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; > EXPLAIN > CREATE TABLE x STORED AS ORC TBLPROPERTIES('transactional'='true') AS > SELECT * FROM SRC x CLUSTER BY x.key; > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22538) RS deduplication does not always enforce hive.optimize.reducededuplication.min.reducer
[ https://issues.apache.org/jira/browse/HIVE-22538?focusedWorklogId=377470=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377470 ] ASF GitHub Bot logged work on HIVE-22538: - Author: ASF GitHub Bot Created on: 27/Jan/20 04:37 Start Date: 27/Jan/20 04:37 Worklog Time Spent: 10m Work Description: jcamachor commented on pull request #877: HIVE-22538: RS deduplication does not always enforce hive.optimize.reducededuplication.min.reducer URL: https://github.com/apache/hive/pull/877#discussion_r371062397 ## File path: ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction_3.q.out ## @@ -814,30 +791,20 @@ STAGE PLANS: Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: compute_stats(a, 'hll'), compute_stats(b, 'hll') -minReductionHashAggr: 0.99 -mode: hash +mode: complete Review comment: It seems this complete mode is only happening for write type INSERT... I am wondering whether we are setting up the number of reducers to 1 too aggressively since INSERT does not need a single instance. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377470) Time Spent: 0.5h (was: 20m) > RS deduplication does not always enforce > hive.optimize.reducededuplication.min.reducer > -- > > Key: HIVE-22538 > URL: https://issues.apache.org/jira/browse/HIVE-22538 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22538.2.patch, HIVE-22538.3.patch, > HIVE-22538.4.patch, HIVE-22538.5.patch, HIVE-22538.6.patch, > HIVE-22538.6.patch, HIVE-22538.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > For transactional tables, that property might be overriden to 1, which can > lead to merging final aggregation into a single stage (hence leading to > performance degradation). For instance, when autogather column stats is > enabled, this can happen for the following query: > {code} > set hive.support.concurrency=true; > set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; > EXPLAIN > CREATE TABLE x STORED AS ORC TBLPROPERTIES('transactional'='true') AS > SELECT * FROM SRC x CLUSTER BY x.key; > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22538) RS deduplication does not always enforce hive.optimize.reducededuplication.min.reducer
[ https://issues.apache.org/jira/browse/HIVE-22538?focusedWorklogId=377471=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377471 ] ASF GitHub Bot logged work on HIVE-22538: - Author: ASF GitHub Bot Created on: 27/Jan/20 04:37 Start Date: 27/Jan/20 04:37 Worklog Time Spent: 10m Work Description: jcamachor commented on pull request #877: HIVE-22538: RS deduplication does not always enforce hive.optimize.reducededuplication.min.reducer URL: https://github.com/apache/hive/pull/877#discussion_r371060669 ## File path: ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java ## @@ -579,6 +579,7 @@ public void close(boolean abort) throws IOException { OrcFile.WriterOptions wo = OrcFile.writerOptions(this.options.getConfiguration()) .inspector(rowInspector) + .setSchema(createEventSchemaFromTableProperties(this.options.getTableProperties())) Review comment: Why is this needed after this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377471) Time Spent: 0.5h (was: 20m) > RS deduplication does not always enforce > hive.optimize.reducededuplication.min.reducer > -- > > Key: HIVE-22538 > URL: https://issues.apache.org/jira/browse/HIVE-22538 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22538.2.patch, HIVE-22538.3.patch, > HIVE-22538.4.patch, HIVE-22538.5.patch, HIVE-22538.6.patch, > HIVE-22538.6.patch, HIVE-22538.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > For transactional tables, that property might be overriden to 1, which can > lead to merging final aggregation into a single stage (hence leading to > performance degradation). For instance, when autogather column stats is > enabled, this can happen for the following query: > {code} > set hive.support.concurrency=true; > set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; > EXPLAIN > CREATE TABLE x STORED AS ORC TBLPROPERTIES('transactional'='true') AS > SELECT * FROM SRC x CLUSTER BY x.key; > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22753) Fix gradual mem leak: Operationlog related appenders should be cleared up on errors
[ https://issues.apache.org/jira/browse/HIVE-22753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024053#comment-17024053 ] Hive QA commented on HIVE-22753: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12991863/HIVE-22753.3.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 17946 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/20348/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20348/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20348/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12991863 - PreCommit-HIVE-Build > Fix gradual mem leak: Operationlog related appenders should be cleared up on > errors > > > Key: HIVE-22753 > URL: https://issues.apache.org/jira/browse/HIVE-22753 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-22753.1.patch, HIVE-22753.2.patch, > HIVE-22753.3.patch, image-2020-01-21-11-14-37-911.png, > image-2020-01-21-11-17-59-279.png, image-2020-01-21-11-18-37-294.png > > > In case of exception in SQLOperation, operational log does not get cleared > up. This causes gradual build up of HushableRandomAccessFileAppender causing > HS2 to OOM after some time. > !image-2020-01-21-11-14-37-911.png|width=431,height=267! > > Allocation tree > !image-2020-01-21-11-18-37-294.png|width=425,height=178! > > Prod instance mem > !image-2020-01-21-11-17-59-279.png|width=698,height=209! > > Each HushableRandomAccessFileAppender holds internal ref to > RandomAccessFileAppender which holds a 256 KB bytebuffer, causing the mem > leak. > Related ticket: HIVE-18820 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22753) Fix gradual mem leak: Operationlog related appenders should be cleared up on errors
[ https://issues.apache.org/jira/browse/HIVE-22753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024048#comment-17024048 ] Hive QA commented on HIVE-22753: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 12s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 2s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 41s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 43s{color} | {color:blue} ql in master has 1533 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 50s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 23m 8s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-20348/dev-support/hive-personality.sh | | git revision | master / 037eace | | Default Java | 1.8.0_111 | | findbugs | v3.0.1 | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-20348/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Fix gradual mem leak: Operationlog related appenders should be cleared up on > errors > > > Key: HIVE-22753 > URL: https://issues.apache.org/jira/browse/HIVE-22753 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-22753.1.patch, HIVE-22753.2.patch, > HIVE-22753.3.patch, image-2020-01-21-11-14-37-911.png, > image-2020-01-21-11-17-59-279.png, image-2020-01-21-11-18-37-294.png > > > In case of exception in SQLOperation, operational log does not get cleared > up. This causes gradual build up of HushableRandomAccessFileAppender causing > HS2 to OOM after some time. > !image-2020-01-21-11-14-37-911.png|width=431,height=267! > > Allocation tree > !image-2020-01-21-11-18-37-294.png|width=425,height=178! > > Prod instance mem > !image-2020-01-21-11-17-59-279.png|width=698,height=209! > > Each HushableRandomAccessFileAppender holds internal ref to > RandomAccessFileAppender which holds a 256 KB bytebuffer, causing the mem > leak. > Related ticket: HIVE-18820 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22753) Fix gradual mem leak: Operationlog related appenders should be cleared up on errors
[ https://issues.apache.org/jira/browse/HIVE-22753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024039#comment-17024039 ] Rajesh Balamohan commented on HIVE-22753: - [~ashutoshc]: Tried IdlePurgePolicy in LogDivertAppender, but it did not add HushableRandomAccessFileAppender to its tracking set and didn't clear the instances. Attached the revised patch with 1s interval for timeout. > Fix gradual mem leak: Operationlog related appenders should be cleared up on > errors > > > Key: HIVE-22753 > URL: https://issues.apache.org/jira/browse/HIVE-22753 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-22753.1.patch, HIVE-22753.2.patch, > HIVE-22753.3.patch, image-2020-01-21-11-14-37-911.png, > image-2020-01-21-11-17-59-279.png, image-2020-01-21-11-18-37-294.png > > > In case of exception in SQLOperation, operational log does not get cleared > up. This causes gradual build up of HushableRandomAccessFileAppender causing > HS2 to OOM after some time. > !image-2020-01-21-11-14-37-911.png|width=431,height=267! > > Allocation tree > !image-2020-01-21-11-18-37-294.png|width=425,height=178! > > Prod instance mem > !image-2020-01-21-11-17-59-279.png|width=698,height=209! > > Each HushableRandomAccessFileAppender holds internal ref to > RandomAccessFileAppender which holds a 256 KB bytebuffer, causing the mem > leak. > Related ticket: HIVE-18820 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22753) Fix gradual mem leak: Operationlog related appenders should be cleared up on errors
[ https://issues.apache.org/jira/browse/HIVE-22753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-22753: Attachment: HIVE-22753.3.patch > Fix gradual mem leak: Operationlog related appenders should be cleared up on > errors > > > Key: HIVE-22753 > URL: https://issues.apache.org/jira/browse/HIVE-22753 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-22753.1.patch, HIVE-22753.2.patch, > HIVE-22753.3.patch, image-2020-01-21-11-14-37-911.png, > image-2020-01-21-11-17-59-279.png, image-2020-01-21-11-18-37-294.png > > > In case of exception in SQLOperation, operational log does not get cleared > up. This causes gradual build up of HushableRandomAccessFileAppender causing > HS2 to OOM after some time. > !image-2020-01-21-11-14-37-911.png|width=431,height=267! > > Allocation tree > !image-2020-01-21-11-18-37-294.png|width=425,height=178! > > Prod instance mem > !image-2020-01-21-11-17-59-279.png|width=698,height=209! > > Each HushableRandomAccessFileAppender holds internal ref to > RandomAccessFileAppender which holds a 256 KB bytebuffer, causing the mem > leak. > Related ticket: HIVE-18820 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17023898#comment-17023898 ] Hive QA commented on HIVE-22736: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12991856/HIVE-22736.patch {color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17959 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges (batchId=280) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/20347/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20347/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20347/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12991856 - PreCommit-HIVE-Build > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17023891#comment-17023891 ] Hive QA commented on HIVE-22736: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 10s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 28s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 11s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 55s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 16s{color} | {color:blue} shims/common in master has 6 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 19s{color} | {color:blue} shims/0.23 in master has 7 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m 14s{color} | {color:blue} standalone-metastore/metastore-common in master has 37 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 30s{color} | {color:blue} common in master has 63 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m 8s{color} | {color:blue} standalone-metastore/metastore-server in master has 181 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 39s{color} | {color:blue} ql in master has 1533 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 37s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 0s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 8s{color} | {color:green} shims/common: The patch generated 0 new + 93 unchanged - 1 fixed = 93 total (was 94) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 9s{color} | {color:green} The patch 0.23 passed checkstyle {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 11s{color} | {color:red} standalone-metastore/metastore-common: The patch generated 2 new + 98 unchanged - 0 fixed = 100 total (was 98) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 13s{color} | {color:green} The patch common passed checkstyle {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 19s{color} | {color:green} The patch metastore-server passed checkstyle {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 37s{color} | {color:green} The patch ql passed checkstyle {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 18s{color} | {color:green} The patch hive-unit passed checkstyle {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 57s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch
[jira] [Updated] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22736: --- Attachment: HIVE-22736.patch Status: Patch Available (was: In Progress) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22736: --- Status: In Progress (was: Patch Available) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17023855#comment-17023855 ] Hive QA commented on HIVE-22736: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12991850/HIVE-22736.patch {color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 17959 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/20346/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20346/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20346/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12991850 - PreCommit-HIVE-Build > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17023842#comment-17023842 ] Hive QA commented on HIVE-22736: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 13s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 21s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 8s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 1s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 16s{color} | {color:blue} shims/common in master has 6 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 20s{color} | {color:blue} shims/0.23 in master has 7 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m 16s{color} | {color:blue} standalone-metastore/metastore-common in master has 37 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 30s{color} | {color:blue} common in master has 63 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m 9s{color} | {color:blue} standalone-metastore/metastore-server in master has 181 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 43s{color} | {color:blue} ql in master has 1533 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 39s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 3s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 10s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 9s{color} | {color:red} shims/common: The patch generated 1 new + 93 unchanged - 1 fixed = 94 total (was 94) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 11s{color} | {color:red} standalone-metastore/metastore-common: The patch generated 4 new + 98 unchanged - 0 fixed = 102 total (was 98) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 18s{color} | {color:red} itests/hive-unit: The patch generated 2 new + 213 unchanged - 0 fixed = 215 total (was 213) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 2s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 49m 9s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile xml | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality |
[jira] [Updated] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22736: --- Attachment: HIVE-22736.patch Status: Patch Available (was: In Progress) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22736: --- Status: In Progress (was: Patch Available) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22736: --- Attachment: HIVE-22736.patch Status: Patch Available (was: In Progress) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22736: --- Status: In Progress (was: Patch Available) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)