date:20190626



 [ 
https://issues.apache.org/jira/browse/HIVE-21927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-21927 started by Rajkumar Singh.
-
> HiveServer Web UI: Setting the HttpOnly option in the cookies
> -
>
> Key: HIVE-21927
> URL: https://issues.apache.org/jira/browse/HIVE-21927
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 3.1.1
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
>
> Intend of this JIRA is to introduce the HttpOnly option in the cookie.
> cookie: before change
> {code:java}
> hdp32bFALSE   /   FALSE   0   JSESSIONID  
> 8dkibwayfnrc4y4hvpu3vh74
> {code}
> after change:
> {code:java}
> #HttpOnly_hdp32b  FALSE   /   FALSE   0   JSESSIONID  
> e1npdkbo3inj1xnd6gdc6ihws
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-21927) HiveServer Web UI: Setting the HttpOnly option in the cookies



 [ 
https://issues.apache.org/jira/browse/HIVE-21927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajkumar Singh reassigned HIVE-21927:
-


> HiveServer Web UI: Setting the HttpOnly option in the cookies
> -
>
> Key: HIVE-21927
> URL: https://issues.apache.org/jira/browse/HIVE-21927
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 3.1.1
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
>
> Intend of this JIRA is to introduce the HttpOnly option in the cookie.
> cookie: before change
> {code:java}
> hdp32bFALSE   /   FALSE   0   JSESSIONID  
> 8dkibwayfnrc4y4hvpu3vh74
> {code}
> after change:
> {code:java}
> #HttpOnly_hdp32b  FALSE   /   FALSE   0   JSESSIONID  
> e1npdkbo3inj1xnd6gdc6ihws
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy



 [ 
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268262=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268262
 ]

ASF GitHub Bot logged work on HIVE-21886:
-

Author: ASF GitHub Bot
Created on: 27/Jun/19 05:52
Start Date: 27/Jun/19 05:52
Worklog Time Spent: 10m 
  Work Description: maheshk114 commented on pull request #688: HIVE-21886 : 
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r298015828
 
 

 ##
 File path: 
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
 ##
 @@ -92,21 +93,49 @@ private Scenario 
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
 }
   }
 
-  // return true, if event needs to be dumped, else return false.
-  private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+  // Return true, if event needs to be dumped, else return false.
+  private boolean handleRenameForReplacePolicy(Context withinContext, String 
oldName, String newName) {
+// If the table is renamed after being added to the list of tables to be 
bootstrapped, then remove it from the
 
 Review comment:
   with the addition if withinContext.oldReplScope != null ..before calling 
..the assert is not required
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 268262)
Time Spent: 11h  (was: 10h 50m)

> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch, 
> HIVE-21886.03.patch
>
>  Time Spent: 11h
>  Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace 
> policy is getting executed, it needs to take care of the policy inclusion in 
> both the policy for each table name.
>  1. Create a list of tables to be bootstrapped. 
>   2. During handling of alter table, if the alter type is rename 
>       1. If the old table name is present in the list of table to be 
> bootstrapped, remove it.
>        2. If the new table name, matches the new policy, add it to the list 
> of tables to be bootstrapped.
>        3. If the old table does not match the old policy drop it, even if the 
> table is not present at target.
>   3. During handling of drop table
>        1. if the table is in the list of tables to be bootstrapped, then 
> remove it and ignore the event.
>   4. During other event handling 
>        1. if the table is there in the list of tables to be bootstrapped, 
> then ignore the event.
>        2. If the new policy does not match the table name, then ignore the 
> event.
>  
> Rename handling during replace policy
>  # Old name not matching old policy – The old table will not be there at the 
> target cluster. The table will not be returned by get-all-table.
>  ## Old name is not matching new policy
>  ### New name not matching old policy
>   New name not matching new policy
>  * Ignore the event, no need to do anything.
>   New name matching new policy
>  * The table will be returned by get-all-table. Replace policy handler 
> will bootstrap this table as its matching new policy and not matching old 
> policy.
>  * All the future events will be ignored as part of check added by 
> replace policy handling.
>  * All the event with old table name will anyways be ignored as the old 
> name is not matching the new policy.
>  ### New name matching old policy
>   New name not matching new policy
>  * As the new name is not matching the new policy, the table need not be 
> replicated.
>  * As the old name is not matching the new policy, the rename events will 
> be ignored.
>  * So nothing to be done for this scenario.
>   New name matching new policy
>  * As the new name is matching both old and new policy, replace handler 
> will not bootstrap the table.
>  * Add the table to the list of tables to be bootstrapped.
>  * Ignore all the events with new name.
>  * If there is a drop

[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy



 [ 
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268259=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268259
 ]

ASF GitHub Bot logged work on HIVE-21886:
-

Author: ASF GitHub Bot
Created on: 27/Jun/19 05:49
Start Date: 27/Jun/19 05:49
Worklog Time Spent: 10m 
  Work Description: maheshk114 commented on pull request #688: HIVE-21886 : 
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r298015157
 
 

 ##
 File path: 
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
 ##
 @@ -168,10 +205,19 @@ public void handle(Context withinContext) throws 
Exception {
   }
 }
 
-// If the tables are filtered based on name, then needs to handle the 
rename scenarios.
-if (!withinContext.replScope.includeAllTables()) {
-  if (!handleForTableLevelReplication(withinContext)) {
-LOG.info("Alter event for table " + before.getTableName() + " is 
skipped from dumping");
+// Handle renames for table level replication.
+if ((Scenario.RENAME == scenario)
+&& (!withinContext.replScope.includeAllTables() || 
withinContext.oldReplScope != null)) {
+  String oldName = before.getTableName();
+  String newName = after.getTableName();
+  boolean needDump;
+  if (withinContext.oldReplScope == null) {
+needDump = handleRenameForTableLevelReplication(withinContext, 
oldName, newName);
+  } else {
+needDump = handleRenameForReplacePolicy(withinContext, oldName, 
newName);
+  }
+  if (!needDump) {
+LOG.info("Rename event for table " + oldName + " is skipped from 
dumping");
 
 Review comment:
   done
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 268259)
Time Spent: 10h 50m  (was: 10h 40m)

> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch, 
> HIVE-21886.03.patch
>
>  Time Spent: 10h 50m
>  Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace 
> policy is getting executed, it needs to take care of the policy inclusion in 
> both the policy for each table name.
>  1. Create a list of tables to be bootstrapped. 
>   2. During handling of alter table, if the alter type is rename 
>       1. If the old table name is present in the list of table to be 
> bootstrapped, remove it.
>        2. If the new table name, matches the new policy, add it to the list 
> of tables to be bootstrapped.
>        3. If the old table does not match the old policy drop it, even if the 
> table is not present at target.
>   3. During handling of drop table
>        1. if the table is in the list of tables to be bootstrapped, then 
> remove it and ignore the event.
>   4. During other event handling 
>        1. if the table is there in the list of tables to be bootstrapped, 
> then ignore the event.
>        2. If the new policy does not match the table name, then ignore the 
> event.
>  
> Rename handling during replace policy
>  # Old name not matching old policy – The old table will not be there at the 
> target cluster. The table will not be returned by get-all-table.
>  ## Old name is not matching new policy
>  ### New name not matching old policy
>   New name not matching new policy
>  * Ignore the event, no need to do anything.
>   New name matching new policy
>  * The table will be returned by get-all-table. Replace policy handler 
> will bootstrap this table as its matching new policy and not matching old 
> policy.
>  * All the future events will be ignored as part of check added by 
> replace policy handling.
>  * All the event with old table name will anyways be ignored as the old 
> name is not matching the new policy.
>  ### New name matching old policy
>   New name not matching new policy
>  * As the new name is not matching the new policy, the table need not be 
> replicated.
>  * As the old name is not matching the new policy, the rename events

[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy



 [ 
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268258=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268258
 ]

ASF GitHub Bot logged work on HIVE-21886:
-

Author: ASF GitHub Bot
Created on: 27/Jun/19 05:49
Start Date: 27/Jun/19 05:49
Worklog Time Spent: 10m 
  Work Description: maheshk114 commented on pull request #688: HIVE-21886 : 
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r298015144
 
 

 ##
 File path: 
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
 ##
 @@ -168,10 +205,19 @@ public void handle(Context withinContext) throws 
Exception {
   }
 }
 
-// If the tables are filtered based on name, then needs to handle the 
rename scenarios.
-if (!withinContext.replScope.includeAllTables()) {
-  if (!handleForTableLevelReplication(withinContext)) {
-LOG.info("Alter event for table " + before.getTableName() + " is 
skipped from dumping");
+// Handle renames for table level replication.
+if ((Scenario.RENAME == scenario)
+&& (!withinContext.replScope.includeAllTables() || 
withinContext.oldReplScope != null)) {
 
 Review comment:
   done
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 268258)
Time Spent: 10h 40m  (was: 10.5h)

> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch, 
> HIVE-21886.03.patch
>
>  Time Spent: 10h 40m
>  Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace 
> policy is getting executed, it needs to take care of the policy inclusion in 
> both the policy for each table name.
>  1. Create a list of tables to be bootstrapped. 
>   2. During handling of alter table, if the alter type is rename 
>       1. If the old table name is present in the list of table to be 
> bootstrapped, remove it.
>        2. If the new table name, matches the new policy, add it to the list 
> of tables to be bootstrapped.
>        3. If the old table does not match the old policy drop it, even if the 
> table is not present at target.
>   3. During handling of drop table
>        1. if the table is in the list of tables to be bootstrapped, then 
> remove it and ignore the event.
>   4. During other event handling 
>        1. if the table is there in the list of tables to be bootstrapped, 
> then ignore the event.
>        2. If the new policy does not match the table name, then ignore the 
> event.
>  
> Rename handling during replace policy
>  # Old name not matching old policy – The old table will not be there at the 
> target cluster. The table will not be returned by get-all-table.
>  ## Old name is not matching new policy
>  ### New name not matching old policy
>   New name not matching new policy
>  * Ignore the event, no need to do anything.
>   New name matching new policy
>  * The table will be returned by get-all-table. Replace policy handler 
> will bootstrap this table as its matching new policy and not matching old 
> policy.
>  * All the future events will be ignored as part of check added by 
> replace policy handling.
>  * All the event with old table name will anyways be ignored as the old 
> name is not matching the new policy.
>  ### New name matching old policy
>   New name not matching new policy
>  * As the new name is not matching the new policy, the table need not be 
> replicated.
>  * As the old name is not matching the new policy, the rename events will 
> be ignored.
>  * So nothing to be done for this scenario.
>   New name matching new policy
>  * As the new name is matching both old and new policy, replace handler 
> will not bootstrap the table.
>  * Add the table to the list of tables to be bootstrapped.
>  * Ignore all the events with new name.
>  * If there is a drop event for the table (with new name), then remove 
> the table from the the list of table to be bootstrapped.
>  * In

[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy



 [ 
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268251=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268251
 ]

ASF GitHub Bot logged work on HIVE-21886:
-

Author: ASF GitHub Bot
Created on: 27/Jun/19 05:39
Start Date: 27/Jun/19 05:39
Worklog Time Spent: 10m 
  Work Description: sankarh commented on pull request #688: HIVE-21886 : 
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r298010710
 
 

 ##
 File path: 
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
 ##
 @@ -168,10 +205,19 @@ public void handle(Context withinContext) throws 
Exception {
   }
 }
 
-// If the tables are filtered based on name, then needs to handle the 
rename scenarios.
-if (!withinContext.replScope.includeAllTables()) {
-  if (!handleForTableLevelReplication(withinContext)) {
-LOG.info("Alter event for table " + before.getTableName() + " is 
skipped from dumping");
+// Handle renames for table level replication.
+if ((Scenario.RENAME == scenario)
+&& (!withinContext.replScope.includeAllTables() || 
withinContext.oldReplScope != null)) {
+  String oldName = before.getTableName();
+  String newName = after.getTableName();
+  boolean needDump;
+  if (withinContext.oldReplScope == null) {
+needDump = handleRenameForTableLevelReplication(withinContext, 
oldName, newName);
+  } else {
+needDump = handleRenameForReplacePolicy(withinContext, oldName, 
newName);
+  }
+  if (!needDump) {
+LOG.info("Rename event for table " + oldName + " is skipped from 
dumping");
 
 Review comment:
   Shall change the log to include "newName" as well. ... oldName to newName ...
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 268251)
Time Spent: 10.5h  (was: 10h 20m)

> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch, 
> HIVE-21886.03.patch
>
>  Time Spent: 10.5h
>  Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace 
> policy is getting executed, it needs to take care of the policy inclusion in 
> both the policy for each table name.
>  1. Create a list of tables to be bootstrapped. 
>   2. During handling of alter table, if the alter type is rename 
>       1. If the old table name is present in the list of table to be 
> bootstrapped, remove it.
>        2. If the new table name, matches the new policy, add it to the list 
> of tables to be bootstrapped.
>        3. If the old table does not match the old policy drop it, even if the 
> table is not present at target.
>   3. During handling of drop table
>        1. if the table is in the list of tables to be bootstrapped, then 
> remove it and ignore the event.
>   4. During other event handling 
>        1. if the table is there in the list of tables to be bootstrapped, 
> then ignore the event.
>        2. If the new policy does not match the table name, then ignore the 
> event.
>  
> Rename handling during replace policy
>  # Old name not matching old policy – The old table will not be there at the 
> target cluster. The table will not be returned by get-all-table.
>  ## Old name is not matching new policy
>  ### New name not matching old policy
>   New name not matching new policy
>  * Ignore the event, no need to do anything.
>   New name matching new policy
>  * The table will be returned by get-all-table. Replace policy handler 
> will bootstrap this table as its matching new policy and not matching old 
> policy.
>  * All the future events will be ignored as part of check added by 
> replace policy handling.
>  * All the event with old table name will anyways be ignored as the old 
> name is not matching the new policy.
>  ### New name matching old policy
>   New name not matching new policy
>  * As the new name is not matching the new policy, the table need not be 
> replicated.
>  *

[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy



 [ 
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268252=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268252
 ]

ASF GitHub Bot logged work on HIVE-21886:
-

Author: ASF GitHub Bot
Created on: 27/Jun/19 05:39
Start Date: 27/Jun/19 05:39
Worklog Time Spent: 10m 
  Work Description: sankarh commented on pull request #688: HIVE-21886 : 
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r298010206
 
 

 ##
 File path: 
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
 ##
 @@ -168,10 +205,19 @@ public void handle(Context withinContext) throws 
Exception {
   }
 }
 
-// If the tables are filtered based on name, then needs to handle the 
rename scenarios.
-if (!withinContext.replScope.includeAllTables()) {
-  if (!handleForTableLevelReplication(withinContext)) {
-LOG.info("Alter event for table " + before.getTableName() + " is 
skipped from dumping");
+// Handle renames for table level replication.
+if ((Scenario.RENAME == scenario)
+&& (!withinContext.replScope.includeAllTables() || 
withinContext.oldReplScope != null)) {
 
 Review comment:
   These checks can be moved inside as follows. Will be less complex.
   if (Scenario.RENAME == scenario) {
   ...
   boolean needDump = true;
   if (withinContext.oldReplScope != null) {
   needDump = handleRenameForReplacePolicy();
   } else if (!withinContext.replScope.includeAllTables()) {
   needDump = handleRenameForTableLevelReplication();
   }
   if (!needDump) {
   return;
   } 
   }
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 268252)

> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch, 
> HIVE-21886.03.patch
>
>  Time Spent: 10.5h
>  Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace 
> policy is getting executed, it needs to take care of the policy inclusion in 
> both the policy for each table name.
>  1. Create a list of tables to be bootstrapped. 
>   2. During handling of alter table, if the alter type is rename 
>       1. If the old table name is present in the list of table to be 
> bootstrapped, remove it.
>        2. If the new table name, matches the new policy, add it to the list 
> of tables to be bootstrapped.
>        3. If the old table does not match the old policy drop it, even if the 
> table is not present at target.
>   3. During handling of drop table
>        1. if the table is in the list of tables to be bootstrapped, then 
> remove it and ignore the event.
>   4. During other event handling 
>        1. if the table is there in the list of tables to be bootstrapped, 
> then ignore the event.
>        2. If the new policy does not match the table name, then ignore the 
> event.
>  
> Rename handling during replace policy
>  # Old name not matching old policy – The old table will not be there at the 
> target cluster. The table will not be returned by get-all-table.
>  ## Old name is not matching new policy
>  ### New name not matching old policy
>   New name not matching new policy
>  * Ignore the event, no need to do anything.
>   New name matching new policy
>  * The table will be returned by get-all-table. Replace policy handler 
> will bootstrap this table as its matching new policy and not matching old 
> policy.
>  * All the future events will be ignored as part of check added by 
> replace policy handling.
>  * All the event with old table name will anyways be ignored as the old 
> name is not matching the new policy.
>  ### New name matching old policy
>   New name not matching new policy
>  * As the new name is not matching the new policy, the table need not be 
> replicated.
>  * As the old name is not matching the new policy, the rename events will 
> be ignored.
>  * So nothing to be done for this scenario.
>   New

[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy



 [ 
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268250=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268250
 ]

ASF GitHub Bot logged work on HIVE-21886:
-

Author: ASF GitHub Bot
Created on: 27/Jun/19 05:39
Start Date: 27/Jun/19 05:39
Worklog Time Spent: 10m 
  Work Description: sankarh commented on pull request #688: HIVE-21886 : 
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r298012981
 
 

 ##
 File path: 
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
 ##
 @@ -92,21 +93,49 @@ private Scenario 
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
 }
   }
 
-  // return true, if event needs to be dumped, else return false.
-  private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+  // Return true, if event needs to be dumped, else return false.
+  private boolean handleRenameForReplacePolicy(Context withinContext, String 
oldName, String newName) {
+// If the table is renamed after being added to the list of tables to be 
bootstrapped, then remove it from the
 
 Review comment:
   Just add an assert that withinContext.oldReplScope != null.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 268250)
Time Spent: 10h 20m  (was: 10h 10m)

> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch, 
> HIVE-21886.03.patch
>
>  Time Spent: 10h 20m
>  Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace 
> policy is getting executed, it needs to take care of the policy inclusion in 
> both the policy for each table name.
>  1. Create a list of tables to be bootstrapped. 
>   2. During handling of alter table, if the alter type is rename 
>       1. If the old table name is present in the list of table to be 
> bootstrapped, remove it.
>        2. If the new table name, matches the new policy, add it to the list 
> of tables to be bootstrapped.
>        3. If the old table does not match the old policy drop it, even if the 
> table is not present at target.
>   3. During handling of drop table
>        1. if the table is in the list of tables to be bootstrapped, then 
> remove it and ignore the event.
>   4. During other event handling 
>        1. if the table is there in the list of tables to be bootstrapped, 
> then ignore the event.
>        2. If the new policy does not match the table name, then ignore the 
> event.
>  
> Rename handling during replace policy
>  # Old name not matching old policy – The old table will not be there at the 
> target cluster. The table will not be returned by get-all-table.
>  ## Old name is not matching new policy
>  ### New name not matching old policy
>   New name not matching new policy
>  * Ignore the event, no need to do anything.
>   New name matching new policy
>  * The table will be returned by get-all-table. Replace policy handler 
> will bootstrap this table as its matching new policy and not matching old 
> policy.
>  * All the future events will be ignored as part of check added by 
> replace policy handling.
>  * All the event with old table name will anyways be ignored as the old 
> name is not matching the new policy.
>  ### New name matching old policy
>   New name not matching new policy
>  * As the new name is not matching the new policy, the table need not be 
> replicated.
>  * As the old name is not matching the new policy, the rename events will 
> be ignored.
>  * So nothing to be done for this scenario.
>   New name matching new policy
>  * As the new name is matching both old and new policy, replace handler 
> will not bootstrap the table.
>  * Add the table to the list of tables to be bootstrapped.
>  * Ignore all the events with new name.
>  * If there is a drop event for the table (with new name),

[jira] [Updated] (HIVE-21926) REPL - With table list - "TO" and "FROM" clause should not be allowed along with table filter list



 [ 
https://issues.apache.org/jira/browse/HIVE-21926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mahesh kumar behera updated HIVE-21926:
---
Description: If table filter list is provided during dump command, there is 
a chance that tables will be bootstrapped during rename and during policy 
replace. If all existing events are not replayed completely, then it may happen 
that the remaining events in the next replication cycle causes data corruption. 
So all the existing events should be replayed and "TO" and "FROM" should not be 
allowed if table filter list is provided.

> REPL - With table list - "TO" and "FROM" clause should not be allowed along 
> with table filter list
> --
>
> Key: HIVE-21926
> URL: https://issues.apache.org/jira/browse/HIVE-21926
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: DR, Replication, pull-request-available
>
> If table filter list is provided during dump command, there is a chance that 
> tables will be bootstrapped during rename and during policy replace. If all 
> existing events are not replayed completely, then it may happen that the 
> remaining events in the next replication cycle causes data corruption. So all 
> the existing events should be replayed and "TO" and "FROM" should not be 
> allowed if table filter list is provided.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21926) REPL - With table list - "TO" and "FROM" clause should not be allowed along with table filter list



 [ 
https://issues.apache.org/jira/browse/HIVE-21926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mahesh kumar behera updated HIVE-21926:
---
Summary: REPL - With table list - "TO" and "FROM" clause should not be 
allowed along with table filter list  (was: CLONE - REPL - With table list - 
"TO" and "FROM" clause should not be allowed along with table filter list)

> REPL - With table list - "TO" and "FROM" clause should not be allowed along 
> with table filter list
> --
>
> Key: HIVE-21926
> URL: https://issues.apache.org/jira/browse/HIVE-21926
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: DR, Replication, pull-request-available
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21886) REPL - With table list - Handle rename events during replace policy

[
https://issues.apache.org/jira/browse/HIVE-21886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

mahesh kumar behera updated HIVE-21886:
---
Description:
If some rename events are found to be dumped and replayed while replace policy
is getting executed, it needs to take care of the policy inclusion in both the
policy for each table name.

1. Create a list of tables to be bootstrapped.

2. During handling of alter table, if the alter type is rename

1. If the old table name is present in the list of table to be
bootstrapped, remove it.

2. If the new table name, matches the new policy, add it to the list of
tables to be bootstrapped.

3. If the old table does not match the old policy drop it, even if the
table is not present at target.

3. During handling of drop table

1. if the table is in the list of tables to be bootstrapped, then remove
it and ignore the event.

4. During other event handling

1. if the table is there in the list of tables to be bootstrapped, then
ignore the event.

2. If the new policy does not match the table name, then ignore the
event.

Rename handling during replace policy
# Old name not matching old policy – The old table will not be there at the
target cluster. The table will not be returned by get-all-table.
## Old name is not matching new policy
### New name not matching old policy
New name not matching new policy
* Ignore the event, no need to do anything.
New name matching new policy
* The table will be returned by get-all-table. Replace policy handler will
bootstrap this table as its matching new policy and not matching old policy.
* All the future events will be ignored as part of check added by replace
policy handling.
* All the event with old table name will anyways be ignored as the old
name is not matching the new policy.
### New name matching old policy
New name not matching new policy
* As the new name is not matching the new policy, the table need not be
replicated.
* As the old name is not matching the new policy, the rename events will
be ignored.
* So nothing to be done for this scenario.
New name matching new policy
* As the new name is matching both old and new policy, replace handler
will not bootstrap the table.
* Add the table to the list of tables to be bootstrapped.
* Ignore all the events with new name.
* If there is a drop event for the table (with new name), then remove the
table from the the list of table to be bootstrapped.
* In case of rename event (double rename)
** If the new name satisfies the table pattern, then add the new name to
the list of tables to be bootstrapped and remove the old name from the list of
tables to be bootstrapped.
** If the new name does not satisfies then just removed the table name
from the list of tables to be bootstrapped.
## Old name is matching new policy – As per replace policy handler, which
checks based on old table, the table should be bootstrapped and event should be
ignored. But rename handler should decide based on new name.The old table name
will not be returned by get-all-table, so replace handler will not d anything
for the old table.
### New name not matching old policy
New name not matching new policy
* As the old table is not there at target and new name is not matching new
policy. Ignore the event.
* No need to add the table to the list of tables to be bootstrapped.
* All the subsequent events will be ignored as the new name is not
matching the new policy.
New name matching new policy
* As the new name is not matching old policy but matching new policy, the
table will be bootstrapped by replace policy handler. So rename event need not
add this table to list of table to be bootstrapped.
* All the future events will be ignored by replace policy handler.
* For rename event (double rename)
** If there is a rename, the table (with intermittent new name) will not
be present and thus replace handler will not bootstrap the table.
** So if the new name (the latest one) is matching the new policy, then
add it to the list of table to be bootstrapped.
** And If the new name (the latest one) is not matching the new policy,
then just ignore the event as the intermittent new name would not have added
to the list of table to be bootstrapped.
### New name matching old policy
New name not matching new policy
* Dump the event. The table will be dropped by repl load at the target.
New name matching new policy
* Replace handler will not bootstrap this table as the new name is
matching both policies.
* As old name is not matching the old policy, the table will not be there
at target. The rename event should add the new name to the list of table to be
bootstrapped.
* Subsequent

[jira] [Updated] (HIVE-21886) REPL - With table list - Handle rename events during replace policy

[
https://issues.apache.org/jira/browse/HIVE-21886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

1. Create a list of tables to be bootstrapped.

2. During handling of alter table, if the alter type is rename

1. If the old table name is present in the list of table to be
bootstrapped, remove it.

2. If the new table name, matches the new policy, add it to the list of
tables to be bootstrapped.

3. If the old table does not match the old policy drop it, even if the
table is not present at target.

3. During handling of drop table

1. if the table is in the list of tables to be bootstrapped, then remove
it and ignore the event.

4. During other event handling

1. if the table is there in the list of tables to be bootstrapped, then
ignore the event.

[jira] [Updated] (HIVE-21886) REPL - With table list - Handle rename events during replace policy



 [ 
https://issues.apache.org/jira/browse/HIVE-21886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mahesh kumar behera updated HIVE-21886:
---
Status: Patch Available  (was: Open)

> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch, 
> HIVE-21886.03.patch
>
>  Time Spent: 10h 10m
>  Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace 
> policy is getting executed, it needs to take care of the policy inclusion in 
> both the policy for each table name.
>  1. Create a list of tables to be bootstrapped. 
>   2. During handling of alter table, if the alter type is rename 
>       1. If the old table name is present in the list of table to be 
> bootstrapped, remove it.
>        2. If the new table name, matches the new policy, add it to the list 
> of tables to be bootstrapped.
>   3. During handling of drop table
>        1. if the table is in the list of tables to be bootstrapped, then 
> remove it and ignore the event.
>   4. During other event handling 
>        1. if the table is there in the list of tables to be bootstrapped, 
> then ignore the event.
>  
> Rename handling during replace policy
>  # Old name not matching old policy – The old table will not be there at the 
> target cluster. The table will not be returned by get-all-table.
>  ## Old name is not matching new policy
>  ### New name not matching old policy
>   New name not matching new policy
>  * Ignore the event, no need to do anything.
>   New name matching new policy
>  * The table will be returned by get-all-table. Replace policy handler 
> will bootstrap this table as its matching new policy and not matching old 
> policy.
>  * All the future events will be ignored as part of check added by 
> replace policy handling.
>  * All the event with old table name will anyways be ignored as the old 
> name is not matching the new policy.
>  ### New name matching old policy
>   New name not matching new policy
>  * As the new name is not matching the new policy, the table need not be 
> replicated.
>  * As the old name is not matching the new policy, the rename events will 
> be ignored.
>  * So nothing to be done for this scenario.
>   New name matching new policy
>  * As the new name is matching both old and new policy, replace handler 
> will not bootstrap the table.
>  * Add the table to the list of tables to be bootstrapped.
>  * Ignore all the events with new name.
>  * If there is a drop event for the table (with new name), then remove 
> the table from the the list of table to be bootstrapped.
>  * In case of rename event (double rename)
>  ** If the new name satisfies the table pattern, then add the new name to 
> the list of tables to be bootstrapped and remove the old name from the list 
> of tables to be bootstrapped.
>  ** If the new name does not satisfies then just removed the table name 
> from the list of tables to be bootstrapped.
>  ## Old name is matching new policy – As per replace policy handler, which 
> checks based on old table, the table should be bootstrapped and event should 
> be ignored. But rename handler should decide based on new name.The old table 
> name will not be returned by get-all-table, so replace handler will not d 
> anything for the old table.
>  ### New name not matching old policy
>   New name not matching new policy
>  * As the old table is not there at target and new name is not matching 
> new policy. Ignore the event.
>  * No need to add the table to the list of tables to be bootstrapped.
>  * All the subsequent events will be ignored as the new name is not 
> matching the new policy.
>   New name matching new policy
>  * As the new name is not matching old policy but matching new policy, 
> the table will be bootstrapped by replace policy handler. So rename event 
> need not add this table to list of table to be bootstrapped.
>  * All the future events will be ignored by replace policy handler.
>  * For rename event (double rename)
>  ** If there is a rename, the table (with intermittent new name) will not 
> be present and thus replace handler will not bootstrap the table.
>  ** So if the new name (the latest one) is matching the new policy, then 
> add it to the list of table to be bootstrapped.
>  ** And If the new name (the latest one)

[jira] [Updated] (HIVE-21886) REPL - With table list - Handle rename events during replace policy



 [ 
https://issues.apache.org/jira/browse/HIVE-21886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mahesh kumar behera updated HIVE-21886:
---
Attachment: HIVE-21886.03.patch

> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch, 
> HIVE-21886.03.patch
>
>  Time Spent: 10h 10m
>  Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace 
> policy is getting executed, it needs to take care of the policy inclusion in 
> both the policy for each table name.
>  1. Create a list of tables to be bootstrapped. 
>   2. During handling of alter table, if the alter type is rename 
>       1. If the old table name is present in the list of table to be 
> bootstrapped, remove it.
>        2. If the new table name, matches the new policy, add it to the list 
> of tables to be bootstrapped.
>   3. During handling of drop table
>        1. if the table is in the list of tables to be bootstrapped, then 
> remove it and ignore the event.
>   4. During other event handling 
>        1. if the table is there in the list of tables to be bootstrapped, 
> then ignore the event.
>  
> Rename handling during replace policy
>  # Old name not matching old policy – The old table will not be there at the 
> target cluster. The table will not be returned by get-all-table.
>  ## Old name is not matching new policy
>  ### New name not matching old policy
>   New name not matching new policy
>  * Ignore the event, no need to do anything.
>   New name matching new policy
>  * The table will be returned by get-all-table. Replace policy handler 
> will bootstrap this table as its matching new policy and not matching old 
> policy.
>  * All the future events will be ignored as part of check added by 
> replace policy handling.
>  * All the event with old table name will anyways be ignored as the old 
> name is not matching the new policy.
>  ### New name matching old policy
>   New name not matching new policy
>  * As the new name is not matching the new policy, the table need not be 
> replicated.
>  * As the old name is not matching the new policy, the rename events will 
> be ignored.
>  * So nothing to be done for this scenario.
>   New name matching new policy
>  * As the new name is matching both old and new policy, replace handler 
> will not bootstrap the table.
>  * Add the table to the list of tables to be bootstrapped.
>  * Ignore all the events with new name.
>  * If there is a drop event for the table (with new name), then remove 
> the table from the the list of table to be bootstrapped.
>  * In case of rename event (double rename)
>  ** If the new name satisfies the table pattern, then add the new name to 
> the list of tables to be bootstrapped and remove the old name from the list 
> of tables to be bootstrapped.
>  ** If the new name does not satisfies then just removed the table name 
> from the list of tables to be bootstrapped.
>  ## Old name is matching new policy – As per replace policy handler, which 
> checks based on old table, the table should be bootstrapped and event should 
> be ignored. But rename handler should decide based on new name.The old table 
> name will not be returned by get-all-table, so replace handler will not d 
> anything for the old table.
>  ### New name not matching old policy
>   New name not matching new policy
>  * As the old table is not there at target and new name is not matching 
> new policy. Ignore the event.
>  * No need to add the table to the list of tables to be bootstrapped.
>  * All the subsequent events will be ignored as the new name is not 
> matching the new policy.
>   New name matching new policy
>  * As the new name is not matching old policy but matching new policy, 
> the table will be bootstrapped by replace policy handler. So rename event 
> need not add this table to list of table to be bootstrapped.
>  * All the future events will be ignored by replace policy handler.
>  * For rename event (double rename)
>  ** If there is a rename, the table (with intermittent new name) will not 
> be present and thus replace handler will not bootstrap the table.
>  ** So if the new name (the latest one) is matching the new policy, then 
> add it to the list of table to be bootstrapped.
>  ** And If the new name (the latest one)  is

[jira] [Updated] (HIVE-21886) REPL - With table list - Handle rename events during replace policy



 [ 
https://issues.apache.org/jira/browse/HIVE-21886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mahesh kumar behera updated HIVE-21886:
---
Status: Open  (was: Patch Available)

> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
>  Time Spent: 10h 10m
>  Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace 
> policy is getting executed, it needs to take care of the policy inclusion in 
> both the policy for each table name.
>  1. Create a list of tables to be bootstrapped. 
>   2. During handling of alter table, if the alter type is rename 
>       1. If the old table name is present in the list of table to be 
> bootstrapped, remove it.
>        2. If the new table name, matches the new policy, add it to the list 
> of tables to be bootstrapped.
>   3. During handling of drop table
>        1. if the table is in the list of tables to be bootstrapped, then 
> remove it and ignore the event.
>   4. During other event handling 
>        1. if the table is there in the list of tables to be bootstrapped, 
> then ignore the event.
>  
> Rename handling during replace policy
>  # Old name not matching old policy – The old table will not be there at the 
> target cluster. The table will not be returned by get-all-table.
>  ## Old name is not matching new policy
>  ### New name not matching old policy
>   New name not matching new policy
>  * Ignore the event, no need to do anything.
>   New name matching new policy
>  * The table will be returned by get-all-table. Replace policy handler 
> will bootstrap this table as its matching new policy and not matching old 
> policy.
>  * All the future events will be ignored as part of check added by 
> replace policy handling.
>  * All the event with old table name will anyways be ignored as the old 
> name is not matching the new policy.
>  ### New name matching old policy
>   New name not matching new policy
>  * As the new name is not matching the new policy, the table need not be 
> replicated.
>  * As the old name is not matching the new policy, the rename events will 
> be ignored.
>  * So nothing to be done for this scenario.
>   New name matching new policy
>  * As the new name is matching both old and new policy, replace handler 
> will not bootstrap the table.
>  * Add the table to the list of tables to be bootstrapped.
>  * Ignore all the events with new name.
>  * If there is a drop event for the table (with new name), then remove 
> the table from the the list of table to be bootstrapped.
>  * In case of rename event (double rename)
>  ** If the new name satisfies the table pattern, then add the new name to 
> the list of tables to be bootstrapped and remove the old name from the list 
> of tables to be bootstrapped.
>  ** If the new name does not satisfies then just removed the table name 
> from the list of tables to be bootstrapped.
>  ## Old name is matching new policy – As per replace policy handler, which 
> checks based on old table, the table should be bootstrapped and event should 
> be ignored. But rename handler should decide based on new name.The old table 
> name will not be returned by get-all-table, so replace handler will not d 
> anything for the old table.
>  ### New name not matching old policy
>   New name not matching new policy
>  * As the old table is not there at target and new name is not matching 
> new policy. Ignore the event.
>  * No need to add the table to the list of tables to be bootstrapped.
>  * All the subsequent events will be ignored as the new name is not 
> matching the new policy.
>   New name matching new policy
>  * As the new name is not matching old policy but matching new policy, 
> the table will be bootstrapped by replace policy handler. So rename event 
> need not add this table to list of table to be bootstrapped.
>  * All the future events will be ignored by replace policy handler.
>  * For rename event (double rename)
>  ** If there is a rename, the table (with intermittent new name) will not 
> be present and thus replace handler will not bootstrap the table.
>  ** So if the new name (the latest one) is matching the new policy, then 
> add it to the list of table to be bootstrapped.
>  ** And If the new name (the latest one)  is not matching the new

[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy



 [ 
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268234=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268234
 ]

ASF GitHub Bot logged work on HIVE-21886:
-

Author: ASF GitHub Bot
Created on: 27/Jun/19 05:12
Start Date: 27/Jun/19 05:12
Worklog Time Spent: 10m 
  Work Description: maheshk114 commented on pull request #688: HIVE-21886 : 
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r298008351
 
 

 ##
 File path: 
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
 ##
 @@ -92,21 +93,46 @@ private Scenario 
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
 }
   }
 
-  // return true, if event needs to be dumped, else return false.
-  private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+  private boolean isSetForBootstrapByReplaceHandler(Context withinContext, 
String tblName) {
+return (withinContext.oldReplScope != null)
+&& 
!(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))
+&& (ReplUtils.tableIncludedInReplScope(withinContext.replScope, 
tblName));
+  }
+
+  // Return true, if event needs to be dumped, else return false.
+  private boolean handleForTableLevelReplication(Context withinContext, String 
tblName) {
+// For alter, if the table does not satisfy the new policy then ignore the 
event. In case of replace
+// policy, if the table does not satisfy the old policy, then ignore the 
event. As, if the table satisfy the new
+// policy, then the table will be bootstrapped by replace handler anb if 
the table does not satisfy the new policy,
+// then anyways the table should be ignored. The event should be ignored 
if the table is already in the list of
+// tables to be bootstrapped.
+if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope, tblName)) 
{
+  // In case of replace, it will be dropped during load. In normal case 
just ignore the alter event.
+  LOG.debug("Table " + tblName + " does not satisfy the policy");
+  return false;
+} else if 
((withinContext.getTablesForBootstrap().contains(tblName.toLowerCase()))
 
 Review comment:
   done
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 268234)
Time Spent: 10h 10m  (was: 10h)

> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
>  Time Spent: 10h 10m
>  Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace 
> policy is getting executed, it needs to take care of the policy inclusion in 
> both the policy for each table name.
>  1. Create a list of tables to be bootstrapped. 
>   2. During handling of alter table, if the alter type is rename 
>       1. If the old table name is present in the list of table to be 
> bootstrapped, remove it.
>        2. If the new table name, matches the new policy, add it to the list 
> of tables to be bootstrapped.
>   3. During handling of drop table
>        1. if the table is in the list of tables to be bootstrapped, then 
> remove it and ignore the event.
>   4. During other event handling 
>        1. if the table is there in the list of tables to be bootstrapped, 
> then ignore the event.
>  
> Rename handling during replace policy
>  # Old name not matching old policy – The old table will not be there at the 
> target cluster. The table will not be returned by get-all-table.
>  ## Old name is not matching new policy
>  ### New name not matching old policy
>   New name not matching new policy
>  * Ignore the event, no need to do anything.
>   New name matching new policy
>  * The table will be returned by get-all-table. Replace policy handler 
> will bootstrap this table as its matching new policy and not matching old 
> policy.
>  * All the future events will be ignored as part of check added

[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy



 [ 
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268233=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268233
 ]

ASF GitHub Bot logged work on HIVE-21886:
-

Author: ASF GitHub Bot
Created on: 27/Jun/19 05:12
Start Date: 27/Jun/19 05:12
Worklog Time Spent: 10m 
  Work Description: maheshk114 commented on pull request #688: HIVE-21886 : 
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r298008286
 
 

 ##
 File path: 
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
 ##
 @@ -147,16 +197,19 @@ public void handle(Context withinContext) throws 
Exception {
 
 Table qlMdTableBefore = new Table(before);
 Set bootstrapTableList;
+ReplScope oldReplScope;
 if (Scenario.RENAME == scenario) {
   // Handling for table level replication is done in 
handleForTableLevelReplication method.
 
 Review comment:
   code is removed
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 268233)
Time Spent: 10h  (was: 9h 50m)

> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
>  Time Spent: 10h
>  Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace 
> policy is getting executed, it needs to take care of the policy inclusion in 
> both the policy for each table name.
>  1. Create a list of tables to be bootstrapped. 
>   2. During handling of alter table, if the alter type is rename 
>       1. If the old table name is present in the list of table to be 
> bootstrapped, remove it.
>        2. If the new table name, matches the new policy, add it to the list 
> of tables to be bootstrapped.
>   3. During handling of drop table
>        1. if the table is in the list of tables to be bootstrapped, then 
> remove it and ignore the event.
>   4. During other event handling 
>        1. if the table is there in the list of tables to be bootstrapped, 
> then ignore the event.
>  
> Rename handling during replace policy
>  # Old name not matching old policy – The old table will not be there at the 
> target cluster. The table will not be returned by get-all-table.
>  ## Old name is not matching new policy
>  ### New name not matching old policy
>   New name not matching new policy
>  * Ignore the event, no need to do anything.
>   New name matching new policy
>  * The table will be returned by get-all-table. Replace policy handler 
> will bootstrap this table as its matching new policy and not matching old 
> policy.
>  * All the future events will be ignored as part of check added by 
> replace policy handling.
>  * All the event with old table name will anyways be ignored as the old 
> name is not matching the new policy.
>  ### New name matching old policy
>   New name not matching new policy
>  * As the new name is not matching the new policy, the table need not be 
> replicated.
>  * As the old name is not matching the new policy, the rename events will 
> be ignored.
>  * So nothing to be done for this scenario.
>   New name matching new policy
>  * As the new name is matching both old and new policy, replace handler 
> will not bootstrap the table.
>  * Add the table to the list of tables to be bootstrapped.
>  * Ignore all the events with new name.
>  * If there is a drop event for the table (with new name), then remove 
> the table from the the list of table to be bootstrapped.
>  * In case of rename event (double rename)
>  ** If the new name satisfies the table pattern, then add the new name to 
> the list of tables to be bootstrapped and remove the old name from the list 
> of tables to be bootstrapped.
>  ** If the new name does not satisfies then just removed the table name 
> from the list of tables to be bootstrapped.
>  ## Old name is matching new policy – As per replace policy handler, which 
> checks based on old table, the table should be bootstrapped and

[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy



 [ 
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268232=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268232
 ]

ASF GitHub Bot logged work on HIVE-21886:
-

Author: ASF GitHub Bot
Created on: 27/Jun/19 05:11
Start Date: 27/Jun/19 05:11
Worklog Time Spent: 10m 
  Work Description: maheshk114 commented on pull request #688: HIVE-21886 : 
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r298008125
 
 

 ##
 File path: 
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
 ##
 @@ -118,13 +144,30 @@ private boolean handleForTableLevelReplication(Context 
withinContext) {
 
   // If the old table was in the list of tables to be bootstrapped which 
is a multi rename case, the old table
   // is removed from the list of tables to be bootstrapped and new one is 
added.
-  if (oldTableIsPresent) {
+  if (oldTableInBootstrapList) {
+withinContext.addToListOfTablesForBootstrap(newName);
+return false;
+  }
+
+  // All the subsequent events on this table newName are going to be 
skipped as the table is going to be
+  // bootstrapped by replace handler, so the rename is also skipped.
+  if (isSetForBootstrapByReplaceHandler(withinContext, newName)) {
+// This addition is not actually required. But is added just to be in 
safer side.
 withinContext.addToListOfTablesForBootstrap(newName);
+
+// If the old table satisfies the new policy and is not in the list of 
tables to be bootstrapped
+// (as per previous check based on oldTableIsPresent), then drop it.
+if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, 
oldName)) {
+  scenario = Scenario.DROP;
+  LOG.info("Table " + oldName + " will be dropped as the table " + 
newName + " will be bootstrapped.");
+  return true;
+}
+LOG.info("Table " + newName + " is set to be bootstrapped by replace 
policy handler.");
 return false;
   }
 
   // If both old and new table satisfies the filter and old table is 
present at target, then dump the rename event.
-  LOG.info("both old and new table satisfies the filter");
+  LOG.info("Both old " + oldName + " and new table " + newName + " 
satisfies the filter");
 
 Review comment:
   done
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 268232)
Time Spent: 9h 50m  (was: 9h 40m)

> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
>  Time Spent: 9h 50m
>  Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace 
> policy is getting executed, it needs to take care of the policy inclusion in 
> both the policy for each table name.
>  1. Create a list of tables to be bootstrapped. 
>   2. During handling of alter table, if the alter type is rename 
>       1. If the old table name is present in the list of table to be 
> bootstrapped, remove it.
>        2. If the new table name, matches the new policy, add it to the list 
> of tables to be bootstrapped.
>   3. During handling of drop table
>        1. if the table is in the list of tables to be bootstrapped, then 
> remove it and ignore the event.
>   4. During other event handling 
>        1. if the table is there in the list of tables to be bootstrapped, 
> then ignore the event.
>  
> Rename handling during replace policy
>  # Old name not matching old policy – The old table will not be there at the 
> target cluster. The table will not be returned by get-all-table.
>  ## Old name is not matching new policy
>  ### New name not matching old policy
>   New name not matching new policy
>  * Ignore the event, no need to do anything.
>   New name matching new policy
>  * The table will be returned by get-all-table. Replace policy handler 
> will bootstrap this table as its matching new policy and not matching old 
> policy.
>  * All the future

[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy



 [ 
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268231=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268231
 ]

ASF GitHub Bot logged work on HIVE-21886:
-

Author: ASF GitHub Bot
Created on: 27/Jun/19 05:11
Start Date: 27/Jun/19 05:11
Worklog Time Spent: 10m 
  Work Description: maheshk114 commented on pull request #688: HIVE-21886 : 
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r298008102
 
 

 ##
 File path: 
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
 ##
 @@ -135,8 +178,15 @@ private boolean handleForTableLevelReplication(Context 
withinContext) {
 return false;
   }
 
+  // In case of replace policy, even if the old table matches the new 
policy, none of the events including create
+  // table will be replayed as the old table is set of bootstrap by 
replace handler. So rename event can be skipped.
 
 Review comment:
   code is removed
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 268231)
Time Spent: 9h 40m  (was: 9.5h)

> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
>  Time Spent: 9h 40m
>  Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace 
> policy is getting executed, it needs to take care of the policy inclusion in 
> both the policy for each table name.
>  1. Create a list of tables to be bootstrapped. 
>   2. During handling of alter table, if the alter type is rename 
>       1. If the old table name is present in the list of table to be 
> bootstrapped, remove it.
>        2. If the new table name, matches the new policy, add it to the list 
> of tables to be bootstrapped.
>   3. During handling of drop table
>        1. if the table is in the list of tables to be bootstrapped, then 
> remove it and ignore the event.
>   4. During other event handling 
>        1. if the table is there in the list of tables to be bootstrapped, 
> then ignore the event.
>  
> Rename handling during replace policy
>  # Old name not matching old policy – The old table will not be there at the 
> target cluster. The table will not be returned by get-all-table.
>  ## Old name is not matching new policy
>  ### New name not matching old policy
>   New name not matching new policy
>  * Ignore the event, no need to do anything.
>   New name matching new policy
>  * The table will be returned by get-all-table. Replace policy handler 
> will bootstrap this table as its matching new policy and not matching old 
> policy.
>  * All the future events will be ignored as part of check added by 
> replace policy handling.
>  * All the event with old table name will anyways be ignored as the old 
> name is not matching the new policy.
>  ### New name matching old policy
>   New name not matching new policy
>  * As the new name is not matching the new policy, the table need not be 
> replicated.
>  * As the old name is not matching the new policy, the rename events will 
> be ignored.
>  * So nothing to be done for this scenario.
>   New name matching new policy
>  * As the new name is matching both old and new policy, replace handler 
> will not bootstrap the table.
>  * Add the table to the list of tables to be bootstrapped.
>  * Ignore all the events with new name.
>  * If there is a drop event for the table (with new name), then remove 
> the table from the the list of table to be bootstrapped.
>  * In case of rename event (double rename)
>  ** If the new name satisfies the table pattern, then add the new name to 
> the list of tables to be bootstrapped and remove the old name from the list 
> of tables to be bootstrapped.
>  ** If the new name does not satisfies then just removed the table name 
> from the list of tables to be bootstrapped.
>  ## Old name is matching new policy – As per replace policy handler, which 
> checks based on old

[jira] [Assigned] (HIVE-21926) CLONE - REPL - With table list - "TO" and "FROM" clause should not be allowed along with table filter list



 [ 
https://issues.apache.org/jira/browse/HIVE-21926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mahesh kumar behera reassigned HIVE-21926:
--


> CLONE - REPL - With table list - "TO" and "FROM" clause should not be allowed 
> along with table filter list
> --
>
> Key: HIVE-21926
> URL: https://issues.apache.org/jira/browse/HIVE-21926
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: DR, Replication, pull-request-available
>
> If some rename events are found to be dumped and replayed while replace 
> policy is getting executed, it needs to take care of the policy inclusion in 
> both the policy for each table name.
>  1. Create a list of tables to be bootstrapped. 
>   2. During handling of alter table, if the alter type is rename 
>       1. If the old table name is present in the list of table to be 
> bootstrapped, remove it.
>        2. If the new table name, matches the new policy, add it to the list 
> of tables to be bootstrapped.
>   3. During handling of drop table
>        1. if the table is in the list of tables to be bootstrapped, then 
> remove it and ignore the event.
>   4. During other event handling 
>        1. if the table is there in the list of tables to be bootstrapped, 
> then ignore the event.
>  
> Rename handling during replace policy
>  # Old name not matching old policy – The old table will not be there at the 
> target cluster. The table will not be returned by get-all-table.
>  ## Old name is not matching new policy
>  ### New name not matching old policy
>   New name not matching new policy
>  * Ignore the event, no need to do anything.
>   New name matching new policy
>  * The table will be returned by get-all-table. Replace policy handler 
> will bootstrap this table as its matching new policy and not matching old 
> policy.
>  * All the future events will be ignored as part of check added by 
> replace policy handling.
>  * All the event with old table name will anyways be ignored as the old 
> name is not matching the new policy.
>  ### New name matching old policy
>   New name not matching new policy
>  * As the new name is not matching the new policy, the table need not be 
> replicated.
>  * As the old name is not matching the new policy, the rename events will 
> be ignored.
>  * So nothing to be done for this scenario.
>   New name matching new policy
>  * As the new name is matching both old and new policy, replace handler 
> will not bootstrap the table.
>  * Add the table to the list of tables to be bootstrapped.
>  * Ignore all the events with new name.
>  * If there is a drop event for the table (with new name), then remove 
> the table from the the list of table to be bootstrapped.
>  * In case of rename event (double rename)
>  ** If the new name satisfies the table pattern, then add the new name to 
> the list of tables to be bootstrapped and remove the old name from the list 
> of tables to be bootstrapped.
>  ** If the new name does not satisfies then just removed the table name 
> from the list of tables to be bootstrapped.
>  ## Old name is matching new policy – As per replace policy handler, which 
> checks based on old table, the table should be bootstrapped and event should 
> be ignored. But rename handler should decide based on new name.The old table 
> name will not be returned by get-all-table, so replace handler will not d 
> anything for the old table.
>  ### New name not matching old policy
>   New name not matching new policy
>  * As the old table is not there at target and new name is not matching 
> new policy. Ignore the event.
>  * No need to add the table to the list of tables to be bootstrapped.
>  * All the subsequent events will be ignored as the new name is not 
> matching the new policy.
>   New name matching new policy
>  * As the new name is not matching old policy but matching new policy, 
> the table will be bootstrapped by replace policy handler. So rename event 
> need not add this table to list of table to be bootstrapped.
>  * All the future events will be ignored by replace policy handler.
>  * For rename event (double rename)
>  ** If there is a rename, the table (with intermittent new name) will not 
> be present and thus replace handler will not bootstrap the table.
>  ** So if the new name (the latest one) is matching the new policy, then 
> add it to the list of table to be bootstrapped.
>  ** And If the new name (the latest one)  is not matching the new policy, 
> then just ignore the event as the  intermittent new name would

[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy



 [ 
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268229=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268229
 ]

ASF GitHub Bot logged work on HIVE-21886:
-

Author: ASF GitHub Bot
Created on: 27/Jun/19 05:10
Start Date: 27/Jun/19 05:10
Worklog Time Spent: 10m 
  Work Description: maheshk114 commented on pull request #688: HIVE-21886 : 
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r298008044
 
 

 ##
 File path: 
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
 ##
 @@ -92,16 +93,40 @@ private Scenario 
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
 }
   }
 
-  // return true, if event needs to be dumped, else return false.
-  private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+  private boolean isSetForBootstrapByReplaceHandler(Context withinContext, 
String tblName) {
+return (withinContext.oldReplScope != null)
+&& 
!(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))
+&& (ReplUtils.tableIncludedInReplScope(withinContext.replScope, 
tblName));
+  }
 
-if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
-  // If the table is renamed after being added to the list of tables to be 
bootstrapped, then remove it from the
-  // list of tables to be bootstrapped.
-  boolean oldTableIsPresent = 
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
+  // Return true, if event needs to be dumped, else return false.
+  private boolean handleForTableLevelReplication(Context withinContext, String 
tblName) {
+// For alter, if the table does not satisfy the new policy then ignore the 
event. In case of replace
+// policy, if the table does not satisfy the old policy, then ignore the 
event. As, if the table satisfy the new
+// policy, then the table will be bootstrapped by replace handler anb if 
the table does not satisfy the new policy,
+// then anyways the table should be ignored.
+if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope, tblName)) 
{
+  // In case of replace, it will be dropped during load. In normal case 
just ignore the alter event.
+  LOG.debug("Table " + tblName + " does not satisfy the policy");
+  return false;
+} else if ((withinContext.oldReplScope != null)
+&& 
(!ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))) {
+  LOG.debug("Table " + tblName + " is set for bootstrap");
+  return false;
+} else {
+  // Table satisfies both old (if its there) and current policy, dump the 
alter event.
+  return true;
+}
+  }
 
+  // Return true, if event needs to be dumped, else return false.
+  private boolean handleForTableLevelReplicationForRename(Context 
withinContext, String oldName, String newName) {
 
 Review comment:
   new Jira is created
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 268229)
Time Spent: 9h 20m  (was: 9h 10m)

> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
>  Time Spent: 9h 20m
>  Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace 
> policy is getting executed, it needs to take care of the policy inclusion in 
> both the policy for each table name.
>  1. Create a list of tables to be bootstrapped. 
>   2. During handling of alter table, if the alter type is rename 
>       1. If the old table name is present in the list of table to be 
> bootstrapped, remove it.
>        2. If the new table name, matches the new policy, add it to the list 
> of tables to be bootstrapped.
>   3. During handling of drop table
>        1. if the table is in the list of tables to be bootstrapped, then 
> remove it and ignore the event.
>   4. During other event handling 
>

[jira] [Updated] (HIVE-21926) CLONE - REPL - With table list - "TO" and "FROM" clause should not be allowed along with table filter list

[
https://issues.apache.org/jira/browse/HIVE-21926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

mahesh kumar behera updated HIVE-21926:
---
Description: (was: If some rename events are found to be dumped and
replayed while replace policy is getting executed, it needs to take care of the
policy inclusion in both the policy for each table name.

1. Create a list of tables to be bootstrapped.

2. During handling of alter table, if the alter type is rename

1. If the old table name is present in the list of table to be
bootstrapped, remove it.

2. If the new table name, matches the new policy, add it to the list of
tables to be bootstrapped.

3. During handling of drop table

1. if the table is in the list of tables to be bootstrapped, then remove
it and ignore the event.

4. During other event handling

1. if the table is there in the list of tables to be bootstrapped, then
ignore the event.

[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy



 [ 
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268230=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268230
 ]

ASF GitHub Bot logged work on HIVE-21886:
-

Author: ASF GitHub Bot
Created on: 27/Jun/19 05:10
Start Date: 27/Jun/19 05:10
Worklog Time Spent: 10m 
  Work Description: maheshk114 commented on pull request #688: HIVE-21886 : 
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r298008064
 
 

 ##
 File path: 
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestTableLevelReplicationScenarios.java
 ##
 @@ -196,27 +196,27 @@ private void verifyBootstrapDirInIncrementalDump(String 
dumpLocation, String[] b
 
   @Test
   public void testBasicBootstrapWithIncludeList() throws Throwable {
-String[] originalNonAcidTables = new String[] {"t1", "t2" };
-String[] originalFullAcidTables = new String[] {"t3", "t4" };
-String[] originalMMAcidTables = new String[] {"t5" };
+String[] originalNonAcidTables = new String[] {"t1", "t2"};
+String[] originalFullAcidTables = new String[] {"t3", "t4"};
+String[] originalMMAcidTables = new String[] {"t5"};
 createTables(originalNonAcidTables, CreateTableType.NON_ACID);
 createTables(originalFullAcidTables, CreateTableType.FULL_ACID);
 createTables(originalMMAcidTables, CreateTableType.MM_ACID);
 
 // Replicate and verify if only 2 tables are replicated to target.
 String replPolicy = primaryDbName + ".['t1', 't4', 't5']";
-String[] replicatedTables = new String[] {"t1", "t4", "t5" };
+String[] replicatedTables = new String[] {"t1", "t4", "t5"};
 replicateAndVerify(replPolicy, null, null, null, replicatedTables);
   }
 
   @Test
   public void testBasicBootstrapWithIncludeAndExcludeList() throws Throwable {
-String[] originalTables = new String[] {"t1", "t11", "t2", "t3", "t100" };
+String[] originalTables = new String[] {"t1", "t11", "t2", "t3", "t100"};
 
 Review comment:
   done
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 268230)
Time Spent: 9.5h  (was: 9h 20m)

> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
>  Time Spent: 9.5h
>  Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace 
> policy is getting executed, it needs to take care of the policy inclusion in 
> both the policy for each table name.
>  1. Create a list of tables to be bootstrapped. 
>   2. During handling of alter table, if the alter type is rename 
>       1. If the old table name is present in the list of table to be 
> bootstrapped, remove it.
>        2. If the new table name, matches the new policy, add it to the list 
> of tables to be bootstrapped.
>   3. During handling of drop table
>        1. if the table is in the list of tables to be bootstrapped, then 
> remove it and ignore the event.
>   4. During other event handling 
>        1. if the table is there in the list of tables to be bootstrapped, 
> then ignore the event.
>  
> Rename handling during replace policy
>  # Old name not matching old policy – The old table will not be there at the 
> target cluster. The table will not be returned by get-all-table.
>  ## Old name is not matching new policy
>  ### New name not matching old policy
>   New name not matching new policy
>  * Ignore the event, no need to do anything.
>   New name matching new policy
>  * The table will be returned by get-all-table. Replace policy handler 
> will bootstrap this table as its matching new policy and not matching old 
> policy.
>  * All the future events will be ignored as part of check added by 
> replace policy handling.
>  * All the event with old table name will anyways be ignored as the old 
> name is not matching the new policy.
>  ### New name matching old policy
>   New name not matching new policy
>  * As the new name is not matching the new policy, the table need not be 
> replicated.
>

[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy



 [ 
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268226=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268226
 ]

ASF GitHub Bot logged work on HIVE-21886:
-

Author: ASF GitHub Bot
Created on: 27/Jun/19 05:08
Start Date: 27/Jun/19 05:08
Worklog Time Spent: 10m 
  Work Description: maheshk114 commented on pull request #688: HIVE-21886 : 
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r298007695
 
 

 ##
 File path: 
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
 ##
 @@ -123,8 +148,21 @@ private boolean handleForTableLevelReplication(Context 
withinContext) {
 return false;
   }
 
+  // All the subsequent events on this table newName are going to be 
skipped, so the rename is also skipped.
 
 Review comment:
   code removed
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 268226)
Time Spent: 9h 10m  (was: 9h)

> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
>  Time Spent: 9h 10m
>  Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace 
> policy is getting executed, it needs to take care of the policy inclusion in 
> both the policy for each table name.
>  1. Create a list of tables to be bootstrapped. 
>   2. During handling of alter table, if the alter type is rename 
>       1. If the old table name is present in the list of table to be 
> bootstrapped, remove it.
>        2. If the new table name, matches the new policy, add it to the list 
> of tables to be bootstrapped.
>   3. During handling of drop table
>        1. if the table is in the list of tables to be bootstrapped, then 
> remove it and ignore the event.
>   4. During other event handling 
>        1. if the table is there in the list of tables to be bootstrapped, 
> then ignore the event.
>  
> Rename handling during replace policy
>  # Old name not matching old policy – The old table will not be there at the 
> target cluster. The table will not be returned by get-all-table.
>  ## Old name is not matching new policy
>  ### New name not matching old policy
>   New name not matching new policy
>  * Ignore the event, no need to do anything.
>   New name matching new policy
>  * The table will be returned by get-all-table. Replace policy handler 
> will bootstrap this table as its matching new policy and not matching old 
> policy.
>  * All the future events will be ignored as part of check added by 
> replace policy handling.
>  * All the event with old table name will anyways be ignored as the old 
> name is not matching the new policy.
>  ### New name matching old policy
>   New name not matching new policy
>  * As the new name is not matching the new policy, the table need not be 
> replicated.
>  * As the old name is not matching the new policy, the rename events will 
> be ignored.
>  * So nothing to be done for this scenario.
>   New name matching new policy
>  * As the new name is matching both old and new policy, replace handler 
> will not bootstrap the table.
>  * Add the table to the list of tables to be bootstrapped.
>  * Ignore all the events with new name.
>  * If there is a drop event for the table (with new name), then remove 
> the table from the the list of table to be bootstrapped.
>  * In case of rename event (double rename)
>  ** If the new name satisfies the table pattern, then add the new name to 
> the list of tables to be bootstrapped and remove the old name from the list 
> of tables to be bootstrapped.
>  ** If the new name does not satisfies then just removed the table name 
> from the list of tables to be bootstrapped.
>  ## Old name is matching new policy – As per replace policy handler, which 
> checks based on old table, the table should be bootstrapped and event should 
> be ignored. But rename handler should decide based on new name.The old

[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy



 [ 
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268225=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268225
 ]

ASF GitHub Bot logged work on HIVE-21886:
-

Author: ASF GitHub Bot
Created on: 27/Jun/19 05:08
Start Date: 27/Jun/19 05:08
Worklog Time Spent: 10m 
  Work Description: maheshk114 commented on pull request #688: HIVE-21886 : 
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r298007642
 
 

 ##
 File path: 
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
 ##
 @@ -92,16 +93,40 @@ private Scenario 
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
 }
   }
 
-  // return true, if event needs to be dumped, else return false.
-  private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+  private boolean isSetForBootstrapByReplaceHandler(Context withinContext, 
String tblName) {
+return (withinContext.oldReplScope != null)
+&& 
!(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))
+&& (ReplUtils.tableIncludedInReplScope(withinContext.replScope, 
tblName));
+  }
 
-if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
-  // If the table is renamed after being added to the list of tables to be 
bootstrapped, then remove it from the
-  // list of tables to be bootstrapped.
-  boolean oldTableIsPresent = 
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
+  // Return true, if event needs to be dumped, else return false.
+  private boolean handleForTableLevelReplication(Context withinContext, String 
tblName) {
 
 Review comment:
   code removed
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 268225)
Time Spent: 9h  (was: 8h 50m)

> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
>  Time Spent: 9h
>  Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace 
> policy is getting executed, it needs to take care of the policy inclusion in 
> both the policy for each table name.
>  1. Create a list of tables to be bootstrapped. 
>   2. During handling of alter table, if the alter type is rename 
>       1. If the old table name is present in the list of table to be 
> bootstrapped, remove it.
>        2. If the new table name, matches the new policy, add it to the list 
> of tables to be bootstrapped.
>   3. During handling of drop table
>        1. if the table is in the list of tables to be bootstrapped, then 
> remove it and ignore the event.
>   4. During other event handling 
>        1. if the table is there in the list of tables to be bootstrapped, 
> then ignore the event.
>  
> Rename handling during replace policy
>  # Old name not matching old policy – The old table will not be there at the 
> target cluster. The table will not be returned by get-all-table.
>  ## Old name is not matching new policy
>  ### New name not matching old policy
>   New name not matching new policy
>  * Ignore the event, no need to do anything.
>   New name matching new policy
>  * The table will be returned by get-all-table. Replace policy handler 
> will bootstrap this table as its matching new policy and not matching old 
> policy.
>  * All the future events will be ignored as part of check added by 
> replace policy handling.
>  * All the event with old table name will anyways be ignored as the old 
> name is not matching the new policy.
>  ### New name matching old policy
>   New name not matching new policy
>  * As the new name is not matching the new policy, the table need not be 
> replicated.
>  * As the old name is not matching the new policy, the rename events will 
> be ignored.
>  * So nothing to be done for this scenario.
>   New name matching new policy
>  * As the

[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy



 [ 
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268224=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268224
 ]

ASF GitHub Bot logged work on HIVE-21886:
-

Author: ASF GitHub Bot
Created on: 27/Jun/19 05:07
Start Date: 27/Jun/19 05:07
Worklog Time Spent: 10m 
  Work Description: maheshk114 commented on pull request #688: HIVE-21886 : 
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r298007570
 
 

 ##
 File path: 
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
 ##
 @@ -92,16 +93,40 @@ private Scenario 
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
 }
   }
 
-  // return true, if event needs to be dumped, else return false.
-  private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+  private boolean isSetForBootstrapByReplaceHandler(Context withinContext, 
String tblName) {
 
 Review comment:
   code removed
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 268224)
Time Spent: 8h 50m  (was: 8h 40m)

> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
>  Time Spent: 8h 50m
>  Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace 
> policy is getting executed, it needs to take care of the policy inclusion in 
> both the policy for each table name.
>  1. Create a list of tables to be bootstrapped. 
>   2. During handling of alter table, if the alter type is rename 
>       1. If the old table name is present in the list of table to be 
> bootstrapped, remove it.
>        2. If the new table name, matches the new policy, add it to the list 
> of tables to be bootstrapped.
>   3. During handling of drop table
>        1. if the table is in the list of tables to be bootstrapped, then 
> remove it and ignore the event.
>   4. During other event handling 
>        1. if the table is there in the list of tables to be bootstrapped, 
> then ignore the event.
>  
> Rename handling during replace policy
>  # Old name not matching old policy – The old table will not be there at the 
> target cluster. The table will not be returned by get-all-table.
>  ## Old name is not matching new policy
>  ### New name not matching old policy
>   New name not matching new policy
>  * Ignore the event, no need to do anything.
>   New name matching new policy
>  * The table will be returned by get-all-table. Replace policy handler 
> will bootstrap this table as its matching new policy and not matching old 
> policy.
>  * All the future events will be ignored as part of check added by 
> replace policy handling.
>  * All the event with old table name will anyways be ignored as the old 
> name is not matching the new policy.
>  ### New name matching old policy
>   New name not matching new policy
>  * As the new name is not matching the new policy, the table need not be 
> replicated.
>  * As the old name is not matching the new policy, the rename events will 
> be ignored.
>  * So nothing to be done for this scenario.
>   New name matching new policy
>  * As the new name is matching both old and new policy, replace handler 
> will not bootstrap the table.
>  * Add the table to the list of tables to be bootstrapped.
>  * Ignore all the events with new name.
>  * If there is a drop event for the table (with new name), then remove 
> the table from the the list of table to be bootstrapped.
>  * In case of rename event (double rename)
>  ** If the new name satisfies the table pattern, then add the new name to 
> the list of tables to be bootstrapped and remove the old name from the list 
> of tables to be bootstrapped.
>  ** If the new name does not satisfies then just removed the table name 
> from the list of tables to be bootstrapped.
>  ## Old name is matching new

[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy



 [ 
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268222=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268222
 ]

ASF GitHub Bot logged work on HIVE-21886:
-

Author: ASF GitHub Bot
Created on: 27/Jun/19 05:06
Start Date: 27/Jun/19 05:06
Worklog Time Spent: 10m 
  Work Description: maheshk114 commented on pull request #688: HIVE-21886 : 
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r298007457
 
 

 ##
 File path: 
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
 ##
 @@ -120,18 +121,18 @@ private boolean handleForTableLevelReplication(Context 
withinContext, String tbl
   }
 
   // Return true, if event needs to be dumped, else return false.
-  private boolean handleForTableLevelReplicationForRename(Context 
withinContext, String oldName, String newName) {
+  private boolean renameHandlerForTableLevelReplication(Context withinContext, 
String oldName, String newName) {
 // If the table is renamed after being added to the list of tables to be 
bootstrapped, then remove it from the
 // list of tables to be bootstrapped.
-boolean oldTableIsPresent = 
withinContext.removeFromListOfTablesForBootstrap(oldName);
+boolean oldTableInBootstrapList = 
withinContext.removeFromListOfTablesForBootstrap(oldName);
 
 ReplScope oldPolicy = withinContext.oldReplScope == null ? 
withinContext.replScope : withinContext.oldReplScope;
 if (ReplUtils.tableIncludedInReplScope(oldPolicy, oldName)) {
   // If old table satisfies the filter, but the new table does not, then 
the old table should be dropped.
   // This should be done, only if the old table is not in the list of 
tables to be bootstrapped which is a multi
   // rename case. In case of multi rename, only the first rename should do 
the drop.
   if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope, 
newName)) {
-if (oldTableIsPresent) {
+if (oldTableInBootstrapList) {
   // If the old table was present in the list of tables to be 
bootstrapped, then just ignore the event.
   return false;
 } else {
 
 Review comment:
   there is no issue if C doesn't match new policy. As drop is idempotent. But 
the real issue is when C match new policy. The rename event will get dumped and 
that will cause data loss as B is not present at target. I have modified the 
code to avoid all these complexity. Now if old table does not satisfy the old 
policy then dump drop event(although that is also not required). If new policy 
matches new name then add it to the bootstrap. So now for rename with replace 
policy its always bootstrap. Thats will make the code simple and as replace and 
rename are not very frequently executed operations, it should be fine.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 268222)
Time Spent: 8h 40m  (was: 8.5h)

> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
>  Time Spent: 8h 40m
>  Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace 
> policy is getting executed, it needs to take care of the policy inclusion in 
> both the policy for each table name.
>  1. Create a list of tables to be bootstrapped. 
>   2. During handling of alter table, if the alter type is rename 
>       1. If the old table name is present in the list of table to be 
> bootstrapped, remove it.
>        2. If the new table name, matches the new policy, add it to the list 
> of tables to be bootstrapped.
>   3. During handling of drop table
>        1. if the table is in the list of tables to be bootstrapped, then 
> remove it and ignore the event.
>   4. During other event handling 
>        1. if the table is there in the list of tables to be bootstrapped, 
> then ignore the event.
>  
> Rename handling during replace policy
>  # Old name not matching old policy – The old table will not be there at the 
> target

[jira] [Commented] (HIVE-21925) HiveConnection retries should support backoff



[ 
https://issues.apache.org/jira/browse/HIVE-21925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873795#comment-16873795
 ] 

Hive QA commented on HIVE-21925:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973018/HIVE-21925.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 16341 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/17764/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17764/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17764/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12973018 - PreCommit-HIVE-Build

> HiveConnection retries should support backoff
> -
>
> Key: HIVE-21925
> URL: https://issues.apache.org/jira/browse/HIVE-21925
> Project: Hive
>  Issue Type: Bug
>  Components: Clients
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-21925.patch
>
>
> Hive JDBC connection supports retries. In http mode, retries always seem to 
> happen immediately without any backoff.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-14737) Problem accessing /logs in a Kerberized Hive Server 2 Web UI

2019-06-26 Thread Daniel Dai (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-14737:
--
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 4.0.0
   Status: Resolved  (was: Patch Available)

Patch pushed to master. Thanks Rajkumar!

> Problem accessing /logs in a Kerberized Hive Server 2 Web UI
> 
>
> Key: HIVE-14737
> URL: https://issues.apache.org/jira/browse/HIVE-14737
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.1.0, 3.1.1
>Reporter: Matyas Orhidi
>Assignee: Rajkumar Singh
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-14737.01.patch, HIVE-14737.02.patch, 
> HIVE-14737.03.patch, HIVE-14737.04.patch, HIVE-14737.05.patch, 
> HIVE-14737.patch
>
>
> The /logs menu fails with error [1] when the cluster is Kerberized. Other 
> menu items are working properly.
> [1] HTTP ERROR: 401
> Problem accessing /logs/. Reason:
> Unauthenticated users are not authorized to access this page.
> Powered by Jetty://



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-14737) Problem accessing /logs in a Kerberized Hive Server 2 Web UI



 [ 
https://issues.apache.org/jira/browse/HIVE-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajkumar Singh updated HIVE-14737:
--
Attachment: HIVE-14737.05.patch
Status: Patch Available  (was: Open)

> Problem accessing /logs in a Kerberized Hive Server 2 Web UI
> 
>
> Key: HIVE-14737
> URL: https://issues.apache.org/jira/browse/HIVE-14737
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.1.1, 1.1.0
>Reporter: Matyas Orhidi
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-14737.01.patch, HIVE-14737.02.patch, 
> HIVE-14737.03.patch, HIVE-14737.04.patch, HIVE-14737.05.patch, 
> HIVE-14737.patch
>
>
> The /logs menu fails with error [1] when the cluster is Kerberized. Other 
> menu items are working properly.
> [1] HTTP ERROR: 401
> Problem accessing /logs/. Reason:
> Unauthenticated users are not authorized to access this page.
> Powered by Jetty://



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-14737) Problem accessing /logs in a Kerberized Hive Server 2 Web UI



 [ 
https://issues.apache.org/jira/browse/HIVE-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajkumar Singh updated HIVE-14737:
--
Status: Open  (was: Patch Available)

> Problem accessing /logs in a Kerberized Hive Server 2 Web UI
> 
>
> Key: HIVE-14737
> URL: https://issues.apache.org/jira/browse/HIVE-14737
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.1.1, 1.1.0
>Reporter: Matyas Orhidi
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-14737.01.patch, HIVE-14737.02.patch, 
> HIVE-14737.03.patch, HIVE-14737.04.patch, HIVE-14737.patch
>
>
> The /logs menu fails with error [1] when the cluster is Kerberized. Other 
> menu items are working properly.
> [1] HTTP ERROR: 401
> Problem accessing /logs/. Reason:
> Unauthenticated users are not authorized to access this page.
> Powered by Jetty://



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21925) HiveConnection retries should support backoff



[ 
https://issues.apache.org/jira/browse/HIVE-21925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873780#comment-16873780
 ] 

Hive QA commented on HIVE-21925:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
28s{color} | {color:blue} jdbc in master has 16 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
20s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
12s{color} | {color:red} jdbc: The patch generated 1 new + 34 unchanged - 0 
fixed = 35 total (was 34) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 28s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17764/dev-support/hive-personality.sh
 |
| git revision | master / ee3aeb7 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17764/yetus/diff-checkstyle-jdbc.txt
 |
| modules | C: jdbc U: jdbc |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17764/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> HiveConnection retries should support backoff
> -
>
> Key: HIVE-21925
> URL: https://issues.apache.org/jira/browse/HIVE-21925
> Project: Hive
>  Issue Type: Bug
>  Components: Clients
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-21925.patch
>
>
> Hive JDBC connection supports retries. In http mode, retries always seem to 
> happen immediately without any backoff.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21925) HiveConnection retries should support backoff



 [ 
https://issues.apache.org/jira/browse/HIVE-21925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajkumar Singh updated HIVE-21925:
--
Attachment: HIVE-21925.patch
Status: Patch Available  (was: Open)

> HiveConnection retries should support backoff
> -
>
> Key: HIVE-21925
> URL: https://issues.apache.org/jira/browse/HIVE-21925
> Project: Hive
>  Issue Type: Bug
>  Components: Clients
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-21925.patch
>
>
> Hive JDBC connection supports retries. In http mode, retries always seem to 
> happen immediately without any backoff.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21925) HiveConnection retries should support backoff



 [ 
https://issues.apache.org/jira/browse/HIVE-21925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajkumar Singh updated HIVE-21925:
--
Status: Open  (was: Patch Available)

> HiveConnection retries should support backoff
> -
>
> Key: HIVE-21925
> URL: https://issues.apache.org/jira/browse/HIVE-21925
> Project: Hive
>  Issue Type: Bug
>  Components: Clients
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-21925.patch
>
>
> Hive JDBC connection supports retries. In http mode, retries always seem to 
> happen immediately without any backoff.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21925) HiveConnection retries should support backoff



 [ 
https://issues.apache.org/jira/browse/HIVE-21925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajkumar Singh updated HIVE-21925:
--
Attachment: (was: HIVE-21925.patch)

> HiveConnection retries should support backoff
> -
>
> Key: HIVE-21925
> URL: https://issues.apache.org/jira/browse/HIVE-21925
> Project: Hive
>  Issue Type: Bug
>  Components: Clients
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-21925.patch
>
>
> Hive JDBC connection supports retries. In http mode, retries always seem to 
> happen immediately without any backoff.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21925) HiveConnection retries should support backoff



[ 
https://issues.apache.org/jira/browse/HIVE-21925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873763#comment-16873763
 ] 

Hive QA commented on HIVE-21925:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973010/HIVE-21925.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/17763/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17763/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17763/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-06-27 02:17:18.827
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-17763/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-06-27 02:17:18.831
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at ee3aeb7 HIVE-21752 Thread Safety and Memory Leaks in 
HCatRecordObjectInspectorFactory (Jalpan Randeri via Alan Gates)
+ git clean -f -d
Removing ${project.basedir}/
Removing itests/${project.basedir}/
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at ee3aeb7 HIVE-21752 Thread Safety and Memory Leaks in 
HCatRecordObjectInspectorFactory (Jalpan Randeri via Alan Gates)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-06-27 02:17:19.661
+ rm -rf ../yetus_PreCommit-HIVE-Build-17763
+ mkdir ../yetus_PreCommit-HIVE-Build-17763
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-17763
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-17763/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh 
/data/hiveptest/working/scratch/build.patch
error: a/jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java: does not exist 
in index
Going to apply patch with: git apply -p1
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q 
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc4559357827667831756.exe, --version]
libprotoc 2.5.0
protoc-jar: executing: [/tmp/protoc4559357827667831756.exe, 
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
 
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
 
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator  Version 3.5.2
protoc-jar: executing: [/tmp/protoc7682582718199574526.exe, --version]
libprotoc 2.5.0
ANTLR Parser Generator  Version 3.5.2
Output file 
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/target/generated-sources/org/apache/hadoop/hive/metastore/parser/FilterParser.java
 does not exist: must build 
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/parser/Filter.g
org/apache/hadoop/hive/metastore/parser/Filter.g
log4j:WARN No appenders could be found for logger (DataNucleus.Persistence).
log4j:WARN Please initialize the log4j system properly.
DataNucleus Enhancer (version 4.1.17) for API "JDO"
DataNucleus Enhancer completed with success for 41 classes.
ANTLR Parser Generator  Version 3.5.2
Output file

[jira] [Commented] (HIVE-14737) Problem accessing /logs in a Kerberized Hive Server 2 Web UI



[ 
https://issues.apache.org/jira/browse/HIVE-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873761#comment-16873761
 ] 

Hive QA commented on HIVE-14737:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973009/HIVE-14737.04.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/17762/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17762/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17762/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL 
https://issues.apache.org/jira/secure/attachment/12973009/HIVE-14737.04.patch 
was found in seen patch url's cache and a test was probably run already on it. 
Aborting...
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12973009 - PreCommit-HIVE-Build

> Problem accessing /logs in a Kerberized Hive Server 2 Web UI
> 
>
> Key: HIVE-14737
> URL: https://issues.apache.org/jira/browse/HIVE-14737
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.1.0, 3.1.1
>Reporter: Matyas Orhidi
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-14737.01.patch, HIVE-14737.02.patch, 
> HIVE-14737.03.patch, HIVE-14737.04.patch, HIVE-14737.patch
>
>
> The /logs menu fails with error [1] when the cluster is Kerberized. Other 
> menu items are working properly.
> [1] HTTP ERROR: 401
> Problem accessing /logs/. Reason:
> Unauthenticated users are not authorized to access this page.
> Powered by Jetty://



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-14737) Problem accessing /logs in a Kerberized Hive Server 2 Web UI



[ 
https://issues.apache.org/jira/browse/HIVE-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873760#comment-16873760
 ] 

Hive QA commented on HIVE-14737:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973009/HIVE-14737.04.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 16341 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/17761/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17761/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17761/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12973009 - PreCommit-HIVE-Build

> Problem accessing /logs in a Kerberized Hive Server 2 Web UI
> 
>
> Key: HIVE-14737
> URL: https://issues.apache.org/jira/browse/HIVE-14737
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.1.0, 3.1.1
>Reporter: Matyas Orhidi
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-14737.01.patch, HIVE-14737.02.patch, 
> HIVE-14737.03.patch, HIVE-14737.04.patch, HIVE-14737.patch
>
>
> The /logs menu fails with error [1] when the cluster is Kerberized. Other 
> menu items are working properly.
> [1] HTTP ERROR: 401
> Problem accessing /logs/. Reason:
> Unauthenticated users are not authorized to access this page.
> Powered by Jetty://



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-14737) Problem accessing /logs in a Kerberized Hive Server 2 Web UI



[ 
https://issues.apache.org/jira/browse/HIVE-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873737#comment-16873737
 ] 

Hive QA commented on HIVE-14737:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
35s{color} | {color:blue} common in master has 62 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
12s{color} | {color:red} common: The patch generated 1 new + 22 unchanged - 0 
fixed = 23 total (was 22) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 30s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17761/dev-support/hive-personality.sh
 |
| git revision | master / ee3aeb7 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17761/yetus/diff-checkstyle-common.txt
 |
| modules | C: common U: common |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17761/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Problem accessing /logs in a Kerberized Hive Server 2 Web UI
> 
>
> Key: HIVE-14737
> URL: https://issues.apache.org/jira/browse/HIVE-14737
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.1.0, 3.1.1
>Reporter: Matyas Orhidi
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-14737.01.patch, HIVE-14737.02.patch, 
> HIVE-14737.03.patch, HIVE-14737.04.patch, HIVE-14737.patch
>
>
> The /logs menu fails with error [1] when the cluster is Kerberized. Other 
> menu items are working properly.
> [1] HTTP ERROR: 401
> Problem accessing /logs/. Reason:
> Unauthenticated users are not authorized to access this page.
> Powered by Jetty://



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21921) Support for correlated quantified predicates



[ 
https://issues.apache.org/jira/browse/HIVE-21921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873728#comment-16873728
 ] 

Hive QA commented on HIVE-21921:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973002/HIVE-21921.4.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/17760/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17760/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17760/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL 
https://issues.apache.org/jira/secure/attachment/12973002/HIVE-21921.4.patch 
was found in seen patch url's cache and a test was probably run already on it. 
Aborting...
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12973002 - PreCommit-HIVE-Build

> Support for correlated quantified predicates
> 
>
> Key: HIVE-21921
> URL: https://issues.apache.org/jira/browse/HIVE-21921
> Project: Hive
>  Issue Type: New Feature
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21921.1.patch, HIVE-21921.2.patch, 
> HIVE-21921.3.patch, HIVE-21921.4.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21921) Support for correlated quantified predicates



[ 
https://issues.apache.org/jira/browse/HIVE-21921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873727#comment-16873727
 ] 

Hive QA commented on HIVE-21921:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973002/HIVE-21921.4.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 16342 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/17759/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17759/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17759/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12973002 - PreCommit-HIVE-Build

> Support for correlated quantified predicates
> 
>
> Key: HIVE-21921
> URL: https://issues.apache.org/jira/browse/HIVE-21921
> Project: Hive
>  Issue Type: New Feature
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21921.1.patch, HIVE-21921.2.patch, 
> HIVE-21921.3.patch, HIVE-21921.4.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21925) HiveConnection retries should support backoff



 [ 
https://issues.apache.org/jira/browse/HIVE-21925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajkumar Singh updated HIVE-21925:
--
Attachment: HIVE-21925.patch
Status: Patch Available  (was: In Progress)

> HiveConnection retries should support backoff
> -
>
> Key: HIVE-21925
> URL: https://issues.apache.org/jira/browse/HIVE-21925
> Project: Hive
>  Issue Type: Bug
>  Components: Clients
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-21925.patch
>
>
> Hive JDBC connection supports retries. In http mode, retries always seem to 
> happen immediately without any backoff.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-14737) Problem accessing /logs in a Kerberized Hive Server 2 Web UI



 [ 
https://issues.apache.org/jira/browse/HIVE-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajkumar Singh updated HIVE-14737:
--
Affects Version/s: 3.1.1
   Attachment: HIVE-14737.04.patch
   Status: Patch Available  (was: Open)

> Problem accessing /logs in a Kerberized Hive Server 2 Web UI
> 
>
> Key: HIVE-14737
> URL: https://issues.apache.org/jira/browse/HIVE-14737
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.1.1, 1.1.0
>Reporter: Matyas Orhidi
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-14737.01.patch, HIVE-14737.02.patch, 
> HIVE-14737.03.patch, HIVE-14737.04.patch, HIVE-14737.patch
>
>
> The /logs menu fails with error [1] when the cluster is Kerberized. Other 
> menu items are working properly.
> [1] HTTP ERROR: 401
> Problem accessing /logs/. Reason:
> Unauthenticated users are not authorized to access this page.
> Powered by Jetty://



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-21925) HiveConnection retries should support backoff



 [ 
https://issues.apache.org/jira/browse/HIVE-21925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajkumar Singh reassigned HIVE-21925:
-

Assignee: Rajkumar Singh

> HiveConnection retries should support backoff
> -
>
> Key: HIVE-21925
> URL: https://issues.apache.org/jira/browse/HIVE-21925
> Project: Hive
>  Issue Type: Bug
>  Components: Clients
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Rajkumar Singh
>Priority: Major
>
> Hive JDBC connection supports retries. In http mode, retries always seem to 
> happen immediately without any backoff.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work started] (HIVE-21925) HiveConnection retries should support backoff



 [ 
https://issues.apache.org/jira/browse/HIVE-21925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-21925 started by Rajkumar Singh.
-
> HiveConnection retries should support backoff
> -
>
> Key: HIVE-21925
> URL: https://issues.apache.org/jira/browse/HIVE-21925
> Project: Hive
>  Issue Type: Bug
>  Components: Clients
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Rajkumar Singh
>Priority: Major
>
> Hive JDBC connection supports retries. In http mode, retries always seem to 
> happen immediately without any backoff.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21921) Support for correlated quantified predicates



[ 
https://issues.apache.org/jira/browse/HIVE-21921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873715#comment-16873715
 ] 

Hive QA commented on HIVE-21921:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m  
5s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
5s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
41s{color} | {color:red} ql: The patch generated 19 new + 153 unchanged - 1 
fixed = 172 total (was 154) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 48s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17759/dev-support/hive-personality.sh
 |
| git revision | master / ee3aeb7 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17759/yetus/diff-checkstyle-ql.txt
 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17759/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Support for correlated quantified predicates
> 
>
> Key: HIVE-21921
> URL: https://issues.apache.org/jira/browse/HIVE-21921
> Project: Hive
>  Issue Type: New Feature
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21921.1.patch, HIVE-21921.2.patch, 
> HIVE-21921.3.patch, HIVE-21921.4.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21888) Set hive.parquet.timestamp.skip.conversion default to true



[ 
https://issues.apache.org/jira/browse/HIVE-21888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873700#comment-16873700
 ] 

Hive QA commented on HIVE-21888:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972988/HIVE-21888.02.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16341 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT[2] (batchId=350)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/17758/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17758/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17758/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12972988 - PreCommit-HIVE-Build

> Set hive.parquet.timestamp.skip.conversion default to true
> --
>
> Key: HIVE-21888
> URL: https://issues.apache.org/jira/browse/HIVE-21888
> Project: Hive
>  Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-21888.02.patch, HIVE-21888.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21888) Set hive.parquet.timestamp.skip.conversion default to true



[ 
https://issues.apache.org/jira/browse/HIVE-21888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873689#comment-16873689
 ] 

Hive QA commented on HIVE-21888:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
54s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
57s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
35s{color} | {color:blue} common in master has 62 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m  
0s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 51s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17758/dev-support/hive-personality.sh
 |
| git revision | master / ee3aeb7 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17758/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Set hive.parquet.timestamp.skip.conversion default to true
> --
>
> Key: HIVE-21888
> URL: https://issues.apache.org/jira/browse/HIVE-21888
> Project: Hive
>  Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-21888.02.patch, HIVE-21888.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21616) Implement JSON_VALUE, JSON_QUERY, and IS [NOT] JSON



[ 
https://issues.apache.org/jira/browse/HIVE-21616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873681#comment-16873681
 ] 

Hive QA commented on HIVE-21616:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972989/HIVE-21616.2.patch

{color:green}SUCCESS:{color} +1 due to 12 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 16639 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/17757/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17757/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17757/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12972989 - PreCommit-HIVE-Build

> Implement JSON_VALUE, JSON_QUERY, and IS [NOT] JSON
> ---
>
> Key: HIVE-21616
> URL: https://issues.apache.org/jira/browse/HIVE-21616
> Project: Hive
>  Issue Type: Sub-task
>  Components: UDF
>Affects Versions: 4.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
> Attachments: HIVE-21615.patch, HIVE-21616.2.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21616) Implement JSON_VALUE, JSON_QUERY, and IS [NOT] JSON



[ 
https://issues.apache.org/jira/browse/HIVE-21616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873678#comment-16873678
 ] 

Hive QA commented on HIVE-21616:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  4m 
 7s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  6m 
10s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  1m  
4s{color} | {color:blue} hplsql in master has 157 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  8m 
37s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m  
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  8m  
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
52s{color} | {color:red} ql: The patch generated 682 new + 484 unchanged - 0 
fixed = 1166 total (was 484) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  2m 
18s{color} | {color:red} root: The patch generated 682 new + 484 unchanged - 0 
fixed = 1166 total (was 484) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  4m 
23s{color} | {color:red} ql generated 59 new + 2247 unchanged - 6 fixed = 2306 
total (was 2253) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  7m 
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 75m 33s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
|  |  Dead store to LA29_5 in 
org.apache.hadoop.hive.ql.parse.HiveLexer$DFA29.specialStateTransition(int, 
IntStream)  At 
HiveLexer.java:org.apache.hadoop.hive.ql.parse.HiveLexer$DFA29.specialStateTransition(int,
 IntStream)  At HiveLexer.java:[line 12962] |
|  |  Dead store to KW_IN244 in 
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceSimilarExpressionAtom(CommonTree)
  At 
HiveParser_IdentifiersParser.java:org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceSimilarExpressionAtom(CommonTree)
  At HiveParser_IdentifiersParser.java:[line 9701] |
|  |  Dead store to LPAREN234 in 
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.subQueryExpression()
  At 
HiveParser_IdentifiersParser.java:org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.subQueryExpression()
  At HiveParser_IdentifiersParser.java:[line 9203] |
|  |  Dead store to RPAREN236 in 
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.subQueryExpression()
  At 
HiveParser_IdentifiersParser.java:org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.subQueryExpression()
  At HiveParser_IdentifiersParser.java:[line 9210] |
|  |  Redundant nullcheck of nonReserved314, which is known to be non-null in 
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.identifier()  
Redundant null check at

[jira] [Updated] (HIVE-21921) Support for correlated quantified predicates



 [ 
https://issues.apache.org/jira/browse/HIVE-21921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-21921:
---
Status: Patch Available  (was: Open)

> Support for correlated quantified predicates
> 
>
> Key: HIVE-21921
> URL: https://issues.apache.org/jira/browse/HIVE-21921
> Project: Hive
>  Issue Type: New Feature
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21921.1.patch, HIVE-21921.2.patch, 
> HIVE-21921.3.patch, HIVE-21921.4.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21921) Support for correlated quantified predicates



 [ 
https://issues.apache.org/jira/browse/HIVE-21921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-21921:
---
Attachment: HIVE-21921.4.patch

> Support for correlated quantified predicates
> 
>
> Key: HIVE-21921
> URL: https://issues.apache.org/jira/browse/HIVE-21921
> Project: Hive
>  Issue Type: New Feature
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21921.1.patch, HIVE-21921.2.patch, 
> HIVE-21921.3.patch, HIVE-21921.4.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21921) Support for correlated quantified predicates



 [ 
https://issues.apache.org/jira/browse/HIVE-21921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-21921:
---
Status: Open  (was: Patch Available)

> Support for correlated quantified predicates
> 
>
> Key: HIVE-21921
> URL: https://issues.apache.org/jira/browse/HIVE-21921
> Project: Hive
>  Issue Type: New Feature
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21921.1.patch, HIVE-21921.2.patch, 
> HIVE-21921.3.patch, HIVE-21921.4.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21923) Vectorized MapJoin may miss results when only the join key is selected



[ 
https://issues.apache.org/jira/browse/HIVE-21923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873667#comment-16873667
 ] 

Hive QA commented on HIVE-21923:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
 9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 26m 
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
35s{color} | {color:blue} common in master has 62 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m  
1s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  7m 
46s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
34s{color} | {color:red} common: The patch generated 1 new + 1915 unchanged - 0 
fixed = 1916 total (was 1915) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 20m 
55s{color} | {color:red} root: The patch generated 1 new + 73196 unchanged - 0 
fixed = 73197 total (was 73196) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
50s{color} | {color:red} patch/common cannot run setBugDatabaseInfo from 
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  5m 
31s{color} | {color:red} patch/ql cannot run setBugDatabaseInfo from findbugs 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 11m 
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
27s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}122m 32s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17756/dev-support/hive-personality.sh
 |
| git revision | master / ee3aeb7 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17756/yetus/diff-checkstyle-common.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17756/yetus/diff-checkstyle-root.txt
 |
| findbugs | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17756/yetus/patch-findbugs-common.txt
 |
| findbugs | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17756/yetus/patch-findbugs-ql.txt
 |
| modules | C: common ql . U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17756/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Vectorized MapJoin may miss results when only the join key is selected
> --
>
> Key: HIVE-21923
> URL: https://issues.apache.org/jira/browse/HIVE-21923
> Project: Hive
>  Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>

[jira] [Commented] (HIVE-21923) Vectorized MapJoin may miss results when only the join key is selected



[ 
https://issues.apache.org/jira/browse/HIVE-21923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873650#comment-16873650
 ] 

Hive QA commented on HIVE-21923:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972967/HIVE-21923.01.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 16341 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[correlationoptimizer4]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[hybridgrace_hashjoin_2]
 (batchId=110)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join14] 
(batchId=118)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucket_map_join_tez1]
 (batchId=151)
org.apache.hadoop.hive.ql.TestTxnCommandsWithSplitUpdateAndVectorization.testMergeDeleteUpdate
 (batchId=322)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/17756/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17756/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17756/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12972967 - PreCommit-HIVE-Build

> Vectorized MapJoin may miss results when only the join key is selected
> --
>
> Key: HIVE-21923
> URL: https://issues.apache.org/jira/browse/HIVE-21923
> Project: Hive
>  Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21923.01.patch
>
>
> HIVE-21189 have introduced some resultset changes
> in ql/src/test/results/clientpositive/llap/hybridgrace_hashjoin_2.q.out
> https://github.com/apache/hive/commit/5799398450c17d06e8ef144ce835a8524f5abec9#diff-56b3ab96b6c90fdbebe2c4f84e8595afL500



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21868) Vectorize CAST...FORMAT



[ 
https://issues.apache.org/jira/browse/HIVE-21868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873597#comment-16873597
 ] 

Hive QA commented on HIVE-21868:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972946/HIVE-21868.05.patch

{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 16348 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/17755/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17755/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17755/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12972946 - PreCommit-HIVE-Build

> Vectorize CAST...FORMAT
> ---
>
> Key: HIVE-21868
> URL: https://issues.apache.org/jira/browse/HIVE-21868
> Project: Hive
>  Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-21868.01.patch, HIVE-21868.01.patch, 
> HIVE-21868.02.patch, HIVE-21868.03.patch, HIVE-21868.04.patch, 
> HIVE-21868.05.patch
>
>
> Vectorize UDFs for CAST ( AS STRING/CHAR/VARCHAR FORMAT 
> ) and CAST ( AS TIMESTAMP/DATE FORMAT ).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21868) Vectorize CAST...FORMAT



[ 
https://issues.apache.org/jira/browse/HIVE-21868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873577#comment-16873577
 ] 

Hive QA commented on HIVE-21868:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
57s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
34s{color} | {color:blue} common in master has 62 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
59s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
12s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
44s{color} | {color:green} ql: The patch generated 0 new + 412 unchanged - 1 
fixed = 412 total (was 413) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  4m 
13s{color} | {color:red} ql generated 2 new + 2253 unchanged - 0 fixed = 2255 
total (was 2253) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 49s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
|  |  Found reliance on default encoding in 
org.apache.hadoop.hive.ql.exec.vector.expressions.CastDateToString.sqlFormat(BytesColumnVector,
 long[], int, HiveSqlDateTimeFormatter):in 
org.apache.hadoop.hive.ql.exec.vector.expressions.CastDateToString.sqlFormat(BytesColumnVector,
 long[], int, HiveSqlDateTimeFormatter): String.getBytes()  At 
CastDateToString.java:[line 70] |
|  |  Found reliance on default encoding in 
org.apache.hadoop.hive.ql.exec.vector.expressions.CastTimestampToString.sqlFormat(BytesColumnVector,
 TimestampColumnVector, int, HiveSqlDateTimeFormatter):in 
org.apache.hadoop.hive.ql.exec.vector.expressions.CastTimestampToString.sqlFormat(BytesColumnVector,
 TimestampColumnVector, int, HiveSqlDateTimeFormatter): String.getBytes()  At 
CastTimestampToString.java:[line 79] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17755/dev-support/hive-personality.sh
 |
| git revision | master / ee3aeb7 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17755/yetus/new-findbugs-ql.html
 |
| modules | C: common ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17755/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Vectorize CAST...FORMAT
>

[jira] [Updated] (HIVE-21921) Support for correlated quantified predicates



 [ 
https://issues.apache.org/jira/browse/HIVE-21921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-21921:
---
Status: Patch Available  (was: Open)

> Support for correlated quantified predicates
> 
>
> Key: HIVE-21921
> URL: https://issues.apache.org/jira/browse/HIVE-21921
> Project: Hive
>  Issue Type: New Feature
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21921.1.patch, HIVE-21921.2.patch, 
> HIVE-21921.3.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21921) Support for correlated quantified predicates



 [ 
https://issues.apache.org/jira/browse/HIVE-21921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-21921:
---
Status: Open  (was: Patch Available)

> Support for correlated quantified predicates
> 
>
> Key: HIVE-21921
> URL: https://issues.apache.org/jira/browse/HIVE-21921
> Project: Hive
>  Issue Type: New Feature
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21921.1.patch, HIVE-21921.2.patch, 
> HIVE-21921.3.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21921) Support for correlated quantified predicates



 [ 
https://issues.apache.org/jira/browse/HIVE-21921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-21921:
---
Attachment: HIVE-21921.3.patch

> Support for correlated quantified predicates
> 
>
> Key: HIVE-21921
> URL: https://issues.apache.org/jira/browse/HIVE-21921
> Project: Hive
>  Issue Type: New Feature
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21921.1.patch, HIVE-21921.2.patch, 
> HIVE-21921.3.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-14737) Problem accessing /logs in a Kerberized Hive Server 2 Web UI

2019-06-26 Thread Daniel Dai (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873567#comment-16873567
 ] 

Daniel Dai commented on HIVE-14737:
---

+1 pending tests.

> Problem accessing /logs in a Kerberized Hive Server 2 Web UI
> 
>
> Key: HIVE-14737
> URL: https://issues.apache.org/jira/browse/HIVE-14737
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Matyas Orhidi
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-14737.01.patch, HIVE-14737.02.patch, 
> HIVE-14737.03.patch, HIVE-14737.patch
>
>
> The /logs menu fails with error [1] when the cluster is Kerberized. Other 
> menu items are working properly.
> [1] HTTP ERROR: 401
> Problem accessing /logs/. Reason:
> Unauthenticated users are not authorized to access this page.
> Powered by Jetty://



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21886) REPL - With table list - Handle rename events during replace policy



[ 
https://issues.apache.org/jira/browse/HIVE-21886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873558#comment-16873558
 ] 

Hive QA commented on HIVE-21886:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972944/HIVE-21886.02.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 16314 tests 
executed
*Failed tests:*
{noformat}
TestReplAcrossInstancesWithJsonMessageFormat - did not produce a TEST-*.xml 
file (likely timed out) (batchId=255)
org.apache.hive.service.cli.session.TestSessionManagerMetrics.testAbandonedSessionMetrics
 (batchId=236)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/17754/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17754/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17754/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12972944 - PreCommit-HIVE-Build

> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
>  Time Spent: 8.5h
>  Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace 
> policy is getting executed, it needs to take care of the policy inclusion in 
> both the policy for each table name.
>  1. Create a list of tables to be bootstrapped. 
>   2. During handling of alter table, if the alter type is rename 
>       1. If the old table name is present in the list of table to be 
> bootstrapped, remove it.
>        2. If the new table name, matches the new policy, add it to the list 
> of tables to be bootstrapped.
>   3. During handling of drop table
>        1. if the table is in the list of tables to be bootstrapped, then 
> remove it and ignore the event.
>   4. During other event handling 
>        1. if the table is there in the list of tables to be bootstrapped, 
> then ignore the event.
>  
> Rename handling during replace policy
>  # Old name not matching old policy – The old table will not be there at the 
> target cluster. The table will not be returned by get-all-table.
>  ## Old name is not matching new policy
>  ### New name not matching old policy
>   New name not matching new policy
>  * Ignore the event, no need to do anything.
>   New name matching new policy
>  * The table will be returned by get-all-table. Replace policy handler 
> will bootstrap this table as its matching new policy and not matching old 
> policy.
>  * All the future events will be ignored as part of check added by 
> replace policy handling.
>  * All the event with old table name will anyways be ignored as the old 
> name is not matching the new policy.
>  ### New name matching old policy
>   New name not matching new policy
>  * As the new name is not matching the new policy, the table need not be 
> replicated.
>  * As the old name is not matching the new policy, the rename events will 
> be ignored.
>  * So nothing to be done for this scenario.
>   New name matching new policy
>  * As the new name is matching both old and new policy, replace handler 
> will not bootstrap the table.
>  * Add the table to the list of tables to be bootstrapped.
>  * Ignore all the events with new name.
>  * If there is a drop event for the table (with new name), then remove 
> the table from the the list of table to be bootstrapped.
>  * In case of rename event (double rename)
>  ** If the new name satisfies the table pattern, then add the new name to 
> the list of tables to be bootstrapped and remove the old name from the list 
> of tables to be bootstrapped.
>  ** If the new name does not satisfies then just removed the table name 
> from the list of tables to be bootstrapped.
>  ## Old name is matching new policy – As per replace policy handler, which 
> checks based on old table, the table should be bootstrapped and event should 
> be ignored. But rename handler should decide based on new

[jira] [Work logged] (HIVE-21867) Sort semijoin conditions to accelerate query processing



 [ 
https://issues.apache.org/jira/browse/HIVE-21867?focusedWorklogId=267802=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267802
 ]

ASF GitHub Bot logged work on HIVE-21867:
-

Author: ASF GitHub Bot
Created on: 26/Jun/19 17:29
Start Date: 26/Jun/19 17:29
Worklog Time Spent: 10m 
  Work Description: jcamachor commented on pull request #687: HIVE-21867
URL: https://github.com/apache/hive/pull/687#discussion_r297783212
 
 

 ##
 File path: ql/src/test/results/clientpositive/llap/hybridgrace_hashjoin_2.q.out
 ##
 @@ -1421,7 +1421,7 @@ STAGE PLANS:
   outputColumnNames: _col1
   input vertices:
 1 Map 5
-  Statistics: Num rows: 25 Data size: 2225 Basic 
stats: COMPLETE Column stats: COMPLETE
+  Statistics: Num rows: 4 Data size: 356 Basic stats: 
COMPLETE Column stats: COMPLETE
 
 Review comment:
   Good catch, thanks! This is unexpected indeed... Taking a look now.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 267802)
Time Spent: 50m  (was: 40m)

> Sort semijoin conditions to accelerate query processing
> ---
>
> Key: HIVE-21867
> URL: https://issues.apache.org/jira/browse/HIVE-21867
> Project: Hive
>  Issue Type: New Feature
>  Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-21867.02.patch, HIVE-21867.03.patch, 
> HIVE-21867.patch
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> The problem was tackled for CBO in HIVE-21857. Semijoin filters are 
> introduced later in the planning phase. Follow similar approach to sort them, 
> trying to accelerate filter evaluation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (HIVE-21867) Sort semijoin conditions to accelerate query processing



 [ 
https://issues.apache.org/jira/browse/HIVE-21867?focusedWorklogId=267803=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267803
 ]

ASF GitHub Bot logged work on HIVE-21867:
-

Author: ASF GitHub Bot
Created on: 26/Jun/19 17:30
Start Date: 26/Jun/19 17:30
Worklog Time Spent: 10m 
  Work Description: jcamachor commented on pull request #687: HIVE-21867
URL: https://github.com/apache/hive/pull/687#discussion_r297783357
 
 

 ##
 File path: ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java
 ##
 @@ -1766,6 +1774,59 @@ private void 
removeSemijoinOptimizationByBenefit(OptimizeTezProcContext procCtx)
   GenTezUtils.removeBranch(rs);
   GenTezUtils.removeSemiJoinOperator(procCtx.parseContext, rs, ts);
 }
+
+for (Entry> e : 
globalReductionFactorMap.asMap().entrySet()) {
 
 Review comment:
   Good idea, will do.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 267803)
Time Spent: 1h  (was: 50m)

> Sort semijoin conditions to accelerate query processing
> ---
>
> Key: HIVE-21867
> URL: https://issues.apache.org/jira/browse/HIVE-21867
> Project: Hive
>  Issue Type: New Feature
>  Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-21867.02.patch, HIVE-21867.03.patch, 
> HIVE-21867.patch
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> The problem was tackled for CBO in HIVE-21857. Semijoin filters are 
> introduced later in the planning phase. Follow similar approach to sort them, 
> trying to accelerate filter evaluation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21886) REPL - With table list - Handle rename events during replace policy



[ 
https://issues.apache.org/jira/browse/HIVE-21886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873520#comment-16873520
 ] 

Hive QA commented on HIVE-21886:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 1s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m  
5s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
42s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
 6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
17s{color} | {color:red} itests/hive-unit: The patch generated 9 new + 0 
unchanged - 0 fixed = 9 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m  
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
15s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 57s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17754/dev-support/hive-personality.sh
 |
| git revision | master / ee3aeb7 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17754/yetus/diff-checkstyle-itests_hive-unit.txt
 |
| modules | C: ql itests/hive-unit U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17754/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
>  Time Spent: 8.5h
>  Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace 
> policy is getting executed, it needs to take care of the policy inclusion in 
> both the policy for each table name.
>  1. Create a list of tables to be bootstrapped. 
>   2. During handling of alter table, if the alter type is rename 
>       1. If the old table name is present in the list of table to be 
> bootstrapped, remove it.
>        2. If the

[jira] [Updated] (HIVE-21616) Implement JSON_VALUE, JSON_QUERY, and IS [NOT] JSON

2019-06-26 Thread Alan Gates (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-21616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-21616:
--
Attachment: HIVE-21616.2.patch

> Implement JSON_VALUE, JSON_QUERY, and IS [NOT] JSON
> ---
>
> Key: HIVE-21616
> URL: https://issues.apache.org/jira/browse/HIVE-21616
> Project: Hive
>  Issue Type: Sub-task
>  Components: UDF
>Affects Versions: 4.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
> Attachments: HIVE-21615.patch, HIVE-21616.2.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21888) Set hive.parquet.timestamp.skip.conversion default to true

2019-06-26 Thread Karen Coppage (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-21888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873492#comment-16873492
 ] 

Karen Coppage commented on HIVE-21888:
--

Patch 2: removed setting to true in qtests

> Set hive.parquet.timestamp.skip.conversion default to true
> --
>
> Key: HIVE-21888
> URL: https://issues.apache.org/jira/browse/HIVE-21888
> Project: Hive
>  Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-21888.02.patch, HIVE-21888.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21888) Set hive.parquet.timestamp.skip.conversion default to true

2019-06-26 Thread Karen Coppage (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-21888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karen Coppage updated HIVE-21888:
-
Status: Open  (was: Patch Available)

> Set hive.parquet.timestamp.skip.conversion default to true
> --
>
> Key: HIVE-21888
> URL: https://issues.apache.org/jira/browse/HIVE-21888
> Project: Hive
>  Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-21888.02.patch, HIVE-21888.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21888) Set hive.parquet.timestamp.skip.conversion default to true

2019-06-26 Thread Karen Coppage (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-21888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karen Coppage updated HIVE-21888:
-
Attachment: HIVE-21888.02.patch
Status: Patch Available  (was: Open)

> Set hive.parquet.timestamp.skip.conversion default to true
> --
>
> Key: HIVE-21888
> URL: https://issues.apache.org/jira/browse/HIVE-21888
> Project: Hive
>  Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-21888.02.patch, HIVE-21888.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21922) Allow keytabs to be reused in LLAP yarn applications through Yarn localization



[ 
https://issues.apache.org/jira/browse/HIVE-21922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873485#comment-16873485
 ] 

Hive QA commented on HIVE-21922:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972933/HIVE-21922.2.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/17753/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17753/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17753/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL 
https://issues.apache.org/jira/secure/attachment/12972933/HIVE-21922.2.patch 
was found in seen patch url's cache and a test was probably run already on it. 
Aborting...
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12972933 - PreCommit-HIVE-Build

> Allow keytabs to be reused in LLAP yarn applications through Yarn localization
> --
>
> Key: HIVE-21922
> URL: https://issues.apache.org/jira/browse/HIVE-21922
> Project: Hive
>  Issue Type: New Feature
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-21922.0.patch, HIVE-21922.1.patch, 
> HIVE-21922.2.patch
>
>
> In secure clusters LLAP has to be able to reach keytab files for kerberos 
> login.
> Currently _hive.llap.task.scheduler.am.registry.keytab.file_ and 
> _hive.llap.daemon.keytab.file_ configs are used to define the path of such 
> keytabs on the Tez AM and LLAP daemon side respectively. Both presume local 
> file system paths only - hence all nodes in the LLAP cluster (even those that 
> eventually don't end up executing a daemon...) have to have Hive's keytab 
> preinstalled on them.
> The above is described by this strategy: 
> [Pre-installed_Keytabs_for_AM_and_containers|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html#Pre-installed_Keytabs_for_AM_and_containers]
> Another approach can be 
> [Keytabs_for_AM_and_containers_distributed_via_YARN|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html#Keytabs_for_AM_and_containers_distributed_via_YARN]
>  where we rely on HDFS and Yarn resource localization, and no prior keytab 
> distribution is required. I intend to make this strategy an option for 
> Hive-LLAP in this jira.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21846) Create a thread in TezAM which periodically fetches LlapDaemon metrics



[ 
https://issues.apache.org/jira/browse/HIVE-21846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873483#comment-16873483
 ] 

Hive QA commented on HIVE-21846:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972939/HIVE-21846.03.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/17752/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17752/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17752/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL 
https://issues.apache.org/jira/secure/attachment/12972939/HIVE-21846.03.patch 
was found in seen patch url's cache and a test was probably run already on it. 
Aborting...
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12972939 - PreCommit-HIVE-Build

> Create a thread in TezAM which periodically fetches LlapDaemon metrics
> --
>
> Key: HIVE-21846
> URL: https://issues.apache.org/jira/browse/HIVE-21846
> Project: Hive
>  Issue Type: Sub-task
>  Components: llap, Tez
>Reporter: Peter Vary
>Assignee: Antal Sinkovits
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-21846.01.patch, HIVE-21846.02.patch, 
> HIVE-21846.03.patch
>
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> LlapTaskSchedulerService should start a thread which periodically fetches the 
> LlapDaemon metrics and stores them in the NodeInfo object.
> This should be just the first implementation - later we should find a way 
> where we do not need NxM requests between N TezAM and M LlapDaemon



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21874) Implement add partitions related methods on temporary table



[ 
https://issues.apache.org/jira/browse/HIVE-21874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873482#comment-16873482
 ] 

Hive QA commented on HIVE-21874:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972937/HIVE-21874.04.patch

{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 16598 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.addNoSuchTable[Remote]
 (batchId=224)
org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.createGetDrop[Remote]
 (batchId=224)
org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.createTableWithConstraintsPkInOtherCatalog[Remote]
 (batchId=224)
org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.createTableWithConstraintsPk[Remote]
 (batchId=224)
org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.doubleAddUniqueConstraint[Remote]
 (batchId=224)
org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.getNoSuchCatalog[Remote]
 (batchId=224)
org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.getNoSuchDb[Remote]
 (batchId=224)
org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.getNoSuchTable[Remote]
 (batchId=224)
org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.inOtherCatalog[Remote]
 (batchId=224)
org.apache.hive.hcatalog.mapreduce.TestHCatPartitioned.testHCatPartitionedTable[7]
 (batchId=211)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/17751/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17751/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17751/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12972937 - PreCommit-HIVE-Build

> Implement add partitions related methods on temporary table
> ---
>
> Key: HIVE-21874
> URL: https://issues.apache.org/jira/browse/HIVE-21874
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-21874.01.patch, HIVE-21874.02.patch, 
> HIVE-21874.03.patch, HIVE-21874.04.patch
>
>
> IMetaStoreClient exposes the following add partition related methods:
> {code:java}
> Partition add_partition(Partition partition);
> int add_partitions(List partitions);
> int add_partitions_pspec(PartitionSpecProxy partitionSpec);
> List add_partitions(List partitions, boolean 
> ifNotExists, boolean needResults);
> {code}
> These methods should be implemented in order to handle addition of partitions 
> to temporary tables.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21224) Upgrade tests JUnit3 to JUnit4

2019-06-26 Thread Alan Gates (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-21224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873463#comment-16873463
 ] 

Alan Gates commented on HIVE-21224:
---

Before committing this please make sure the ITests under standalone-metastore 
execute properly as well.  Those are not run as part of the standard nightly 
build but are needed whenever the data scripts are updated.

> Upgrade tests JUnit3 to JUnit4
> --
>
> Key: HIVE-21224
> URL: https://issues.apache.org/jira/browse/HIVE-21224
> Project: Hive
>  Issue Type: Improvement
>Reporter: Bruno Pusztahazi
>Assignee: Bruno Pusztahazi
>Priority: Major
> Attachments: HIVE-21224.1.patch, HIVE-21224.2.patch, 
> HIVE-21224.3.patch, HIVE-21224.4.patch, HIVE-21224.5.patch, 
> HIVE-21224.6.patch, HIVE-21224.7.patch, HIVE-21224.8.patch, HIVE-21224.9.patch
>
>
> Old JUnit3 tests should be upgraded to JUnit4



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21752) Thread Safety and Memory Leaks in HCatRecordObjectInspectorFactory

2019-06-26 Thread Alan Gates (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-21752:
--
   Resolution: Fixed
Fix Version/s: 4.0.0
   Status: Resolved  (was: Patch Available)

Patch committed to master.  Thanks Jalpan for the patch and sorry it took so 
long to get it committed.

> Thread Safety and Memory Leaks in HCatRecordObjectInspectorFactory
> --
>
> Key: HIVE-21752
> URL: https://issues.apache.org/jira/browse/HIVE-21752
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Reporter: Jalpan Randeri
>Assignee: Jalpan Randeri
>Priority: Minor
>  Labels: newbie, patch
> Fix For: 4.0.0
>
> Attachments: HIVE-21752.patch, HIVE-21752.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> h3. Summary
> There are a couple of issues in HCatRecordObjectInspectorFactory[1] because 
> it uses a static Java HashMap to cache objects:
>  # Java HashMap is not thread safe. This can lead to data corruptions and 
> race conditions in multithreaded servers when two threads update the 
> ObjectInspector.
>  # There is no eviction policy and as a result, this can result in memory 
> leaks. If user reads a lot of different schemas, Hive server will start 
> seeing memory pressure, once it start going to have a lot of cached record 
> and object inspectors.
> This patch propose to replace the cache using a Guava cache which enables 
> cache evictions and thread safety. Guava cache is already used in Hive 
> ObjectInspectorFactory [2], so this change is consistent with the rest of 
> Hive.
> Attached is a patch that fixes this issue.
> h3. References:
>  # 
> [https://github.com/apache/hive/blob/b58d50cb73a1f79a5d079e0a2c5ac33d2efc33a0/hcatalog/core/src/main/java/org/apache/hive/hcatalog/data/HCatRecordObjectInspectorFactory.java#L44-L47]
>  # 
> [https://github.com/apache/hive/blob/b58d50cb73a1f79a5d079e0a2c5ac33d2efc33a0/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java#L68-L87]
>  
> h4. Review Board Link:
>  *  [https://reviews.apache.org/r/70674/]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21914) Move Function and Macro related DDL operations into the DDL framework



 [ 
https://issues.apache.org/jira/browse/HIVE-21914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zoltan Haindrich updated HIVE-21914:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

pushed to master. Thank you [~mgergely]!

> Move Function and Macro related DDL operations into the DDL framework
> -
>
> Key: HIVE-21914
> URL: https://issues.apache.org/jira/browse/HIVE-21914
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
>  Labels: refactor-ddl
> Fix For: 4.0.0
>
> Attachments: HIVE-21914.01.patch, HIVE-21914.02.patch, 
> HIVE-21914.03.patch
>
>
> Some Function and Macro related operations are handled by FunctionTask, and 
> FunctionWork while they belong to the DDL framework.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21896) SHOW FUNCTIONS / SHOW FUNCTIONS LIKE - clarify



 [ 
https://issues.apache.org/jira/browse/HIVE-21896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zoltan Haindrich updated HIVE-21896:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

pushed to master. Thank you [~mgergely]!

> SHOW FUNCTIONS / SHOW FUNCTIONS LIKE - clarify
> --
>
> Key: HIVE-21896
> URL: https://issues.apache.org/jira/browse/HIVE-21896
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
>  Labels: incompatibleChange, todoc4.0
> Fix For: 4.0.0
>
> Attachments: HIVE-21896.01.patch, HIVE-21896.02.patch
>
>
> According to 
> [https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowFunctions]
>  the currently available functions can be listed like this:
> {code:java}
> SHOW FUNCTIONS ;{code}
> If the user executes this command, they will get the correct list of 
> functions, but they will also see this on the standard output:
> {code:java}
> SHOW FUNCTIONS is deprecated, please use SHOW FUNCTIONS LIKE instead.{code}
> If the user uses the
> {code:java}
> SHOW FUNCTIONS LIKE ;{code}
> command then they will receive the exact same result (though through 
> different codes). The only difference is that one can get all the function 
> names with "SHOW FUNCTIONS;", while "SHOW FUNCTIONS LIKE;" returns an 
> exception, so in this case the pattern is mandatory.
> So there should be a decision if we still accept "SHOW FUNCTIONS" without the 
> "LIKE". My suggestion is to accept it only if there is no pattern. so "SHOW 
> FUNCTIONS;" is ok, without deprecation message, but "SHOW FUNCTIONS 
> " should throw an exception.
> Whatever we decide, we should document it appropriately.
> cc [~krishahn]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21874) Implement add partitions related methods on temporary table



[ 
https://issues.apache.org/jira/browse/HIVE-21874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873439#comment-16873439
 ] 

Hive QA commented on HIVE-21874:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
 7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
57s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  1m  
6s{color} | {color:blue} standalone-metastore/metastore-server in master has 
179 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m  
2s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
 1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
40s{color} | {color:red} ql: The patch generated 1 new + 15 unchanged - 0 fixed 
= 16 total (was 15) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 45s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17751/dev-support/hive-personality.sh
 |
| git revision | master / 967a1cc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17751/yetus/diff-checkstyle-ql.txt
 |
| modules | C: standalone-metastore/metastore-server ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17751/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Implement add partitions related methods on temporary table
> ---
>
> Key: HIVE-21874
> URL: https://issues.apache.org/jira/browse/HIVE-21874
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-21874.01.patch, HIVE-21874.02.patch, 
> HIVE-21874.03.patch, HIVE-21874.04.patch
>
>
> IMetaStoreClient exposes the following add partition related methods:
> {code:java}
> Partition add_partition(Partition partition);
> int add_partitions(List partitions);
> int add_partitions_pspec(PartitionSpecProxy partitionSpec);
> List add_partitions(List partitions, boolean 
> ifNotExists, boolean needResults);
> {code}
> These methods should be implemented in order to handle addition of partitions 
> to temporary tables.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-21713) Explain consistency Task/Operator



 [ 
https://issues.apache.org/jira/browse/HIVE-21713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manoj Narayanan reassigned HIVE-21713:
--

Assignee: Manoj Narayanan

> Explain consistency Task/Operator
> -
>
> Key: HIVE-21713
> URL: https://issues.apache.org/jira/browse/HIVE-21713
> Project: Hive
>  Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Manoj Narayanan
>Priority: Major
>
> there seems to be some inconsistency what gets printed in the explain;
> for example "MoveTask" is shown as "Move Operator" in the explain
> Becase Tasks and Operators are basically different things - this should be 
> corrected as it may cause confusion



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21713) Explain consistency Task/Operator



[ 
https://issues.apache.org/jira/browse/HIVE-21713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873410#comment-16873410
 ] 

Manoj Narayanan commented on HIVE-21713:


In the above comment I tried to document each Operator and Task I could find 
and what their 'ExplainString' is. Also tried to note whether it should be 
changed (Column Needschange) and if so whether it should 'Add Operator' or 
'Change to Operator' or 'Add Task' or 'Change to Task'. 

> Explain consistency Task/Operator
> -
>
> Key: HIVE-21713
> URL: https://issues.apache.org/jira/browse/HIVE-21713
> Project: Hive
>  Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Priority: Major
>
> there seems to be some inconsistency what gets printed in the explain;
> for example "MoveTask" is shown as "Move Operator" in the explain
> Becase Tasks and Operators are basically different things - this should be 
> corrected as it may cause confusion



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HIVE-21713) Explain consistency Task/Operator



[ 
https://issues.apache.org/jira/browse/HIVE-21713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873410#comment-16873410
 ] 

Manoj Narayanan edited comment on HIVE-21713 at 6/26/19 3:04 PM:
-

In the above comment I tried to document each Operator and Task I could find 
and what their 'ExplainString' is. Also tried to note whether it should be 
changed (Column Needschange) and if so whether it should 'Add Operator' or 
'Change to Operator' or 'Add Task' or 'Change to Task'. Please review 
[~kgyrtkirk]


was (Author: mnarayanan2018):
In the above comment I tried to document each Operator and Task I could find 
and what their 'ExplainString' is. Also tried to note whether it should be 
changed (Column Needschange) and if so whether it should 'Add Operator' or 
'Change to Operator' or 'Add Task' or 'Change to Task'. 

> Explain consistency Task/Operator
> -
>
> Key: HIVE-21713
> URL: https://issues.apache.org/jira/browse/HIVE-21713
> Project: Hive
>  Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Priority: Major
>
> there seems to be some inconsistency what gets printed in the explain;
> for example "MoveTask" is shown as "Move Operator" in the explain
> Becase Tasks and Operators are basically different things - this should be 
> corrected as it may cause confusion



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21713) Explain consistency Task/Operator



[ 
https://issues.apache.org/jira/browse/HIVE-21713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873408#comment-16873408
 ] 

Manoj Narayanan commented on HIVE-21713:


|OperatorClass|OfType|ExplainString|hasOperator|hasTask|NeedsChange|ChangeToOperator|AddOperator|
|CommonMergeJoinOperator|CommonMergeJoinDesc|Merge Join 
Operator|TRUE|FALSE|FALSE|FALSE|FALSE|
|SMBMapJoinOperator|SMBJoinDesc|Sorted Merge Bucket Map Join 
Operator|TRUE|FALSE|FALSE|FALSE|FALSE|
|ScriptOperator|ScriptDesc|Transform Operator|TRUE|FALSE|FALSE|FALSE|FALSE|
|AbstractMapOperator|MapWork|NO_EXPLAIN_STRING|FALSE|FALSE|TRUE|FALSE|TRUE|
|AbstractMapJoinOperator|MapJoinDesc|Map Join 
Operator|TRUE|FALSE|FALSE|FALSE|FALSE|
|GroupByOperator|GroupByDesc|Group By Operator|TRUE|FALSE|FALSE|FALSE|FALSE|
|LimitOperator|LimitDesc|Limit|FALSE|FALSE|TRUE|FALSE|TRUE|
|AbstractFileMergeOperator|FileMergeDesc|NO_EXPLAIN_STRING|FALSE|FALSE|TRUE|FALSE|TRUE|
|HashTableDummyOperator|HashTableDummyDesc|HashTable Dummy 
Operator|TRUE|FALSE|FALSE|FALSE|FALSE|
|SelectOperator|SelectDesc|Select Operator|TRUE|FALSE|FALSE|FALSE|FALSE|
|LateralViewJoinOperator|LateralViewJoinDesc|Lateral View Join 
Operator|TRUE|FALSE|FALSE|FALSE|FALSE|
|ReduceSinkOperator|ReduceSinkDesc|Reduce Output 
Operator|TRUE|FALSE|FALSE|FALSE|FALSE|
|HashTableSinkOperator|HashTableSinkDesc|HashTable Sink 
Operator|TRUE|FALSE|FALSE|FALSE|FALSE|
|ForwardOperator|ForwardDesc|Forward|FALSE|FALSE|TRUE|FALSE|TRUE|
|TableScanOperator|TableScanDesc|TableScan|FALSE|FALSE|TRUE|FALSE|TRUE|
|TopNKeyOperator|TopNKeyDesc|Top N Key Operator|TRUE|FALSE|FALSE|FALSE|FALSE|
|TopNKeyOperator|TopNKeyDesc|keys|FALSE|FALSE|TRUE|FALSE|TRUE|
|FileSinkOperator|FileSinkDesc|File Output 
Operator|TRUE|FALSE|FALSE|FALSE|FALSE|
|JoinOperator|JoinDesc|Join Operator|TRUE|FALSE|FALSE|FALSE|FALSE|
|UnionOperator|UnionDesc|Union|FALSE|FALSE|TRUE|FALSE|TRUE|
|UDTFOperator|UDTFDesc|UDTF Operator|TRUE|FALSE|FALSE|FALSE|FALSE|
|CollectOperator|CollectDesc|Collect|FALSE|FALSE|TRUE|FALSE|TRUE|
|FilterOperator|FilterDesc|Filter Operator|TRUE|FALSE|FALSE|FALSE|FALSE|
|MapJoinOperator|MapJoinDesc|Map Join Operator|TRUE|FALSE|FALSE|FALSE|FALSE|
|DemuxOperator|DemuxDesc|Demux Operator|TRUE|FALSE|FALSE|FALSE|FALSE|
|PTFOperator|PTFDesc|PTF Operator|TRUE|FALSE|FALSE|FALSE|FALSE|
|AppMasterEventOperator|AppMasterEventDesc|Application Master Event 
Operator|TRUE|FALSE|FALSE|FALSE|FALSE|
|LateralViewForwardOperator|LateralViewForwardDesc|Lateral View 
Forward|FALSE|FALSE|TRUE|FALSE|TRUE|
|CommonJoinOperator|JoinDesc|Join Operator|TRUE|FALSE|FALSE|FALSE|FALSE|
|DummyStoreOperator|DummyStoreDesc|Dummy Store|FALSE|FALSE|TRUE|FALSE|TRUE|
|MuxOperator|MuxDesc|Mux Operator|TRUE|FALSE|FALSE|FALSE|FALSE|
|ListSinkOperator|ListSinkDesc|ListSink|FALSE|FALSE|TRUE|FALSE|TRUE|
|FetchOperator|FetchWork|Fetch Operator|TRUE|FALSE|FALSE|FALSE|FALSE|
|MapOperator| | | | | | | |
|OrcFileMergeOperator|OrcFileMergeDesc|ORC File Merge 
Operator|TRUE|FALSE|FALSE|FALSE|FALSE|
|RCFileMergeOperator|RCFileMergeDesc|RCFile Merge 
Operator|TRUE|FALSE|FALSE|FALSE|FALSE|
|SparkHashTableSinkOperator|SparkHashTableSinkDesc|Spark HashTable Sink 
Operator|TRUE|FALSE|FALSE|FALSE|FALSE|
|TemporaryHashSinkOperator| |HashTable Sink Operator| | | | | |
|TezDummyStoreOperator| |Dummy Store| | | | | |
| | | | | | | | |
|OperatorClass|OfType|ExplainString|hasOperator|hasTask|NeedsChange|ChangeToTask|AddTask|
| | | | | | | | |
|FunctionTask|FunctionWork|NO_EXPLAIN_STRING|FALSE|FALSE|TRUE|FALSE|TRUE|
| | | | | | | | |
|DDLTask|DDLWork|NO_EXPLAIN_STRING|FALSE|FALSE|TRUE|FALSE|TRUE|
|FetchTask|FetchWork|Fetch Operator|FALSE|TRUE|TRUE|TRUE|FALSE|
|ReplCopyTask|ReplCopyWork|Repl Copy|FALSE|FALSE|TRUE|FALSE|TRUE|
|ConditionalTask|ConditionalWork|Conditional 
Operator|FALSE|TRUE|TRUE|TRUE|FALSE|
|MaterializedViewTask|MaterializedViewDesc|Materialized View 
Work|FALSE|FALSE|TRUE|FALSE|TRUE|
|ColumnStatsUpdateTask|ColumnStatsUpdateWork|Column Stats Update 
Work|FALSE|FALSE|TRUE|FALSE|TRUE|
|ReplTxnTask|ReplTxnWork|Replication Transaction|FALSE|FALSE|TRUE|FALSE|TRUE|
|DependencyCollectionTask|DependencyCollectionWork|Dependency 
Collection|FALSE|FALSE|TRUE|FALSE|TRUE|
|ExplainTask|ExplainWork|NO_EXPLAIN_STRING|FALSE|FALSE|TRUE|FALSE|TRUE|
|ExportTask|ExportWork|Export Work|FALSE|FALSE|TRUE|FALSE|TRUE|
|CopyTask|CopyWork|Copy|FALSE|FALSE|TRUE|FALSE|TRUE|
|MoveTask|MoveWork|Move Operator|FALSE|TRUE|TRUE|TRUE|FALSE|
|ExplainSQRewriteTask|ExplainSQRewriteWork|NO_EXPLAIN_STRING|FALSE|FALSE|TRUE|FALSE|TRUE|
|StatsTask|StatsWork|Stats Work|FALSE|FALSE|TRUE|FALSE|TRUE|
| | | | | | |

> Explain consistency Task/Operator
> -
>
> Key: HIVE-21713
> URL: https://issues.apache.org/jira/browse/HIVE-21713
> Project: Hive
>  Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Priority: Major
>
> there seems to be some

[jira] [Commented] (HIVE-18735) Create table like loses transactional attribute



[ 
https://issues.apache.org/jira/browse/HIVE-18735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873404#comment-16873404
 ] 

Hive QA commented on HIVE-18735:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972938/HIVE-18735.04.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16340 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.llap.security.TestLlapSignerImpl.testSigning 
(batchId=350)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/17750/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17750/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17750/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12972938 - PreCommit-HIVE-Build

> Create table like loses transactional attribute
> ---
>
> Key: HIVE-18735
> URL: https://issues.apache.org/jira/browse/HIVE-18735
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 2.0.0
>Reporter: Eugene Koifman
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-18735.01.patch, HIVE-18735.02.patch, 
> HIVE-18735.03.patch, HIVE-18735.04.patch
>
>
> {noformat}
> create table T1(a int, b int) clustered by (a) into 2 buckets stored as orc 
> TBLPROPERTIES ('transactional'='true')";
> create table T like T1;
> show create table T ;
> CREATE TABLE `T`(
>   `a` int,
>   `b` int)
> CLUSTERED BY (
>   a)
> INTO 2 BUCKETS
> ROW FORMAT SERDE
>   'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
>  
> 'file:/Users/ekoifman/IdeaProjects/hive/ql/target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands-1518813536099/warehouse/t'
> TBLPROPERTIES (
>   'transient_lastDdlTime'='1518813564')
> {noformat}
> Specifying props explicitly does work 
> {noformat}
> create table T1(a int, b int) clustered by (a) into 2 buckets stored as orc 
> TBLPROPERTIES ('transactional'='true')";
> create table T like T1 TBLPROPERTIES ('transactional'='true');
> show create table T ;
> CREATE TABLE `T`(
>   `a` int,
>   `b` int)
> CLUSTERED BY (
>   a)
> INTO 2 BUCKETS
> ROW FORMAT SERDE
>   'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
>   
> 'file:/Users/ekoifman/IdeaProjects/hive/ql/target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands-1518814098564/warehouse/t'
> TBLPROPERTIES (
>   'transactional'='true',
>   'transactional_properties'='default',
>   'transient_lastDdlTime'='1518814111')
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21923) Vectorized MapJoin may miss results when only the join key is selected



 [ 
https://issues.apache.org/jira/browse/HIVE-21923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zoltan Haindrich updated HIVE-21923:

Summary: Vectorized MapJoin may miss results when only the join key is 
selected  (was: Disabling n-way joins caused some resultset changes)

> Vectorized MapJoin may miss results when only the join key is selected
> --
>
> Key: HIVE-21923
> URL: https://issues.apache.org/jira/browse/HIVE-21923
> Project: Hive
>  Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21923.01.patch
>
>
> HIVE-21189 have introduced some resultset changes
> in ql/src/test/results/clientpositive/llap/hybridgrace_hashjoin_2.q.out
> https://github.com/apache/hive/commit/5799398450c17d06e8ef144ce835a8524f5abec9#diff-56b3ab96b6c90fdbebe2c4f84e8595afL500



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18735) Create table like loses transactional attribute



[ 
https://issues.apache.org/jira/browse/HIVE-18735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873365#comment-16873365
 ] 

Hive QA commented on HIVE-18735:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
55s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
58s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
32s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
15s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 38s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17750/dev-support/hive-personality.sh
 |
| git revision | master / 967a1cc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql hbase-handler U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17750/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Create table like loses transactional attribute
> ---
>
> Key: HIVE-18735
> URL: https://issues.apache.org/jira/browse/HIVE-18735
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 2.0.0
>Reporter: Eugene Koifman
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-18735.01.patch, HIVE-18735.02.patch, 
> HIVE-18735.03.patch, HIVE-18735.04.patch
>
>
> {noformat}
> create table T1(a int, b int) clustered by (a) into 2 buckets stored as orc 
> TBLPROPERTIES ('transactional'='true')";
> create table T like T1;
> show create table T ;
> CREATE TABLE `T`(
>   `a` int,
>   `b` int)
> CLUSTERED BY (
>   a)
> INTO 2 BUCKETS
> ROW FORMAT SERDE
>   'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
>  
> 'file:/Users/ekoifman/IdeaProjects/hive/ql/target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands-1518813536099/warehouse/t'
> TBLPROPERTIES (
>   'transient_lastDdlTime'='1518813564')
> {noformat}
>

[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy



 [ 
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267667=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267667
 ]

ASF GitHub Bot logged work on HIVE-21886:
-

Author: ASF GitHub Bot
Created on: 26/Jun/19 14:09
Start Date: 26/Jun/19 14:09
Worklog Time Spent: 10m 
  Work Description: sankarh commented on pull request #688: HIVE-21886 : 
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297686895
 
 

 ##
 File path: 
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
 ##
 @@ -92,21 +93,46 @@ private Scenario 
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
 }
   }
 
-  // return true, if event needs to be dumped, else return false.
-  private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+  private boolean isSetForBootstrapByReplaceHandler(Context withinContext, 
String tblName) {
+return (withinContext.oldReplScope != null)
+&& 
!(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))
+&& (ReplUtils.tableIncludedInReplScope(withinContext.replScope, 
tblName));
+  }
+
+  // Return true, if event needs to be dumped, else return false.
+  private boolean handleForTableLevelReplication(Context withinContext, String 
tblName) {
+// For alter, if the table does not satisfy the new policy then ignore the 
event. In case of replace
+// policy, if the table does not satisfy the old policy, then ignore the 
event. As, if the table satisfy the new
+// policy, then the table will be bootstrapped by replace handler anb if 
the table does not satisfy the new policy,
+// then anyways the table should be ignored. The event should be ignored 
if the table is already in the list of
+// tables to be bootstrapped.
+if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope, tblName)) 
{
+  // In case of replace, it will be dropped during load. In normal case 
just ignore the alter event.
+  LOG.debug("Table " + tblName + " does not satisfy the policy");
+  return false;
+} else if 
((withinContext.getTablesForBootstrap().contains(tblName.toLowerCase()))
 
 Review comment:
   I think, the else if flow is dead code as we skip this event from caller 
itself. Utils.shouldReplicate method returns false for this case. Pls check.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 267667)
Time Spent: 8.5h  (was: 8h 20m)

> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
>  Time Spent: 8.5h
>  Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace 
> policy is getting executed, it needs to take care of the policy inclusion in 
> both the policy for each table name.
>  1. Create a list of tables to be bootstrapped. 
>   2. During handling of alter table, if the alter type is rename 
>       1. If the old table name is present in the list of table to be 
> bootstrapped, remove it.
>        2. If the new table name, matches the new policy, add it to the list 
> of tables to be bootstrapped.
>   3. During handling of drop table
>        1. if the table is in the list of tables to be bootstrapped, then 
> remove it and ignore the event.
>   4. During other event handling 
>        1. if the table is there in the list of tables to be bootstrapped, 
> then ignore the event.
>  
> Rename handling during replace policy
>  # Old name not matching old policy – The old table will not be there at the 
> target cluster. The table will not be returned by get-all-table.
>  ## Old name is not matching new policy
>  ### New name not matching old policy
>   New name not matching new policy
>  * Ignore the event, no need to do anything.
>   New name matching new policy
>  * The table will be returned by get-all-table. Replace policy handler 
> will bootstrap

[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy



 [ 
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267660=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267660
 ]

ASF GitHub Bot logged work on HIVE-21886:
-

Author: ASF GitHub Bot
Created on: 26/Jun/19 14:03
Start Date: 26/Jun/19 14:03
Worklog Time Spent: 10m 
  Work Description: sankarh commented on pull request #688: HIVE-21886 : 
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297683504
 
 

 ##
 File path: 
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
 ##
 @@ -147,16 +197,19 @@ public void handle(Context withinContext) throws 
Exception {
 
 Table qlMdTableBefore = new Table(before);
 Set bootstrapTableList;
+ReplScope oldReplScope;
 if (Scenario.RENAME == scenario) {
   // Handling for table level replication is done in 
handleForTableLevelReplication method.
 
 Review comment:
   handleForTableLevelReplication doesn't handle rename flow. Incorrect comment.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 267660)
Time Spent: 8h 20m  (was: 8h 10m)

> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
>  Time Spent: 8h 20m
>  Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace 
> policy is getting executed, it needs to take care of the policy inclusion in 
> both the policy for each table name.
>  1. Create a list of tables to be bootstrapped. 
>   2. During handling of alter table, if the alter type is rename 
>       1. If the old table name is present in the list of table to be 
> bootstrapped, remove it.
>        2. If the new table name, matches the new policy, add it to the list 
> of tables to be bootstrapped.
>   3. During handling of drop table
>        1. if the table is in the list of tables to be bootstrapped, then 
> remove it and ignore the event.
>   4. During other event handling 
>        1. if the table is there in the list of tables to be bootstrapped, 
> then ignore the event.
>  
> Rename handling during replace policy
>  # Old name not matching old policy – The old table will not be there at the 
> target cluster. The table will not be returned by get-all-table.
>  ## Old name is not matching new policy
>  ### New name not matching old policy
>   New name not matching new policy
>  * Ignore the event, no need to do anything.
>   New name matching new policy
>  * The table will be returned by get-all-table. Replace policy handler 
> will bootstrap this table as its matching new policy and not matching old 
> policy.
>  * All the future events will be ignored as part of check added by 
> replace policy handling.
>  * All the event with old table name will anyways be ignored as the old 
> name is not matching the new policy.
>  ### New name matching old policy
>   New name not matching new policy
>  * As the new name is not matching the new policy, the table need not be 
> replicated.
>  * As the old name is not matching the new policy, the rename events will 
> be ignored.
>  * So nothing to be done for this scenario.
>   New name matching new policy
>  * As the new name is matching both old and new policy, replace handler 
> will not bootstrap the table.
>  * Add the table to the list of tables to be bootstrapped.
>  * Ignore all the events with new name.
>  * If there is a drop event for the table (with new name), then remove 
> the table from the the list of table to be bootstrapped.
>  * In case of rename event (double rename)
>  ** If the new name satisfies the table pattern, then add the new name to 
> the list of tables to be bootstrapped and remove the old name from the list 
> of tables to be bootstrapped.
>  ** If the new name does not satisfies then just removed the table name 
> from the list of tables to be bootstrapped.
>  ## Old name is matching new policy – As per replace policy handler, which 
>

[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy



 [ 
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267656=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267656
 ]

ASF GitHub Bot logged work on HIVE-21886:
-

Author: ASF GitHub Bot
Created on: 26/Jun/19 14:01
Start Date: 26/Jun/19 14:01
Worklog Time Spent: 10m 
  Work Description: sankarh commented on pull request #688: HIVE-21886 : 
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297682711
 
 

 ##
 File path: 
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
 ##
 @@ -118,13 +144,30 @@ private boolean handleForTableLevelReplication(Context 
withinContext) {
 
   // If the old table was in the list of tables to be bootstrapped which 
is a multi rename case, the old table
   // is removed from the list of tables to be bootstrapped and new one is 
added.
-  if (oldTableIsPresent) {
+  if (oldTableInBootstrapList) {
+withinContext.addToListOfTablesForBootstrap(newName);
+return false;
+  }
+
+  // All the subsequent events on this table newName are going to be 
skipped as the table is going to be
+  // bootstrapped by replace handler, so the rename is also skipped.
+  if (isSetForBootstrapByReplaceHandler(withinContext, newName)) {
+// This addition is not actually required. But is added just to be in 
safer side.
 withinContext.addToListOfTablesForBootstrap(newName);
+
+// If the old table satisfies the new policy and is not in the list of 
tables to be bootstrapped
+// (as per previous check based on oldTableIsPresent), then drop it.
+if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, 
oldName)) {
+  scenario = Scenario.DROP;
+  LOG.info("Table " + oldName + " will be dropped as the table " + 
newName + " will be bootstrapped.");
+  return true;
+}
+LOG.info("Table " + newName + " is set to be bootstrapped by replace 
policy handler.");
 return false;
   }
 
   // If both old and new table satisfies the filter and old table is 
present at target, then dump the rename event.
-  LOG.info("both old and new table satisfies the filter");
+  LOG.info("Both old " + oldName + " and new table " + newName + " 
satisfies the filter");
 
 Review comment:
   filter is not the right term. We can say, replication scope/policy. Pls 
check other places as well.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 267656)
Time Spent: 8h 10m  (was: 8h)

> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
>  Time Spent: 8h 10m
>  Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace 
> policy is getting executed, it needs to take care of the policy inclusion in 
> both the policy for each table name.
>  1. Create a list of tables to be bootstrapped. 
>   2. During handling of alter table, if the alter type is rename 
>       1. If the old table name is present in the list of table to be 
> bootstrapped, remove it.
>        2. If the new table name, matches the new policy, add it to the list 
> of tables to be bootstrapped.
>   3. During handling of drop table
>        1. if the table is in the list of tables to be bootstrapped, then 
> remove it and ignore the event.
>   4. During other event handling 
>        1. if the table is there in the list of tables to be bootstrapped, 
> then ignore the event.
>  
> Rename handling during replace policy
>  # Old name not matching old policy – The old table will not be there at the 
> target cluster. The table will not be returned by get-all-table.
>  ## Old name is not matching new policy
>  ### New name not matching old policy
>   New name not matching new policy
>  * Ignore the event, no need to do anything.
>   New name matching new policy
>  * The table will be returned by get-all-table. Replace policy handler 
> will bootstrap this

[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy



 [ 
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267654=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267654
 ]

ASF GitHub Bot logged work on HIVE-21886:
-

Author: ASF GitHub Bot
Created on: 26/Jun/19 13:59
Start Date: 26/Jun/19 13:59
Worklog Time Spent: 10m 
  Work Description: sankarh commented on pull request #688: HIVE-21886 : 
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297681357
 
 

 ##
 File path: 
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
 ##
 @@ -135,8 +178,15 @@ private boolean handleForTableLevelReplication(Context 
withinContext) {
 return false;
   }
 
+  // In case of replace policy, even if the old table matches the new 
policy, none of the events including create
+  // table will be replayed as the old table is set of bootstrap by 
replace handler. So rename event can be skipped.
 
 Review comment:
   There is nothing called replace handler. So, shall change the comment that 
"Old table is set for bootstrap due to replace policy."
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 267654)
Time Spent: 8h  (was: 7h 50m)

> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
>  Time Spent: 8h
>  Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace 
> policy is getting executed, it needs to take care of the policy inclusion in 
> both the policy for each table name.
>  1. Create a list of tables to be bootstrapped. 
>   2. During handling of alter table, if the alter type is rename 
>       1. If the old table name is present in the list of table to be 
> bootstrapped, remove it.
>        2. If the new table name, matches the new policy, add it to the list 
> of tables to be bootstrapped.
>   3. During handling of drop table
>        1. if the table is in the list of tables to be bootstrapped, then 
> remove it and ignore the event.
>   4. During other event handling 
>        1. if the table is there in the list of tables to be bootstrapped, 
> then ignore the event.
>  
> Rename handling during replace policy
>  # Old name not matching old policy – The old table will not be there at the 
> target cluster. The table will not be returned by get-all-table.
>  ## Old name is not matching new policy
>  ### New name not matching old policy
>   New name not matching new policy
>  * Ignore the event, no need to do anything.
>   New name matching new policy
>  * The table will be returned by get-all-table. Replace policy handler 
> will bootstrap this table as its matching new policy and not matching old 
> policy.
>  * All the future events will be ignored as part of check added by 
> replace policy handling.
>  * All the event with old table name will anyways be ignored as the old 
> name is not matching the new policy.
>  ### New name matching old policy
>   New name not matching new policy
>  * As the new name is not matching the new policy, the table need not be 
> replicated.
>  * As the old name is not matching the new policy, the rename events will 
> be ignored.
>  * So nothing to be done for this scenario.
>   New name matching new policy
>  * As the new name is matching both old and new policy, replace handler 
> will not bootstrap the table.
>  * Add the table to the list of tables to be bootstrapped.
>  * Ignore all the events with new name.
>  * If there is a drop event for the table (with new name), then remove 
> the table from the the list of table to be bootstrapped.
>  * In case of rename event (double rename)
>  ** If the new name satisfies the table pattern, then add the new name to 
> the list of tables to be bootstrapped and remove the old name from the list 
> of tables to be bootstrapped.
>  ** If the new name does not satisfies then just removed the table name 
> from the list of tables to be

[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy



 [ 
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267648=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267648
 ]

ASF GitHub Bot logged work on HIVE-21886:
-

Author: ASF GitHub Bot
Created on: 26/Jun/19 13:54
Start Date: 26/Jun/19 13:54
Worklog Time Spent: 10m 
  Work Description: sankarh commented on pull request #688: HIVE-21886 : 
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297678333
 
 

 ##
 File path: 
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
 ##
 @@ -92,16 +93,40 @@ private Scenario 
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
 }
   }
 
-  // return true, if event needs to be dumped, else return false.
-  private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+  private boolean isSetForBootstrapByReplaceHandler(Context withinContext, 
String tblName) {
+return (withinContext.oldReplScope != null)
+&& 
!(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))
+&& (ReplUtils.tableIncludedInReplScope(withinContext.replScope, 
tblName));
+  }
 
-if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
-  // If the table is renamed after being added to the list of tables to be 
bootstrapped, then remove it from the
-  // list of tables to be bootstrapped.
-  boolean oldTableIsPresent = 
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
+  // Return true, if event needs to be dumped, else return false.
+  private boolean handleForTableLevelReplication(Context withinContext, String 
tblName) {
+// For alter, if the table does not satisfy the new policy then ignore the 
event. In case of replace
+// policy, if the table does not satisfy the old policy, then ignore the 
event. As, if the table satisfy the new
+// policy, then the table will be bootstrapped by replace handler anb if 
the table does not satisfy the new policy,
+// then anyways the table should be ignored.
+if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope, tblName)) 
{
+  // In case of replace, it will be dropped during load. In normal case 
just ignore the alter event.
+  LOG.debug("Table " + tblName + " does not satisfy the policy");
+  return false;
+} else if ((withinContext.oldReplScope != null)
 
 Review comment:
   As discussed, let's keep the checks.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 267648)
Time Spent: 7.5h  (was: 7h 20m)

> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
>  Time Spent: 7.5h
>  Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace 
> policy is getting executed, it needs to take care of the policy inclusion in 
> both the policy for each table name.
>  1. Create a list of tables to be bootstrapped. 
>   2. During handling of alter table, if the alter type is rename 
>       1. If the old table name is present in the list of table to be 
> bootstrapped, remove it.
>        2. If the new table name, matches the new policy, add it to the list 
> of tables to be bootstrapped.
>   3. During handling of drop table
>        1. if the table is in the list of tables to be bootstrapped, then 
> remove it and ignore the event.
>   4. During other event handling 
>        1. if the table is there in the list of tables to be bootstrapped, 
> then ignore the event.
>  
> Rename handling during replace policy
>  # Old name not matching old policy – The old table will not be there at the 
> target cluster. The table will not be returned by get-all-table.
>  ## Old name is not matching new policy
>  ### New name not matching old policy
>   New name not matching new policy
>  * Ignore the event, no need to do anything.
>   New name matching new

[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy



 [ 
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267650=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267650
 ]

ASF GitHub Bot logged work on HIVE-21886:
-

Author: ASF GitHub Bot
Created on: 26/Jun/19 13:54
Start Date: 26/Jun/19 13:54
Worklog Time Spent: 10m 
  Work Description: sankarh commented on pull request #688: HIVE-21886 : 
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297678731
 
 

 ##
 File path: 
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
 ##
 @@ -135,8 +173,15 @@ private boolean handleForTableLevelReplication(Context 
withinContext) {
 return false;
 
 Review comment:
   As discussed, let's keep the checks.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 267650)
Time Spent: 7h 50m  (was: 7h 40m)

> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
>  Time Spent: 7h 50m
>  Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace 
> policy is getting executed, it needs to take care of the policy inclusion in 
> both the policy for each table name.
>  1. Create a list of tables to be bootstrapped. 
>   2. During handling of alter table, if the alter type is rename 
>       1. If the old table name is present in the list of table to be 
> bootstrapped, remove it.
>        2. If the new table name, matches the new policy, add it to the list 
> of tables to be bootstrapped.
>   3. During handling of drop table
>        1. if the table is in the list of tables to be bootstrapped, then 
> remove it and ignore the event.
>   4. During other event handling 
>        1. if the table is there in the list of tables to be bootstrapped, 
> then ignore the event.
>  
> Rename handling during replace policy
>  # Old name not matching old policy – The old table will not be there at the 
> target cluster. The table will not be returned by get-all-table.
>  ## Old name is not matching new policy
>  ### New name not matching old policy
>   New name not matching new policy
>  * Ignore the event, no need to do anything.
>   New name matching new policy
>  * The table will be returned by get-all-table. Replace policy handler 
> will bootstrap this table as its matching new policy and not matching old 
> policy.
>  * All the future events will be ignored as part of check added by 
> replace policy handling.
>  * All the event with old table name will anyways be ignored as the old 
> name is not matching the new policy.
>  ### New name matching old policy
>   New name not matching new policy
>  * As the new name is not matching the new policy, the table need not be 
> replicated.
>  * As the old name is not matching the new policy, the rename events will 
> be ignored.
>  * So nothing to be done for this scenario.
>   New name matching new policy
>  * As the new name is matching both old and new policy, replace handler 
> will not bootstrap the table.
>  * Add the table to the list of tables to be bootstrapped.
>  * Ignore all the events with new name.
>  * If there is a drop event for the table (with new name), then remove 
> the table from the the list of table to be bootstrapped.
>  * In case of rename event (double rename)
>  ** If the new name satisfies the table pattern, then add the new name to 
> the list of tables to be bootstrapped and remove the old name from the list 
> of tables to be bootstrapped.
>  ** If the new name does not satisfies then just removed the table name 
> from the list of tables to be bootstrapped.
>  ## Old name is matching new policy – As per replace policy handler, which 
> checks based on old table, the table should be bootstrapped and event should 
> be ignored. But rename handler should decide based on new name.The old table 
> name will not be returned by get-all-table, so replace handler will not d 
> anything for the

[jira] [Commented] (HIVE-21407) Parquet predicate pushdown is not working correctly for char column types



[ 
https://issues.apache.org/jira/browse/HIVE-21407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873349#comment-16873349
 ] 

Zoltan Haindrich commented on HIVE-21407:
-

The format may store it however it want that value; but, it is Hive's type 
systems responsibility to handle a "rightly" type constant ; for example a 
Char(n) type if that's what it is - you may ask for the expanded or the 
non-expanded form...but when you do I think you have to consider what contracts 
parquet is comforming to.

To keep this short; I think the following example might help:
{code}
select 'a' = 'a ', cast('a' as char(3)) = 'a ', cast('a ' as char(3)) = 'a';
{code}

Would it be possible that parquet stores 'a ' somehow? because if that's 
possible then neither 'a' nor 'a  ' will match that...

> Parquet predicate pushdown is not working correctly for char column types
> -
>
> Key: HIVE-21407
> URL: https://issues.apache.org/jira/browse/HIVE-21407
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Major
> Attachments: HIVE-21407.2.patch, HIVE-21407.3.patch, HIVE-21407.patch
>
>
> If the 'hive.optimize.index.filter' parameter is false, the filter predicate 
> is not pushed to parquet, so the filtering only happens within Hive. If the 
> parameter is true, the filter is pushed to parquet, but for a char type, the 
> value which is pushed to Parquet will be padded with spaces:
> {noformat}
>   @Override
>   public void setValue(String val, int len) {
> super.setValue(HiveBaseChar.getPaddedValue(val, len), -1);
>   }
> {noformat} 
> So if we have a char(10) column which contains the value "apple" and the 
> where condition looks like 'where c='apple'', the value pushed to Paquet will 
> be 'apple' followed by 5 spaces. But the stored values are not padded, so no 
> rows will be returned from Parquet.
> How to reproduce:
> {noformat}
> $ create table ppd (c char(10), v varchar(10), i int) stored as parquet;
> $ insert into ppd values ('apple', 'bee', 1),('apple', 'tree', 2),('hello', 
> 'world', 1),('hello','vilag',3);
> $ set hive.optimize.ppd.storage=true;
> $ set hive.vectorized.execution.enabled=true;
> $ set hive.vectorized.execution.enabled=false;
> $ set hive.optimize.ppd=true;
> $ set hive.optimize.index.filter=true;
> $ set hive.parquet.timestamp.skip.conversion=false;
> $ select * from ppd where c='apple';
> ++++
> | ppd.c  | ppd.v  | ppd.i  |
> ++++
> ++++
> $ set hive.optimize.index.filter=false; or set 
> hive.optimize.ppd.storage=false;
> $ select * from ppd where c='apple';
> +-+++
> |ppd.c| ppd.v  | ppd.i  |
> +-+++
> | apple   | bee| 1  |
> | apple   | tree   | 2  |
> +-+++
> {noformat}
> The issue surfaced after uploading the fix for 
> [HIVE-21327|https://issues.apache.org/jira/browse/HIVE-21327] was uploaded 
> upstream. Before the HIVE-21327 fix, setting the parameter 
> 'hive.parquet.timestamp.skip.conversion' to true in the parquet_ppd_char.q 
> test hid this issue.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy



 [ 
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267649=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267649
 ]

ASF GitHub Bot logged work on HIVE-21886:
-

Author: ASF GitHub Bot
Created on: 26/Jun/19 13:54
Start Date: 26/Jun/19 13:54
Worklog Time Spent: 10m 
  Work Description: sankarh commented on pull request #688: HIVE-21886 : 
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297678557
 
 

 ##
 File path: 
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
 ##
 @@ -135,8 +173,15 @@ private boolean handleForTableLevelReplication(Context 
withinContext) {
 return false;
   }
 
+  // In case of replace policy, even if the old table matches the new 
policy, none of the events including create
+  // table will be replayed as the old table is set of bootstrap by 
replace handler. So rename event can be skipped.
+  if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, 
oldName)) {
 
 Review comment:
   Ok
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 267649)
Time Spent: 7h 40m  (was: 7.5h)

> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
>  Time Spent: 7h 40m
>  Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace 
> policy is getting executed, it needs to take care of the policy inclusion in 
> both the policy for each table name.
>  1. Create a list of tables to be bootstrapped. 
>   2. During handling of alter table, if the alter type is rename 
>       1. If the old table name is present in the list of table to be 
> bootstrapped, remove it.
>        2. If the new table name, matches the new policy, add it to the list 
> of tables to be bootstrapped.
>   3. During handling of drop table
>        1. if the table is in the list of tables to be bootstrapped, then 
> remove it and ignore the event.
>   4. During other event handling 
>        1. if the table is there in the list of tables to be bootstrapped, 
> then ignore the event.
>  
> Rename handling during replace policy
>  # Old name not matching old policy – The old table will not be there at the 
> target cluster. The table will not be returned by get-all-table.
>  ## Old name is not matching new policy
>  ### New name not matching old policy
>   New name not matching new policy
>  * Ignore the event, no need to do anything.
>   New name matching new policy
>  * The table will be returned by get-all-table. Replace policy handler 
> will bootstrap this table as its matching new policy and not matching old 
> policy.
>  * All the future events will be ignored as part of check added by 
> replace policy handling.
>  * All the event with old table name will anyways be ignored as the old 
> name is not matching the new policy.
>  ### New name matching old policy
>   New name not matching new policy
>  * As the new name is not matching the new policy, the table need not be 
> replicated.
>  * As the old name is not matching the new policy, the rename events will 
> be ignored.
>  * So nothing to be done for this scenario.
>   New name matching new policy
>  * As the new name is matching both old and new policy, replace handler 
> will not bootstrap the table.
>  * Add the table to the list of tables to be bootstrapped.
>  * Ignore all the events with new name.
>  * If there is a drop event for the table (with new name), then remove 
> the table from the the list of table to be bootstrapped.
>  * In case of rename event (double rename)
>  ** If the new name satisfies the table pattern, then add the new name to 
> the list of tables to be bootstrapped and remove the old name from the list 
> of tables to be bootstrapped.
>  ** If the new name does not satisfies then just removed the table name 
> from the list of tables to be bootstrapped.
>  ## Old name is matching new

[jira] [Comment Edited] (HIVE-21407) Parquet predicate pushdown is not working correctly for char column types



[ 
https://issues.apache.org/jira/browse/HIVE-21407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873349#comment-16873349
 ] 

Zoltan Haindrich edited comment on HIVE-21407 at 6/26/19 1:54 PM:
--

The format may store it however it want that value; but, it is Hive's type 
systems responsibility to handle a "rightly" type constant ; for example a 
{{Char( n )}} type if that's what it is - you may ask for the expanded or the 
non-expanded form...but when you do I think you have to consider what contracts 
parquet is comforming to.

To keep this short; I think the following example might help:
{code}
select 'a' = 'a ', cast('a' as char(3)) = 'a ', cast('a ' as char(3)) = 'a';
{code}

Would it be possible that parquet stores 'a ' somehow? because if that's 
possible then neither 'a' nor 'a  ' will match that...


was (Author: kgyrtkirk):
The format may store it however it want that value; but, it is Hive's type 
systems responsibility to handle a "rightly" type constant ; for example a 
Char(n) type if that's what it is - you may ask for the expanded or the 
non-expanded form...but when you do I think you have to consider what contracts 
parquet is comforming to.

To keep this short; I think the following example might help:
{code}
select 'a' = 'a ', cast('a' as char(3)) = 'a ', cast('a ' as char(3)) = 'a';
{code}

Would it be possible that parquet stores 'a ' somehow? because if that's 
possible then neither 'a' nor 'a  ' will match that...

> Parquet predicate pushdown is not working correctly for char column types
> -
>
> Key: HIVE-21407
> URL: https://issues.apache.org/jira/browse/HIVE-21407
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Major
> Attachments: HIVE-21407.2.patch, HIVE-21407.3.patch, HIVE-21407.patch
>
>
> If the 'hive.optimize.index.filter' parameter is false, the filter predicate 
> is not pushed to parquet, so the filtering only happens within Hive. If the 
> parameter is true, the filter is pushed to parquet, but for a char type, the 
> value which is pushed to Parquet will be padded with spaces:
> {noformat}
>   @Override
>   public void setValue(String val, int len) {
> super.setValue(HiveBaseChar.getPaddedValue(val, len), -1);
>   }
> {noformat} 
> So if we have a char(10) column which contains the value "apple" and the 
> where condition looks like 'where c='apple'', the value pushed to Paquet will 
> be 'apple' followed by 5 spaces. But the stored values are not padded, so no 
> rows will be returned from Parquet.
> How to reproduce:
> {noformat}
> $ create table ppd (c char(10), v varchar(10), i int) stored as parquet;
> $ insert into ppd values ('apple', 'bee', 1),('apple', 'tree', 2),('hello', 
> 'world', 1),('hello','vilag',3);
> $ set hive.optimize.ppd.storage=true;
> $ set hive.vectorized.execution.enabled=true;
> $ set hive.vectorized.execution.enabled=false;
> $ set hive.optimize.ppd=true;
> $ set hive.optimize.index.filter=true;
> $ set hive.parquet.timestamp.skip.conversion=false;
> $ select * from ppd where c='apple';
> ++++
> | ppd.c  | ppd.v  | ppd.i  |
> ++++
> ++++
> $ set hive.optimize.index.filter=false; or set 
> hive.optimize.ppd.storage=false;
> $ select * from ppd where c='apple';
> +-+++
> |ppd.c| ppd.v  | ppd.i  |
> +-+++
> | apple   | bee| 1  |
> | apple   | tree   | 2  |
> +-+++
> {noformat}
> The issue surfaced after uploading the fix for 
> [HIVE-21327|https://issues.apache.org/jira/browse/HIVE-21327] was uploaded 
> upstream. Before the HIVE-21327 fix, setting the parameter 
> 'hive.parquet.timestamp.skip.conversion' to true in the parquet_ppd_char.q 
> test hid this issue.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy



 [ 
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267647=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267647
 ]

ASF GitHub Bot logged work on HIVE-21886:
-

Author: ASF GitHub Bot
Created on: 26/Jun/19 13:52
Start Date: 26/Jun/19 13:52
Worklog Time Spent: 10m 
  Work Description: sankarh commented on pull request #688: HIVE-21886 : 
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297677615
 
 

 ##
 File path: 
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
 ##
 @@ -92,16 +93,40 @@ private Scenario 
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
 }
   }
 
-  // return true, if event needs to be dumped, else return false.
-  private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+  private boolean isSetForBootstrapByReplaceHandler(Context withinContext, 
String tblName) {
+return (withinContext.oldReplScope != null)
+&& 
!(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))
+&& (ReplUtils.tableIncludedInReplScope(withinContext.replScope, 
tblName));
+  }
 
-if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
-  // If the table is renamed after being added to the list of tables to be 
bootstrapped, then remove it from the
-  // list of tables to be bootstrapped.
-  boolean oldTableIsPresent = 
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
+  // Return true, if event needs to be dumped, else return false.
+  private boolean handleForTableLevelReplication(Context withinContext, String 
tblName) {
 
 Review comment:
   I disagree. If renameHandler method is called within this method, then this 
name is fine. As per your argument, it handles only alter and truncate but 
gives the feel that it handles rename as well.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 267647)
Time Spent: 7h 20m  (was: 7h 10m)

> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
>  Time Spent: 7h 20m
>  Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace 
> policy is getting executed, it needs to take care of the policy inclusion in 
> both the policy for each table name.
>  1. Create a list of tables to be bootstrapped. 
>   2. During handling of alter table, if the alter type is rename 
>       1. If the old table name is present in the list of table to be 
> bootstrapped, remove it.
>        2. If the new table name, matches the new policy, add it to the list 
> of tables to be bootstrapped.
>   3. During handling of drop table
>        1. if the table is in the list of tables to be bootstrapped, then 
> remove it and ignore the event.
>   4. During other event handling 
>        1. if the table is there in the list of tables to be bootstrapped, 
> then ignore the event.
>  
> Rename handling during replace policy
>  # Old name not matching old policy – The old table will not be there at the 
> target cluster. The table will not be returned by get-all-table.
>  ## Old name is not matching new policy
>  ### New name not matching old policy
>   New name not matching new policy
>  * Ignore the event, no need to do anything.
>   New name matching new policy
>  * The table will be returned by get-all-table. Replace policy handler 
> will bootstrap this table as its matching new policy and not matching old 
> policy.
>  * All the future events will be ignored as part of check added by 
> replace policy handling.
>  * All the event with old table name will anyways be ignored as the old 
> name is not matching the new policy.
>  ### New name matching old policy
>   New name not matching new policy
>  * As the new name is not matching the new policy, the table need not be 
> replicated.
>

[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy



 [ 
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267645=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267645
 ]

ASF GitHub Bot logged work on HIVE-21886:
-

Author: ASF GitHub Bot
Created on: 26/Jun/19 13:48
Start Date: 26/Jun/19 13:48
Worklog Time Spent: 10m 
  Work Description: sankarh commented on pull request #688: HIVE-21886 : 
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297675249
 
 

 ##
 File path: 
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
 ##
 @@ -135,8 +173,15 @@ private boolean handleForTableLevelReplication(Context 
withinContext) {
 return false;
   }
 
+  // In case of replace policy, even if the old table matches the new 
policy, none of the events including create
+  // table will be replayed as the old table is set of bootstrap by 
replace handler. So rename event can be skipped.
+  if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, 
oldName)) {
 
 Review comment:
   ok
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 267645)
Time Spent: 7h  (was: 6h 50m)

> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
>  Time Spent: 7h
>  Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace 
> policy is getting executed, it needs to take care of the policy inclusion in 
> both the policy for each table name.
>  1. Create a list of tables to be bootstrapped. 
>   2. During handling of alter table, if the alter type is rename 
>       1. If the old table name is present in the list of table to be 
> bootstrapped, remove it.
>        2. If the new table name, matches the new policy, add it to the list 
> of tables to be bootstrapped.
>   3. During handling of drop table
>        1. if the table is in the list of tables to be bootstrapped, then 
> remove it and ignore the event.
>   4. During other event handling 
>        1. if the table is there in the list of tables to be bootstrapped, 
> then ignore the event.
>  
> Rename handling during replace policy
>  # Old name not matching old policy – The old table will not be there at the 
> target cluster. The table will not be returned by get-all-table.
>  ## Old name is not matching new policy
>  ### New name not matching old policy
>   New name not matching new policy
>  * Ignore the event, no need to do anything.
>   New name matching new policy
>  * The table will be returned by get-all-table. Replace policy handler 
> will bootstrap this table as its matching new policy and not matching old 
> policy.
>  * All the future events will be ignored as part of check added by 
> replace policy handling.
>  * All the event with old table name will anyways be ignored as the old 
> name is not matching the new policy.
>  ### New name matching old policy
>   New name not matching new policy
>  * As the new name is not matching the new policy, the table need not be 
> replicated.
>  * As the old name is not matching the new policy, the rename events will 
> be ignored.
>  * So nothing to be done for this scenario.
>   New name matching new policy
>  * As the new name is matching both old and new policy, replace handler 
> will not bootstrap the table.
>  * Add the table to the list of tables to be bootstrapped.
>  * Ignore all the events with new name.
>  * If there is a drop event for the table (with new name), then remove 
> the table from the the list of table to be bootstrapped.
>  * In case of rename event (double rename)
>  ** If the new name satisfies the table pattern, then add the new name to 
> the list of tables to be bootstrapped and remove the old name from the list 
> of tables to be bootstrapped.
>  ** If the new name does not satisfies then just removed the table name 
> from the list of tables to be bootstrapped.
>  ## Old name is matching new policy –

[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy



 [ 
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267646=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267646
 ]

ASF GitHub Bot logged work on HIVE-21886:
-

Author: ASF GitHub Bot
Created on: 26/Jun/19 13:48
Start Date: 26/Jun/19 13:48
Worklog Time Spent: 10m 
  Work Description: sankarh commented on pull request #688: HIVE-21886 : 
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297668771
 
 

 ##
 File path: 
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
 ##
 @@ -120,18 +121,18 @@ private boolean handleForTableLevelReplication(Context 
withinContext, String tbl
   }
 
   // Return true, if event needs to be dumped, else return false.
-  private boolean handleForTableLevelReplicationForRename(Context 
withinContext, String oldName, String newName) {
+  private boolean renameHandlerForTableLevelReplication(Context withinContext, 
String oldName, String newName) {
 // If the table is renamed after being added to the list of tables to be 
bootstrapped, then remove it from the
 // list of tables to be bootstrapped.
-boolean oldTableIsPresent = 
withinContext.removeFromListOfTablesForBootstrap(oldName);
+boolean oldTableInBootstrapList = 
withinContext.removeFromListOfTablesForBootstrap(oldName);
 
 ReplScope oldPolicy = withinContext.oldReplScope == null ? 
withinContext.replScope : withinContext.oldReplScope;
 if (ReplUtils.tableIncludedInReplScope(oldPolicy, oldName)) {
   // If old table satisfies the filter, but the new table does not, then 
the old table should be dropped.
   // This should be done, only if the old table is not in the list of 
tables to be bootstrapped which is a multi
   // rename case. In case of multi rename, only the first rename should do 
the drop.
   if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope, 
newName)) {
-if (oldTableIsPresent) {
+if (oldTableInBootstrapList) {
   // If the old table was present in the list of tables to be 
bootstrapped, then just ignore the event.
   return false;
 } else {
 
 Review comment:
   A -> B where A doesn't matches old policy but B matches old but not new 
policy. So, no-op and don't dump event as well. Now, B -> C where B matches old 
and new policy and C doesn't match new policy. In this case, we add drop event 
for B which is wrong as it is missing in bootstrap list. Pls check if this is 
possible.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 267646)
Time Spent: 7h 10m  (was: 7h)

> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
>  Time Spent: 7h 10m
>  Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace 
> policy is getting executed, it needs to take care of the policy inclusion in 
> both the policy for each table name.
>  1. Create a list of tables to be bootstrapped. 
>   2. During handling of alter table, if the alter type is rename 
>       1. If the old table name is present in the list of table to be 
> bootstrapped, remove it.
>        2. If the new table name, matches the new policy, add it to the list 
> of tables to be bootstrapped.
>   3. During handling of drop table
>        1. if the table is in the list of tables to be bootstrapped, then 
> remove it and ignore the event.
>   4. During other event handling 
>        1. if the table is there in the list of tables to be bootstrapped, 
> then ignore the event.
>  
> Rename handling during replace policy
>  # Old name not matching old policy – The old table will not be there at the 
> target cluster. The table will not be returned by get-all-table.
>  ## Old name is not matching new policy
>  ### New name not matching old policy
>   New name not matching new policy
>  * Ignore the event, no need to do anything.
>   New name matching new policy
>  * The table will be returned by

[jira] [Commented] (HIVE-21846) Create a thread in TezAM which periodically fetches LlapDaemon metrics

2019-06-26 Thread Peter Vary (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-21846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873341#comment-16873341
 ] 

Peter Vary commented on HIVE-21846:
---

+1

> Create a thread in TezAM which periodically fetches LlapDaemon metrics
> --
>
> Key: HIVE-21846
> URL: https://issues.apache.org/jira/browse/HIVE-21846
> Project: Hive
>  Issue Type: Sub-task
>  Components: llap, Tez
>Reporter: Peter Vary
>Assignee: Antal Sinkovits
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-21846.01.patch, HIVE-21846.02.patch, 
> HIVE-21846.03.patch
>
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> LlapTaskSchedulerService should start a thread which periodically fetches the 
> LlapDaemon metrics and stores them in the NodeInfo object.
> This should be just the first implementation - later we should find a way 
> where we do not need NxM requests between N TezAM and M LlapDaemon



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy



 [ 
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267633=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267633
 ]

ASF GitHub Bot logged work on HIVE-21886:
-

Author: ASF GitHub Bot
Created on: 26/Jun/19 13:36
Start Date: 26/Jun/19 13:36
Worklog Time Spent: 10m 
  Work Description: sankarh commented on pull request #688: HIVE-21886 : 
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297668771
 
 

 ##
 File path: 
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
 ##
 @@ -120,18 +121,18 @@ private boolean handleForTableLevelReplication(Context 
withinContext, String tbl
   }
 
   // Return true, if event needs to be dumped, else return false.
-  private boolean handleForTableLevelReplicationForRename(Context 
withinContext, String oldName, String newName) {
+  private boolean renameHandlerForTableLevelReplication(Context withinContext, 
String oldName, String newName) {
 // If the table is renamed after being added to the list of tables to be 
bootstrapped, then remove it from the
 // list of tables to be bootstrapped.
-boolean oldTableIsPresent = 
withinContext.removeFromListOfTablesForBootstrap(oldName);
+boolean oldTableInBootstrapList = 
withinContext.removeFromListOfTablesForBootstrap(oldName);
 
 ReplScope oldPolicy = withinContext.oldReplScope == null ? 
withinContext.replScope : withinContext.oldReplScope;
 if (ReplUtils.tableIncludedInReplScope(oldPolicy, oldName)) {
   // If old table satisfies the filter, but the new table does not, then 
the old table should be dropped.
   // This should be done, only if the old table is not in the list of 
tables to be bootstrapped which is a multi
   // rename case. In case of multi rename, only the first rename should do 
the drop.
   if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope, 
newName)) {
-if (oldTableIsPresent) {
+if (oldTableInBootstrapList) {
   // If the old table was present in the list of tables to be 
bootstrapped, then just ignore the event.
   return false;
 } else {
 
 Review comment:
   A -> B where A doesn't matches old policy but B matches old but not new 
policy. So, no-op and don't dump event as well. Now, B -> C where B matches old 
policy and C doesn't match new policy. In this case, we add drop event for B 
which is wrong as it is missing in bootstrap list. Pls check if this is 
possible.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 267633)
Time Spent: 6h 50m  (was: 6h 40m)

> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
>  Time Spent: 6h 50m
>  Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace 
> policy is getting executed, it needs to take care of the policy inclusion in 
> both the policy for each table name.
>  1. Create a list of tables to be bootstrapped. 
>   2. During handling of alter table, if the alter type is rename 
>       1. If the old table name is present in the list of table to be 
> bootstrapped, remove it.
>        2. If the new table name, matches the new policy, add it to the list 
> of tables to be bootstrapped.
>   3. During handling of drop table
>        1. if the table is in the list of tables to be bootstrapped, then 
> remove it and ignore the event.
>   4. During other event handling 
>        1. if the table is there in the list of tables to be bootstrapped, 
> then ignore the event.
>  
> Rename handling during replace policy
>  # Old name not matching old policy – The old table will not be there at the 
> target cluster. The table will not be returned by get-all-table.
>  ## Old name is not matching new policy
>  ### New name not matching old policy
>   New name not matching new policy
>  * Ignore the event, no need to do anything.
>   New name matching new policy
>  * The table will be returned by

[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy



 [ 
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267632=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267632
 ]

ASF GitHub Bot logged work on HIVE-21886:
-

Author: ASF GitHub Bot
Created on: 26/Jun/19 13:34
Start Date: 26/Jun/19 13:34
Worklog Time Spent: 10m 
  Work Description: maheshk114 commented on pull request #688: HIVE-21886 : 
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297668081
 
 

 ##
 File path: 
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
 ##
 @@ -92,16 +93,40 @@ private Scenario 
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
 }
   }
 
-  // return true, if event needs to be dumped, else return false.
-  private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+  private boolean isSetForBootstrapByReplaceHandler(Context withinContext, 
String tblName) {
+return (withinContext.oldReplScope != null)
+&& 
!(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))
+&& (ReplUtils.tableIncludedInReplScope(withinContext.replScope, 
tblName));
+  }
 
-if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
-  // If the table is renamed after being added to the list of tables to be 
bootstrapped, then remove it from the
-  // list of tables to be bootstrapped.
-  boolean oldTableIsPresent = 
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
+  // Return true, if event needs to be dumped, else return false.
+  private boolean handleForTableLevelReplication(Context withinContext, String 
tblName) {
 
 Review comment:
   i think the name looks fine ..adding default does not make difference.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 267632)
Time Spent: 6h 40m  (was: 6.5h)

> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
>  Time Spent: 6h 40m
>  Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace 
> policy is getting executed, it needs to take care of the policy inclusion in 
> both the policy for each table name.
>  1. Create a list of tables to be bootstrapped. 
>   2. During handling of alter table, if the alter type is rename 
>       1. If the old table name is present in the list of table to be 
> bootstrapped, remove it.
>        2. If the new table name, matches the new policy, add it to the list 
> of tables to be bootstrapped.
>   3. During handling of drop table
>        1. if the table is in the list of tables to be bootstrapped, then 
> remove it and ignore the event.
>   4. During other event handling 
>        1. if the table is there in the list of tables to be bootstrapped, 
> then ignore the event.
>  
> Rename handling during replace policy
>  # Old name not matching old policy – The old table will not be there at the 
> target cluster. The table will not be returned by get-all-table.
>  ## Old name is not matching new policy
>  ### New name not matching old policy
>   New name not matching new policy
>  * Ignore the event, no need to do anything.
>   New name matching new policy
>  * The table will be returned by get-all-table. Replace policy handler 
> will bootstrap this table as its matching new policy and not matching old 
> policy.
>  * All the future events will be ignored as part of check added by 
> replace policy handling.
>  * All the event with old table name will anyways be ignored as the old 
> name is not matching the new policy.
>  ### New name matching old policy
>   New name not matching new policy
>  * As the new name is not matching the new policy, the table need not be 
> replicated.
>  * As the old name is not matching the new policy, the rename events will 
> be ignored.
>  * So nothing to be done for

[jira] [Commented] (HIVE-21846) Create a thread in TezAM which periodically fetches LlapDaemon metrics