[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[ https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=446191=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446191 ] ASF GitHub Bot logged work on HIVE-21764: - Author: ASF GitHub Bot Created on: 16/Jun/20 00:24 Start Date: 16/Jun/20 00:24 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #679: URL: https://github.com/apache/hive/pull/679 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 446191) Time Spent: 7.5h (was: 7h 20m) > REPL DUMP should detect and bootstrap any rename table events where old table > was excluded but renamed table is included. > - > > Key: HIVE-21764 > URL: https://issues.apache.org/jira/browse/HIVE-21764 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Sankar Hariappan >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch, > HIVE-21764.03.patch > > Time Spent: 7.5h > Remaining Estimate: 0h > > REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular > expression + inclusion/exclusion list. So, in case of rename table event, the > event will be ignored if old table doesn't match the pattern but the new > table should be bootstrapped. REPL DUMP should have a mechanism to detect > such tables and automatically bootstrap with incremental replication.Also, if > renamed table is excluded from replication policy, then need to drop the old > table at target as well. > There are 4 scenarios that needs to be handled. > # Both new name and old name satisfies the table name pattern filter. > ## No need to do anything. The incremental event for rename should take care > of the replication. > # Both the names does not satisfy the table name pattern filter. > ## Both the names are not in the scope of the policy and thus nothing needs > to be done. > # New name satisfies the pattern but the old name does not. > ## The table will not be present at the target. > ## Rename event handler for dump should detect this case and add the new > table name to the list of table for bootstrap. > ## All the events related to the table (new name) should be ignored. > ## If there is a drop event for the table (with new name), then remove the > table from the list of tables to be bootstrapped. > ## In case of rename (double rename) > ### If the new name satisfies the table pattern, then add the new name to > the list of tables to be bootstrapped and remove the old name from the list > of tables to be bootstrapped. > ### If the new name does not satisfies then just removed the table name from > the list of tables to be bootstrapped. > # New name does not satisfies the pattern but the old name satisfies. > ## Change the rename event to a drop event. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[ https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=442908=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-442908 ] ASF GitHub Bot logged work on HIVE-21764: - Author: ASF GitHub Bot Created on: 09/Jun/20 16:11 Start Date: 09/Jun/20 16:11 Worklog Time Spent: 10m Work Description: github-actions[bot] commented on pull request #679: URL: https://github.com/apache/hive/pull/679#issuecomment-640955813 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Feel free to reach out on the d...@hive.apache.org list if the patch is in need of reviews. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 442908) Time Spent: 7h 20m (was: 7h 10m) > REPL DUMP should detect and bootstrap any rename table events where old table > was excluded but renamed table is included. > - > > Key: HIVE-21764 > URL: https://issues.apache.org/jira/browse/HIVE-21764 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Sankar Hariappan >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch, > HIVE-21764.03.patch > > Time Spent: 7h 20m > Remaining Estimate: 0h > > REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular > expression + inclusion/exclusion list. So, in case of rename table event, the > event will be ignored if old table doesn't match the pattern but the new > table should be bootstrapped. REPL DUMP should have a mechanism to detect > such tables and automatically bootstrap with incremental replication.Also, if > renamed table is excluded from replication policy, then need to drop the old > table at target as well. > There are 4 scenarios that needs to be handled. > # Both new name and old name satisfies the table name pattern filter. > ## No need to do anything. The incremental event for rename should take care > of the replication. > # Both the names does not satisfy the table name pattern filter. > ## Both the names are not in the scope of the policy and thus nothing needs > to be done. > # New name satisfies the pattern but the old name does not. > ## The table will not be present at the target. > ## Rename event handler for dump should detect this case and add the new > table name to the list of table for bootstrap. > ## All the events related to the table (new name) should be ignored. > ## If there is a drop event for the table (with new name), then remove the > table from the list of tables to be bootstrapped. > ## In case of rename (double rename) > ### If the new name satisfies the table pattern, then add the new name to > the list of tables to be bootstrapped and remove the old name from the list > of tables to be bootstrapped. > ### If the new name does not satisfies then just removed the table name from > the list of tables to be bootstrapped. > # New name does not satisfies the pattern but the old name satisfies. > ## Change the rename event to a drop event. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264690=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264690
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 14:47
Start Date: 21/Jun/19 14:47
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296268041
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/event/filters/ReplEventFilter.java
##
@@ -30,12 +31,22 @@ public ReplEventFilter(final ReplScope replScope) {
this.replScope = replScope;
}
+ // Return false if all the tables are included, as bootstrap during rename
is done only in case filter set for tables.
+ boolean isAlterDuringTableLevelReplication(final NotificationEvent event) {
Review comment:
This is not alter table handler. It is ReplEventFilter.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264690)
Time Spent: 7h 10m (was: 7h)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 7h 10m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264688=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264688
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 14:43
Start Date: 21/Jun/19 14:43
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296266307
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/event/filters/ReplEventFilter.java
##
@@ -33,7 +33,7 @@ public ReplEventFilter(final ReplScope replScope) {
// Return false if all the tables are included, as bootstrap during rename
is done only in case filter set for tables.
boolean isAlterDuringTableLevelReplication(final NotificationEvent event) {
-if (replScope.includeAllTables() ||
!replScope.getDbNamePattern().matcher(event.getDbName()).matches()) {
+if (!replScope.dbIncludedInReplScope(event.getDbName())) {
return false;
}
return event.getEventType().equals(MessageBuilder.ALTER_TABLE_EVENT);
Review comment:
Can combine to one line as replScope.dbIncludedInReplScope() &&
event.getEventType().equals().
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264688)
Time Spent: 7h (was: 6h 50m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 7h
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264681=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264681
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 14:28
Start Date: 21/Jun/19 14:28
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296259733
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -82,14 +92,71 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
+ // return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext) {
+String oldName = before.getTableName();
+String newName = after.getTableName();
+
+if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
+ // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
+ // list of tables to be bootstrapped.
+ boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
Review comment:
i think new Jira is required for this ..willl create one ..thus scenario
needs some more thinking
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264681)
Time Spent: 6h 50m (was: 6h 40m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 6h 50m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264680=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264680
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 14:28
Start Date: 21/Jun/19 14:28
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296259435
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/event/filters/ReplEventFilter.java
##
@@ -30,12 +31,22 @@ public ReplEventFilter(final ReplScope replScope) {
this.replScope = replScope;
}
+ // Return false if all the tables are included, as bootstrap during rename
is done only in case filter set for tables.
+ boolean isAlterDuringTableLevelReplication(final NotificationEvent event) {
+if (replScope.includeAllTables() ||
!replScope.getDbNamePattern().matcher(event.getDbName()).matches()) {
Review comment:
shall use dbIncludedInReplScope
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264680)
Time Spent: 6h 40m (was: 6.5h)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 6h 40m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264677=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264677
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 14:22
Start Date: 21/Jun/19 14:22
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296256649
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -82,14 +92,71 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
+ // return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext) {
+String oldName = before.getTableName();
+String newName = after.getTableName();
+
+if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
+ // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
+ // list of tables to be bootstrapped.
+ boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
Review comment:
I think, we look at beforeTableObject in case of shouldReplicate method
which allows this event as the before table was not external or ACID. However,
if at all this is a bug, it is not related to this patch. It is a general issue.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264677)
Time Spent: 6.5h (was: 6h 20m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 6.5h
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264676=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264676
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 14:21
Start Date: 21/Jun/19 14:21
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296256649
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -82,14 +92,71 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
+ // return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext) {
+String oldName = before.getTableName();
+String newName = after.getTableName();
+
+if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
+ // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
+ // list of tables to be bootstrapped.
+ boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
Review comment:
I think, we look at beforeTableObject in case of shouldReplicate method
which allows this event as the before table was not external or ACID.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264676)
Time Spent: 6h 20m (was: 6h 10m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 6h 20m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264675=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264675
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 14:20
Start Date: 21/Jun/19 14:20
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296256293
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/event/filters/ReplEventFilter.java
##
@@ -30,12 +31,22 @@ public ReplEventFilter(final ReplScope replScope) {
this.replScope = replScope;
}
+ // Return false if all the tables are included, as bootstrap during rename
is done only in case filter set for tables.
+ boolean isAlterDuringTableLevelReplication(final NotificationEvent event) {
+if (replScope.includeAllTables() ||
!replScope.getDbNamePattern().matcher(event.getDbName()).matches()) {
Review comment:
Well. my point is not about exposing dbNamePattern. It's more about
functionality related to ReplScope. So, the db matching is a functionality
which can be kept isolated within ReplScope. Why need to expose this
functionality outside? Also, having it inside ReplScope would make others to
use it in future instead of duplicating it in the caller.
Anyways, it's your call.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264675)
Time Spent: 6h 10m (was: 6h)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264674=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264674
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 14:18
Start Date: 21/Jun/19 14:18
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296255409
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/event/filters/ReplEventFilter.java
##
@@ -30,12 +31,22 @@ public ReplEventFilter(final ReplScope replScope) {
this.replScope = replScope;
}
+ // Return false if all the tables are included, as bootstrap during rename
is done only in case filter set for tables.
+ boolean isAlterDuringTableLevelReplication(final NotificationEvent event) {
+if (replScope.includeAllTables() ||
!replScope.getDbNamePattern().matcher(event.getDbName()).matches()) {
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264674)
Time Spent: 6h (was: 5h 50m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 6h
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264673=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264673
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 14:18
Start Date: 21/Jun/19 14:18
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296255346
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -82,14 +92,71 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
+ // return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext) {
+String oldName = before.getTableName();
+String newName = after.getTableName();
+
+if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
+ // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
+ // list of tables to be bootstrapped.
+ boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
+
+ // If old table satisfies the filter, but the new table does not, then
the old table should be dropped.
+ // This should be done, only if the old table is not in the list of
tables to be bootstrapped which is a multi
+ // rename case. In case of multi rename, only the first rename should do
the drop.
+ if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope,
newName)) {
+if (oldTableIsPresent) {
+ // If the old table was present in the list of tables to be
bootstrapped, then just ignore the event.
+ return false;
+} else {
+ scenario = Scenario.DROP;
+ LOG.info("Table " + oldName + " will be dropped as the table is
renamed to " + newName);
+ return true;
+}
+ }
+
+ // If the old table was in the list of tables to be bootstrapped which
is a multi rename case, the old table
+ // is removed from the list of tables to be bootstrapped and new one is
added.
+ if (oldTableIsPresent) {
+withinContext.addToListOfTablesForBootstrap(newName);
+return false;
+ }
+
+ // If both old and new table satisfies the filter and old table is
present at target, then dump the rename event.
+ LOG.info("both old and new table satisfies the filter");
+ return true;
+} else {
+ // if the old table does not satisfies the filter, but the new one
satisfies, then the new table should be
+ // added to the list of tables to be bootstrapped and don't dump the
event.
+ if (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
newName)) {
+LOG.info("Table " + newName + " is added for bootstrap " + " during
rename from " + oldName);
+withinContext.addToListOfTablesForBootstrap(newName);
+return false;
+ }
+
+ // if both old and new table does not satisfies the filter, then don't
dump the event.
+ LOG.info("both old and new table not satisfies the filter");
+ return false;
+}
+ }
+
@Override
public void handle(Context withinContext) throws Exception {
LOG.info("Processing#{} ALTER_TABLE message : {}", fromEventId(),
eventMessageAsJSON);
Table qlMdTableBefore = new Table(before);
+Set bootstrapTableList;
+if (Scenario.RENAME == scenario) {
+ // Handling for table level replication is done in
handleForTableLevelReplication method.
+ bootstrapTableList = null;
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264673)
Time Spent: 5h 50m (was: 5h 40m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication,
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264670=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264670
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 14:11
Start Date: 21/Jun/19 14:11
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296252578
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -82,14 +90,49 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
+ // return true, if event needs to be dumped, else return false.
+ private boolean handleRenameDuringTableLevelReplication(Context
withinContext) {
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264670)
Time Spent: 5h 40m (was: 5.5h)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264665=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264665
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 14:07
Start Date: 21/Jun/19 14:07
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296250742
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/event/filters/ReplEventFilter.java
##
@@ -30,12 +31,22 @@ public ReplEventFilter(final ReplScope replScope) {
this.replScope = replScope;
}
+ // Return false if all the tables are included, as bootstrap during rename
is done only in case filter set for tables.
+ boolean isAlterDuringTableLevelReplication(final NotificationEvent event) {
+if (replScope.includeAllTables() ||
!replScope.getDbNamePattern().matcher(event.getDbName()).matches()) {
Review comment:
if db name can be exposed ..this i don't think has any issue ..
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264665)
Time Spent: 5.5h (was: 5h 20m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264664=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264664
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 14:06
Start Date: 21/Jun/19 14:06
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296250437
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/event/filters/ReplEventFilter.java
##
@@ -30,12 +31,22 @@ public ReplEventFilter(final ReplScope replScope) {
this.replScope = replScope;
}
+ // Return false if all the tables are included, as bootstrap during rename
is done only in case filter set for tables.
+ boolean isAlterDuringTableLevelReplication(final NotificationEvent event) {
+if (replScope.includeAllTables() ||
!replScope.getDbNamePattern().matcher(event.getDbName()).matches()) {
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264664)
Time Spent: 5h 20m (was: 5h 10m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264662=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264662
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 14:05
Start Date: 21/Jun/19 14:05
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296250117
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/event/filters/ReplEventFilter.java
##
@@ -30,12 +31,22 @@ public ReplEventFilter(final ReplScope replScope) {
this.replScope = replScope;
}
+ // Return false if all the tables are included, as bootstrap during rename
is done only in case filter set for tables.
+ boolean isAlterDuringTableLevelReplication(final NotificationEvent event) {
Review comment:
its inside alter table handler class so not a problem :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264662)
Time Spent: 5h 10m (was: 5h)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264658=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264658
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 14:02
Start Date: 21/Jun/19 14:02
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296248646
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -82,14 +92,71 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
+ // return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext) {
+String oldName = before.getTableName();
+String newName = after.getTableName();
+
+if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
+ // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
+ // list of tables to be bootstrapped.
+ boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
Review comment:
bootstrap external or bootstrap acid is handled in shuldreplicate method
..the control should not reach till here
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264658)
Time Spent: 5h (was: 4h 50m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 5h
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264651=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264651
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 14:00
Start Date: 21/Jun/19 14:00
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296247839
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/event/filters/ReplEventFilter.java
##
@@ -30,12 +31,22 @@ public ReplEventFilter(final ReplScope replScope) {
this.replScope = replScope;
}
+ // Return false if all the tables are included, as bootstrap during rename
is done only in case filter set for tables.
+ boolean isAlterDuringTableLevelReplication(final NotificationEvent event) {
+if (replScope.includeAllTables() ||
!replScope.getDbNamePattern().matcher(event.getDbName()).matches()) {
Review comment:
i think db rename is not supported
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264651)
Time Spent: 4h 50m (was: 4h 40m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264649=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264649
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 13:59
Start Date: 21/Jun/19 13:59
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296247443
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -82,14 +92,71 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
+ // return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext) {
+String oldName = before.getTableName();
+String newName = after.getTableName();
+
+if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
+ // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
+ // list of tables to be bootstrapped.
+ boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
+
+ // If old table satisfies the filter, but the new table does not, then
the old table should be dropped.
+ // This should be done, only if the old table is not in the list of
tables to be bootstrapped which is a multi
+ // rename case. In case of multi rename, only the first rename should do
the drop.
+ if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope,
newName)) {
+if (oldTableIsPresent) {
+ // If the old table was present in the list of tables to be
bootstrapped, then just ignore the event.
+ return false;
+} else {
+ scenario = Scenario.DROP;
+ LOG.info("Table " + oldName + " will be dropped as the table is
renamed to " + newName);
+ return true;
+}
+ }
+
+ // If the old table was in the list of tables to be bootstrapped which
is a multi rename case, the old table
+ // is removed from the list of tables to be bootstrapped and new one is
added.
+ if (oldTableIsPresent) {
+withinContext.addToListOfTablesForBootstrap(newName);
+return false;
+ }
+
+ // If both old and new table satisfies the filter and old table is
present at target, then dump the rename event.
+ LOG.info("both old and new table satisfies the filter");
+ return true;
+} else {
+ // if the old table does not satisfies the filter, but the new one
satisfies, then the new table should be
+ // added to the list of tables to be bootstrapped and don't dump the
event.
+ if (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
newName)) {
+LOG.info("Table " + newName + " is added for bootstrap " + " during
rename from " + oldName);
+withinContext.addToListOfTablesForBootstrap(newName);
+return false;
+ }
+
+ // if both old and new table does not satisfies the filter, then don't
dump the event.
+ LOG.info("both old and new table not satisfies the filter");
+ return false;
+}
+ }
+
@Override
public void handle(Context withinContext) throws Exception {
LOG.info("Processing#{} ALTER_TABLE message : {}", fromEventId(),
eventMessageAsJSON);
Table qlMdTableBefore = new Table(before);
+Set bootstrapTableList;
+if (Scenario.RENAME == scenario) {
+ // Handling for table level replication is done in
handleForTableLevelReplication method.
+ bootstrapTableList = null;
Review comment:
replScope is for replace policy (old policy) ..its n not handled in this
patch
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264649)
Time Spent: 4h 40m (was: 4.5h)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264577=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264577
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 12:23
Start Date: 21/Jun/19 12:23
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296202122
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/event/filters/ReplEventFilter.java
##
@@ -30,12 +31,22 @@ public ReplEventFilter(final ReplScope replScope) {
this.replScope = replScope;
}
+ // Return false if all the tables are included, as bootstrap during rename
is done only in case filter set for tables.
+ boolean isAlterDuringTableLevelReplication(final NotificationEvent event) {
+if (replScope.includeAllTables() ||
!replScope.getDbNamePattern().matcher(event.getDbName()).matches()) {
Review comment:
replScope.includeAllTables() will always be false. If true, then it wouldn't
reach here as replScope.includedInReplScope will be true in the caller.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264577)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264575=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264575
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 12:23
Start Date: 21/Jun/19 12:23
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296212406
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -82,14 +92,71 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
+ // return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext) {
+String oldName = before.getTableName();
+String newName = after.getTableName();
+
+if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
+ // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
+ // list of tables to be bootstrapped.
+ boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
Review comment:
Let's say, a non-acid table is converted to external. Now, since, both names
match ReplScope, this event is dumped. Now, all the other events on this alter
table would be skipped as BOOTSTRAP_EXTERNAL config is true. Now, we replicate
the event and also, bootstrap the table which may fail. This is not related to
this patch but still want to check if it can happen. I noticed, we block change
table type only if STRICT_MANAGED=true.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264575)
Time Spent: 4h 20m (was: 4h 10m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264578=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264578
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 12:23
Start Date: 21/Jun/19 12:23
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296202255
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/event/filters/ReplEventFilter.java
##
@@ -30,12 +31,22 @@ public ReplEventFilter(final ReplScope replScope) {
this.replScope = replScope;
}
+ // Return false if all the tables are included, as bootstrap during rename
is done only in case filter set for tables.
+ boolean isAlterDuringTableLevelReplication(final NotificationEvent event) {
+if (replScope.includeAllTables() ||
!replScope.getDbNamePattern().matcher(event.getDbName()).matches()) {
Review comment:
Is it possible to rename to different DB altogether?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264578)
Time Spent: 4.5h (was: 4h 20m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264574=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264574
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 12:23
Start Date: 21/Jun/19 12:23
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296203142
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/event/filters/ReplEventFilter.java
##
@@ -30,12 +31,22 @@ public ReplEventFilter(final ReplScope replScope) {
this.replScope = replScope;
}
+ // Return false if all the tables are included, as bootstrap during rename
is done only in case filter set for tables.
+ boolean isAlterDuringTableLevelReplication(final NotificationEvent event) {
+if (replScope.includeAllTables() ||
!replScope.getDbNamePattern().matcher(event.getDbName()).matches()) {
Review comment:
Instead of exposing dbNamePattern outside ReplScope, can we add a method
that checks if Db matches the replScope. That seems cleaner.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264574)
Time Spent: 4h 20m (was: 4h 10m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264576=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264576
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 12:23
Start Date: 21/Jun/19 12:23
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296201139
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -82,14 +92,71 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
+ // return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext) {
+String oldName = before.getTableName();
+String newName = after.getTableName();
+
+if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
+ // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
+ // list of tables to be bootstrapped.
+ boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
+
+ // If old table satisfies the filter, but the new table does not, then
the old table should be dropped.
+ // This should be done, only if the old table is not in the list of
tables to be bootstrapped which is a multi
+ // rename case. In case of multi rename, only the first rename should do
the drop.
+ if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope,
newName)) {
+if (oldTableIsPresent) {
+ // If the old table was present in the list of tables to be
bootstrapped, then just ignore the event.
+ return false;
+} else {
+ scenario = Scenario.DROP;
+ LOG.info("Table " + oldName + " will be dropped as the table is
renamed to " + newName);
+ return true;
+}
+ }
+
+ // If the old table was in the list of tables to be bootstrapped which
is a multi rename case, the old table
+ // is removed from the list of tables to be bootstrapped and new one is
added.
+ if (oldTableIsPresent) {
+withinContext.addToListOfTablesForBootstrap(newName);
+return false;
+ }
+
+ // If both old and new table satisfies the filter and old table is
present at target, then dump the rename event.
+ LOG.info("both old and new table satisfies the filter");
+ return true;
+} else {
+ // if the old table does not satisfies the filter, but the new one
satisfies, then the new table should be
+ // added to the list of tables to be bootstrapped and don't dump the
event.
+ if (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
newName)) {
+LOG.info("Table " + newName + " is added for bootstrap " + " during
rename from " + oldName);
+withinContext.addToListOfTablesForBootstrap(newName);
+return false;
+ }
+
+ // if both old and new table does not satisfies the filter, then don't
dump the event.
+ LOG.info("both old and new table not satisfies the filter");
+ return false;
+}
+ }
+
@Override
public void handle(Context withinContext) throws Exception {
LOG.info("Processing#{} ALTER_TABLE message : {}", fromEventId(),
eventMessageAsJSON);
Table qlMdTableBefore = new Table(before);
+Set bootstrapTableList;
+if (Scenario.RENAME == scenario) {
+ // Handling for table level replication is done in
handleForTableLevelReplication method.
+ bootstrapTableList = null;
Review comment:
If bootstrapTableList is passed as null, then if old table doesn't match the
replScope, then this event will be skipped before reaching
handleForTableLevelReplication.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264576)
Time Spent: 4h 20m (was: 4h 10m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
>
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264545=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264545
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 11:22
Start Date: 21/Jun/19 11:22
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296197082
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/Utils.java
##
@@ -214,29 +215,35 @@ public static boolean shouldReplicate(ReplicationSpec
replicationSpec, Table tab
return false;
}
-// Skip dumping events related to ACID tables if bootstrap is enabled
on it.
-// Also, skip if current table is included only in new policy but not
in old policy.
-if (isEventDump) {
- return
!hiveConf.getBoolVar(HiveConf.ConfVars.REPL_BOOTSTRAP_ACID_TABLES)
- && ReplUtils.tableIncludedInReplScope(oldReplScope,
tableHandle.getTableName());
+// Skip dumping events related to ACID tables if bootstrap is enabled
for ACID tables.
+if (isEventDump &&
hiveConf.getBoolVar(HiveConf.ConfVars.REPL_BOOTSTRAP_ACID_TABLES)) {
+ return false;
}
}
- // If replication policy is replaced with new included/excluded tables
list, then events
- // corresponding to tables which are not included in old policy but
included in new policy
- // should be skipped. Those tables would be bootstrapped along with the
current incremental
- // replication dump.
- // Note: If any event dump reaches here, it means, table is included in
new replication policy.
- if (isEventDump && !ReplUtils.tableIncludedInReplScope(oldReplScope,
tableHandle.getTableName())) {
-return false;
+ // Tables which are selected for bootstrap should be skipped. Those
tables would be bootstrapped
+ // along with the current incremental replication dump and thus no need
to dump events for them.
+ // Note: If any event (other than alter table with table level
replication) dump reaches here, it means, table is
+ // included in new replication policy.
+ if (isEventDump) {
+// If replication policy is replaced with new included/excluded tables
list, then events
+// corresponding to tables which are not included in old policy but
included in new policy
+// should be skipped.
+if (!ReplUtils.tableIncludedInReplScope(oldReplScope,
tableHandle.getTableName())) {
+ return false;
+}
+
+// Tables in the list of tables to be bootstrapped should be skipped.
Review comment:
I meant, "Events are skipped not tables."
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264545)
Time Spent: 4h 10m (was: 4h)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264542=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264542
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 11:20
Start Date: 21/Jun/19 11:20
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296196665
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/event/filters/ReplEventFilter.java
##
@@ -30,12 +31,22 @@ public ReplEventFilter(final ReplScope replScope) {
this.replScope = replScope;
}
+ // Return false if all the tables are included, as bootstrap during rename
is done only in case filter set for tables.
+ boolean isAlterDuringTableLevelReplication(final NotificationEvent event) {
Review comment:
ok... shall change to isAlterTableEvent. We have alter event for partitions
as well. :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264542)
Time Spent: 3h 50m (was: 3h 40m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264543=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264543
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 11:21
Start Date: 21/Jun/19 11:21
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296196726
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -101,6 +144,14 @@ public void handle(Context withinContext) throws
Exception {
}
}
+// If the tables are filtered based on name, then needs to handle the
rename scenarios.
+if (withinContext.replScope != null &&
!withinContext.replScope.includeAllTables()) {
+ if (!handleRenameDuringTableLevelReplication(withinContext)) {
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264543)
Time Spent: 4h (was: 3h 50m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264540=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264540
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 11:18
Start Date: 21/Jun/19 11:18
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296196167
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -82,14 +90,49 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
+ // return true, if event needs to be dumped, else return false.
+ private boolean handleRenameDuringTableLevelReplication(Context
withinContext) {
Review comment:
It says, handleRename but it actually handles all alter events. So, name can
be relevant to that.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264540)
Time Spent: 3.5h (was: 3h 20m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264541=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264541
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 11:19
Start Date: 21/Jun/19 11:19
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296196222
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/event/filters/ReplEventFilter.java
##
@@ -30,12 +31,22 @@ public ReplEventFilter(final ReplScope replScope) {
this.replScope = replScope;
}
+ // Return false if all the tables are included, as bootstrap during rename
is done only in case filter set for tables.
+ boolean isAlterDuringTableLevelReplication(final NotificationEvent event) {
+if (replScope.includeAllTables() ||
!replScope.getDbNamePattern().matcher(event.getDbName()).matches()) {
+ return false;
+}
+return event.getEventType().equals(MessageBuilder.ALTER_TABLE_EVENT);
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264541)
Time Spent: 3h 40m (was: 3.5h)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264514=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264514
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 10:44
Start Date: 21/Jun/19 10:44
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296166878
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/Utils.java
##
@@ -214,29 +215,35 @@ public static boolean shouldReplicate(ReplicationSpec
replicationSpec, Table tab
return false;
}
-// Skip dumping events related to ACID tables if bootstrap is enabled
on it.
-// Also, skip if current table is included only in new policy but not
in old policy.
-if (isEventDump) {
- return
!hiveConf.getBoolVar(HiveConf.ConfVars.REPL_BOOTSTRAP_ACID_TABLES)
- && ReplUtils.tableIncludedInReplScope(oldReplScope,
tableHandle.getTableName());
+// Skip dumping events related to ACID tables if bootstrap is enabled
for ACID tables.
+if (isEventDump &&
hiveConf.getBoolVar(HiveConf.ConfVars.REPL_BOOTSTRAP_ACID_TABLES)) {
+ return false;
}
}
- // If replication policy is replaced with new included/excluded tables
list, then events
- // corresponding to tables which are not included in old policy but
included in new policy
- // should be skipped. Those tables would be bootstrapped along with the
current incremental
- // replication dump.
- // Note: If any event dump reaches here, it means, table is included in
new replication policy.
- if (isEventDump && !ReplUtils.tableIncludedInReplScope(oldReplScope,
tableHandle.getTableName())) {
-return false;
+ // Tables which are selected for bootstrap should be skipped. Those
tables would be bootstrapped
+ // along with the current incremental replication dump and thus no need
to dump events for them.
+ // Note: If any event (other than alter table with table level
replication) dump reaches here, it means, table is
+ // included in new replication policy.
+ if (isEventDump) {
+// If replication policy is replaced with new included/excluded tables
list, then events
+// corresponding to tables which are not included in old policy but
included in new policy
+// should be skipped.
+if (!ReplUtils.tableIncludedInReplScope(oldReplScope,
tableHandle.getTableName())) {
+ return false;
+}
+
+// Tables in the list of tables to be bootstrapped should be skipped.
Review comment:
its already there ."Tables which are selected for bootstrap should be
skipped."
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264514)
Time Spent: 3h 10m (was: 3h)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264517=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264517
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 10:44
Start Date: 21/Jun/19 10:44
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296170253
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/Utils.java
##
@@ -214,29 +215,35 @@ public static boolean shouldReplicate(ReplicationSpec
replicationSpec, Table tab
return false;
}
-// Skip dumping events related to ACID tables if bootstrap is enabled
on it.
-// Also, skip if current table is included only in new policy but not
in old policy.
-if (isEventDump) {
- return
!hiveConf.getBoolVar(HiveConf.ConfVars.REPL_BOOTSTRAP_ACID_TABLES)
- && ReplUtils.tableIncludedInReplScope(oldReplScope,
tableHandle.getTableName());
+// Skip dumping events related to ACID tables if bootstrap is enabled
for ACID tables.
+if (isEventDump &&
hiveConf.getBoolVar(HiveConf.ConfVars.REPL_BOOTSTRAP_ACID_TABLES)) {
+ return false;
}
}
- // If replication policy is replaced with new included/excluded tables
list, then events
- // corresponding to tables which are not included in old policy but
included in new policy
- // should be skipped. Those tables would be bootstrapped along with the
current incremental
- // replication dump.
- // Note: If any event dump reaches here, it means, table is included in
new replication policy.
- if (isEventDump && !ReplUtils.tableIncludedInReplScope(oldReplScope,
tableHandle.getTableName())) {
-return false;
+ // Tables which are selected for bootstrap should be skipped. Those
tables would be bootstrapped
+ // along with the current incremental replication dump and thus no need
to dump events for them.
+ // Note: If any event (other than alter table with table level
replication) dump reaches here, it means, table is
+ // included in new replication policy.
+ if (isEventDump) {
+// If replication policy is replaced with new included/excluded tables
list, then events
+// corresponding to tables which are not included in old policy but
included in new policy
+// should be skipped.
+if (!ReplUtils.tableIncludedInReplScope(oldReplScope,
tableHandle.getTableName())) {
+ return false;
+}
+
+// Tables in the list of tables to be bootstrapped should be skipped.
+return (!bootstrapTableList.contains(tableHandle.getTableName()));
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264517)
Time Spent: 3h 20m (was: 3h 10m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264513=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264513
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 10:44
Start Date: 21/Jun/19 10:44
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296162776
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestTableLevelReplicationScenarios.java
##
@@ -585,4 +585,143 @@ public void
testReplacePolicyOnBootstrapExternalTablesIncrementalPhase() throws
.run("show tables")
.verifyResults(incrementalReplicatedTables);
}
+
+ @Test
+ public void testRenameTableScenariosBasic() throws Throwable {
+String replPolicy = primaryDbName + ".['in[0-9]+'].['out[0-9]+']";
+String lastReplId = replicateAndVerify(replPolicy, null, null, null,
+null, new String[]{}, new String[]{});
+
+String[] originalNonAcidTables = new String[]{"in1", "in2", "out3",
"out4", "out5", "out6"};
+createTables(originalNonAcidTables, CreateTableType.NON_ACID);
+
+// Replicate and verify if only 2 tables are replicated to target.
+String[] replicatedTables = new String[]{"in1", "in2"};
+String[] bootstrapTables = new String[]{};
+lastReplId = replicateAndVerify(replPolicy, null, lastReplId, null,
+null, bootstrapTables, replicatedTables);
+
+// Rename tables to make them satisfy the filter.
+primary.run("use " + primaryDbName)
+.run("alter table out3 rename to in3")
+.run("alter table out4 rename to in4")
+.run("alter table out5 rename to in5");
+
+replicatedTables = new String[]{"in1", "in2", "in3", "in4", "in5"};
+bootstrapTables = new String[]{"in3", "in4", "in5"};
+lastReplId = replicateAndVerify(replPolicy, null, lastReplId, null,
+null, bootstrapTables, replicatedTables);
+
+primary.run("use " + primaryDbName)
+.run("alter table in3 rename to in7")
+.run("alter table in7 rename to in8") // Double rename, both
satisfying the filter, so no bootstrap.
+.run("alter table in4 rename to out9") // out9 does not match the
filter so in4 should be dropped.
+.run("alter table in5 rename to out10") // Rename from satisfying
name to not satisfying name.
+.run("alter table out10 rename to in11");// from non satisfying to
satisfying, should be bootstrapped
+
+replicatedTables = new String[]{"in1", "in2", "in8", "in11"};
+bootstrapTables = new String[]{"in11"};
+lastReplId = replicateAndVerify(replPolicy, null, lastReplId, null,
+null, bootstrapTables, replicatedTables);
+
+primary.run("use " + primaryDbName)
+.run("alter table in8 rename to in12") // table is renamed from
satisfying to satisfying, no bootstrap
+.run("alter table out9 rename to in13") // out9 does not match the
filter so in13 should be bootstrapped.
+.run("alter table in13 rename to in14") // table is renamed from
satisfying to satisfying
+.run("drop table in14"); // table in14 is dropped, so no
bootstrap should happen.
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264513)
Time Spent: 3h (was: 2h 50m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264509=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264509
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 10:44
Start Date: 21/Jun/19 10:44
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296137435
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -82,14 +90,49 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
+ // return true, if event needs to be dumped, else return false.
+ private boolean handleRenameDuringTableLevelReplication(Context
withinContext) {
Review comment:
its not just checking but handling like adding drop and all ..so the name
looks fine
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264509)
Time Spent: 2h 20m (was: 2h 10m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264510=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264510
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 10:44
Start Date: 21/Jun/19 10:44
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296170587
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -82,14 +90,49 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
+ // return true, if event needs to be dumped, else return false.
+ private boolean handleRenameDuringTableLevelReplication(Context
withinContext) {
+String oldName = before.getTableName();
+String newName = after.getTableName();
+
+if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
+ // If old table satisfies the filter, but the new table does not, then
the old table should be dropped.
+ if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope,
newName)) {
+scenario = Scenario.DROP;
+LOG.info("Table " + oldName + " will be dropped as the table is
renamed to " + newName);
+return true;
+ }
+
+ // If both old and new table satisfies the filter, then dump the rename
event.
+ return true;
+} else {
+ // if the old table does not satisfies the filter, but the new one
satisfies, then the new table should be
+ // added to the list of tables to be bootstrapped and don't dump the
event.
+ if (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
newName)) {
+LOG.info("Table " + newName + " is added for bootstrap " + " during
rename from " + oldName);
+withinContext.tablesForBootstrap.add(newName);
+return false;
+ }
+
+ // if both old and new table does not satisfies the filter, then don't
dump the event.
+ return false;
+}
+ }
+
@Override
public void handle(Context withinContext) throws Exception {
LOG.info("Processing#{} ALTER_TABLE message : {}", fromEventId(),
eventMessageAsJSON);
Table qlMdTableBefore = new Table(before);
+if (Scenario.RENAME == scenario) {
+ // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
+ // list of tables to bne bootstrapped.
+ withinContext.tablesForBootstrap.remove(before.getTableName());
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264510)
Time Spent: 2.5h (was: 2h 20m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264515=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264515
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 10:44
Start Date: 21/Jun/19 10:44
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296170541
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -82,14 +90,49 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
+ // return true, if event needs to be dumped, else return false.
+ private boolean handleRenameDuringTableLevelReplication(Context
withinContext) {
+String oldName = before.getTableName();
+String newName = after.getTableName();
+
+if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
+ // If old table satisfies the filter, but the new table does not, then
the old table should be dropped.
+ if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope,
newName)) {
+scenario = Scenario.DROP;
+LOG.info("Table " + oldName + " will be dropped as the table is
renamed to " + newName);
+return true;
+ }
+
+ // If both old and new table satisfies the filter, then dump the rename
event.
+ return true;
+} else {
+ // if the old table does not satisfies the filter, but the new one
satisfies, then the new table should be
+ // added to the list of tables to be bootstrapped and don't dump the
event.
+ if (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
newName)) {
+LOG.info("Table " + newName + " is added for bootstrap " + " during
rename from " + oldName);
+withinContext.tablesForBootstrap.add(newName);
+return false;
+ }
+
+ // if both old and new table does not satisfies the filter, then don't
dump the event.
+ return false;
+}
+ }
+
@Override
public void handle(Context withinContext) throws Exception {
LOG.info("Processing#{} ALTER_TABLE message : {}", fromEventId(),
eventMessageAsJSON);
Table qlMdTableBefore = new Table(before);
+if (Scenario.RENAME == scenario) {
+ // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
+ // list of tables to bne bootstrapped.
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264515)
Time Spent: 3h 20m (was: 3h 10m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264505=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264505
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 10:44
Start Date: 21/Jun/19 10:44
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296135283
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/event/filters/ReplEventFilter.java
##
@@ -30,12 +31,22 @@ public ReplEventFilter(final ReplScope replScope) {
this.replScope = replScope;
}
+ // Return false if all the tables are included, as bootstrap during rename
is done only in case filter set for tables.
+ boolean isAlterDuringTableLevelReplication(final NotificationEvent event) {
+if (replScope.includeAllTables() ||
!replScope.getDbNamePattern().matcher(event.getDbName()).matches()) {
+ return false;
+}
+return event.getEventType().equals(MessageBuilder.ALTER_TABLE_EVENT);
Review comment:
for that we have to deserialize the event ..as new table name is not present
in the event structure
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264505)
Time Spent: 1h 50m (was: 1h 40m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264507=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264507
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 10:44
Start Date: 21/Jun/19 10:44
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296135880
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -101,6 +144,14 @@ public void handle(Context withinContext) throws
Exception {
}
}
+// If the tables are filtered based on name, then needs to handle the
rename scenarios.
+if (withinContext.replScope != null &&
!withinContext.replScope.includeAllTables()) {
+ if (!handleRenameDuringTableLevelReplication(withinContext)) {
Review comment:
its handled for other alter table scenarios also ..as the filter is not
applied in event filter
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264507)
Time Spent: 2h 10m (was: 2h)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264516=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264516
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 10:44
Start Date: 21/Jun/19 10:44
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296170880
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/load/message/DropTableHandler.java
##
@@ -32,9 +34,18 @@
@Override
public List> handle(Context context)
throws SemanticException {
-DropTableMessage msg =
deserializer.getDropTableMessage(context.dmd.getPayload());
-String actualDbName = context.isDbNameEmpty() ? msg.getDB() :
context.dbName;
-String actualTblName = msg.getTable();
+String actualDbName;
+String actualTblName;
+if (context.dmd.getDumpType() == DumpType.EVENT_DROP_TABLE) {
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264516)
Time Spent: 3h 20m (was: 3h 10m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264506=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264506
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 10:44
Start Date: 21/Jun/19 10:44
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296135476
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/event/filters/ReplEventFilter.java
##
@@ -30,12 +31,22 @@ public ReplEventFilter(final ReplScope replScope) {
this.replScope = replScope;
}
+ // Return false if all the tables are included, as bootstrap during rename
is done only in case filter set for tables.
+ boolean isAlterDuringTableLevelReplication(final NotificationEvent event) {
Review comment:
as this function is not doing any check for rename ..so its kept like this
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264506)
Time Spent: 2h (was: 1h 50m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264511=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264511
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 10:44
Start Date: 21/Jun/19 10:44
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296170372
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -82,14 +90,49 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
+ // return true, if event needs to be dumped, else return false.
+ private boolean handleRenameDuringTableLevelReplication(Context
withinContext) {
+String oldName = before.getTableName();
+String newName = after.getTableName();
+
+if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
+ // If old table satisfies the filter, but the new table does not, then
the old table should be dropped.
+ if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope,
newName)) {
+scenario = Scenario.DROP;
+LOG.info("Table " + oldName + " will be dropped as the table is
renamed to " + newName);
+return true;
+ }
+
+ // If both old and new table satisfies the filter, then dump the rename
event.
+ return true;
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264511)
Time Spent: 2h 40m (was: 2.5h)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264508=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264508
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 10:44
Start Date: 21/Jun/19 10:44
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296170773
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -101,6 +144,14 @@ public void handle(Context withinContext) throws
Exception {
}
}
+// If the tables are filtered based on name, then needs to handle the
rename scenarios.
+if (withinContext.replScope != null &&
!withinContext.replScope.includeAllTables()) {
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264508)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264512=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264512
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 10:44
Start Date: 21/Jun/19 10:44
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296166521
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestTableLevelReplicationScenarios.java
##
@@ -585,4 +585,143 @@ public void
testReplacePolicyOnBootstrapExternalTablesIncrementalPhase() throws
.run("show tables")
.verifyResults(incrementalReplicatedTables);
}
+
+ @Test
+ public void testRenameTableScenariosBasic() throws Throwable {
+String replPolicy = primaryDbName + ".['in[0-9]+'].['out[0-9]+']";
+String lastReplId = replicateAndVerify(replPolicy, null, null, null,
+null, new String[]{}, new String[]{});
+
+String[] originalNonAcidTables = new String[]{"in1", "in2", "out3",
"out4", "out5", "out6"};
+createTables(originalNonAcidTables, CreateTableType.NON_ACID);
+
+// Replicate and verify if only 2 tables are replicated to target.
+String[] replicatedTables = new String[]{"in1", "in2"};
+String[] bootstrapTables = new String[]{};
+lastReplId = replicateAndVerify(replPolicy, null, lastReplId, null,
+null, bootstrapTables, replicatedTables);
+
+// Rename tables to make them satisfy the filter.
+primary.run("use " + primaryDbName)
+.run("alter table out3 rename to in3")
+.run("alter table out4 rename to in4")
+.run("alter table out5 rename to in5");
+
+replicatedTables = new String[]{"in1", "in2", "in3", "in4", "in5"};
+bootstrapTables = new String[]{"in3", "in4", "in5"};
+lastReplId = replicateAndVerify(replPolicy, null, lastReplId, null,
+null, bootstrapTables, replicatedTables);
+
+primary.run("use " + primaryDbName)
+.run("alter table in3 rename to in7")
+.run("alter table in7 rename to in8") // Double rename, both
satisfying the filter, so no bootstrap.
+.run("alter table in4 rename to out9") // out9 does not match the
filter so in4 should be dropped.
+.run("alter table in5 rename to out10") // Rename from satisfying
name to not satisfying name.
+.run("alter table out10 rename to in11");// from non satisfying to
satisfying, should be bootstrapped
+
+replicatedTables = new String[]{"in1", "in2", "in8", "in11"};
+bootstrapTables = new String[]{"in11"};
+lastReplId = replicateAndVerify(replPolicy, null, lastReplId, null,
+null, bootstrapTables, replicatedTables);
+
+primary.run("use " + primaryDbName)
+.run("alter table in8 rename to in12") // table is renamed from
satisfying to satisfying, no bootstrap
+.run("alter table out9 rename to in13") // out9 does not match the
filter so in13 should be bootstrapped.
+.run("alter table in13 rename to in14") // table is renamed from
satisfying to satisfying
+.run("drop table in14"); // table in14 is dropped, so no
bootstrap should happen.
+
+replicatedTables = new String[]{"in1", "in2", "in12", "in12"};
+bootstrapTables = new String[]{};
+replicateAndVerify(replPolicy, null, lastReplId, null,
+null, bootstrapTables, replicatedTables);
+ }
+
+ @Test
+ public void testRenameTableScenariosAcidTable() throws Throwable {
+String replPolicy = primaryDbName + ".['in[0-9]+'].['out[0-9]+']";
+List dumpWithClause = Arrays.asList("'" +
HiveConf.ConfVars.REPL_BOOTSTRAP_ACID_TABLES.varname + "'='false'",
+"'" + ReplUtils.REPL_DUMP_INCLUDE_ACID_TABLES + "'='false'");
+String lastReplId = replicateAndVerify(replPolicy, null, null,
dumpWithClause,
+null, new String[]{}, new String[]{});
+
+String[] originalNonAcidTables = new String[]{"in1", "out4"};
+String[] originalFullAcidTables = new String[]{"in2", "out5"};
+String[] originalMMAcidTables = new String[]{"out3", "out6"};
+createTables(originalNonAcidTables, CreateTableType.NON_ACID);
+createTables(originalFullAcidTables, CreateTableType.FULL_ACID);
+createTables(originalMMAcidTables, CreateTableType.MM_ACID);
+
+// Replicate and verify if only 1 tables are replicated to target. Acid
tables are not dumped.
+String[] replicatedTables = new String[]{"in1"};
+String[] bootstrapTables = new String[]{};
+lastReplId = replicateAndVerify(replPolicy, null, lastReplId,
dumpWithClause,
+null,
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264378=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264378
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 06:27
Start Date: 21/Jun/19 06:27
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296106604
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -82,14 +90,49 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
+ // return true, if event needs to be dumped, else return false.
+ private boolean handleRenameDuringTableLevelReplication(Context
withinContext) {
+String oldName = before.getTableName();
+String newName = after.getTableName();
+
+if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
+ // If old table satisfies the filter, but the new table does not, then
the old table should be dropped.
+ if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope,
newName)) {
+scenario = Scenario.DROP;
+LOG.info("Table " + oldName + " will be dropped as the table is
renamed to " + newName);
+return true;
+ }
+
+ // If both old and new table satisfies the filter, then dump the rename
event.
+ return true;
+} else {
+ // if the old table does not satisfies the filter, but the new one
satisfies, then the new table should be
+ // added to the list of tables to be bootstrapped and don't dump the
event.
+ if (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
newName)) {
+LOG.info("Table " + newName + " is added for bootstrap " + " during
rename from " + oldName);
+withinContext.tablesForBootstrap.add(newName);
+return false;
+ }
+
+ // if both old and new table does not satisfies the filter, then don't
dump the event.
+ return false;
+}
+ }
+
@Override
public void handle(Context withinContext) throws Exception {
LOG.info("Processing#{} ALTER_TABLE message : {}", fromEventId(),
eventMessageAsJSON);
Table qlMdTableBefore = new Table(before);
+if (Scenario.RENAME == scenario) {
+ // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
+ // list of tables to bne bootstrapped.
+ withinContext.tablesForBootstrap.remove(before.getTableName());
Review comment:
Should we use lower case?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264378)
Time Spent: 40m (was: 0.5h)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264382=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264382
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 06:27
Start Date: 21/Jun/19 06:27
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296103770
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/event/filters/ReplEventFilter.java
##
@@ -30,12 +31,22 @@ public ReplEventFilter(final ReplScope replScope) {
this.replScope = replScope;
}
+ // Return false if all the tables are included, as bootstrap during rename
is done only in case filter set for tables.
+ boolean isAlterDuringTableLevelReplication(final NotificationEvent event) {
+if (replScope.includeAllTables() ||
!replScope.getDbNamePattern().matcher(event.getDbName()).matches()) {
+ return false;
+}
+return event.getEventType().equals(MessageBuilder.ALTER_TABLE_EVENT);
Review comment:
It is incorrect to include all alter events. If it is rename event and if
new table matches the replScope, then only include it. I think, no need to
expose getDbNamePattern method also.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264382)
Time Spent: 1h (was: 50m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264377=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264377
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 06:27
Start Date: 21/Jun/19 06:27
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296110148
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -82,14 +90,49 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
+ // return true, if event needs to be dumped, else return false.
+ private boolean handleRenameDuringTableLevelReplication(Context
withinContext) {
+String oldName = before.getTableName();
+String newName = after.getTableName();
+
+if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
+ // If old table satisfies the filter, but the new table does not, then
the old table should be dropped.
+ if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope,
newName)) {
+scenario = Scenario.DROP;
+LOG.info("Table " + oldName + " will be dropped as the table is
renamed to " + newName);
+return true;
+ }
+
+ // If both old and new table satisfies the filter, then dump the rename
event.
+ return true;
Review comment:
For case, out1 -> in1 -> in2, in2 should be part of bootstrap list but it
won't as caller has removed in1 from the list and here we don't add in2 to it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264377)
Time Spent: 0.5h (was: 20m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264383=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264383
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 06:27
Start Date: 21/Jun/19 06:27
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296102235
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -101,6 +144,14 @@ public void handle(Context withinContext) throws
Exception {
}
}
+// If the tables are filtered based on name, then needs to handle the
rename scenarios.
+if (withinContext.replScope != null &&
!withinContext.replScope.includeAllTables()) {
Review comment:
withinContext.replScope can never be null. The redundant check can be
removed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264383)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264384=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264384
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 06:27
Start Date: 21/Jun/19 06:27
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296103088
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/load/message/DropTableHandler.java
##
@@ -32,9 +34,18 @@
@Override
public List> handle(Context context)
throws SemanticException {
-DropTableMessage msg =
deserializer.getDropTableMessage(context.dmd.getPayload());
-String actualDbName = context.isDbNameEmpty() ? msg.getDB() :
context.dbName;
-String actualTblName = msg.getTable();
+String actualDbName;
+String actualTblName;
+if (context.dmd.getDumpType() == DumpType.EVENT_DROP_TABLE) {
Review comment:
I think, if flow is for EVENT_RENAME_DROP_TABLE and else flow for
EVENT_DROP_TABLE. Looks incorrect here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264384)
Time Spent: 1h 10m (was: 1h)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264386=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264386
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 06:27
Start Date: 21/Jun/19 06:27
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296104074
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/event/filters/ReplEventFilter.java
##
@@ -30,12 +31,22 @@ public ReplEventFilter(final ReplScope replScope) {
this.replScope = replScope;
}
+ // Return false if all the tables are included, as bootstrap during rename
is done only in case filter set for tables.
+ boolean isAlterDuringTableLevelReplication(final NotificationEvent event) {
Review comment:
The function name can be isRenameTableEventToBeIncluded.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264386)
Time Spent: 1h 20m (was: 1h 10m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264381=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264381
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 06:27
Start Date: 21/Jun/19 06:27
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296102051
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -82,14 +90,49 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
+ // return true, if event needs to be dumped, else return false.
+ private boolean handleRenameDuringTableLevelReplication(Context
withinContext) {
Review comment:
The function name can be changed to shouldDumpRenameTableEvent().
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264381)
Time Spent: 50m (was: 40m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264385=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264385
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 06:27
Start Date: 21/Jun/19 06:27
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296109918
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestTableLevelReplicationScenarios.java
##
@@ -585,4 +585,143 @@ public void
testReplacePolicyOnBootstrapExternalTablesIncrementalPhase() throws
.run("show tables")
.verifyResults(incrementalReplicatedTables);
}
+
+ @Test
+ public void testRenameTableScenariosBasic() throws Throwable {
+String replPolicy = primaryDbName + ".['in[0-9]+'].['out[0-9]+']";
+String lastReplId = replicateAndVerify(replPolicy, null, null, null,
+null, new String[]{}, new String[]{});
+
+String[] originalNonAcidTables = new String[]{"in1", "in2", "out3",
"out4", "out5", "out6"};
+createTables(originalNonAcidTables, CreateTableType.NON_ACID);
+
+// Replicate and verify if only 2 tables are replicated to target.
+String[] replicatedTables = new String[]{"in1", "in2"};
+String[] bootstrapTables = new String[]{};
+lastReplId = replicateAndVerify(replPolicy, null, lastReplId, null,
+null, bootstrapTables, replicatedTables);
+
+// Rename tables to make them satisfy the filter.
+primary.run("use " + primaryDbName)
+.run("alter table out3 rename to in3")
+.run("alter table out4 rename to in4")
+.run("alter table out5 rename to in5");
+
+replicatedTables = new String[]{"in1", "in2", "in3", "in4", "in5"};
+bootstrapTables = new String[]{"in3", "in4", "in5"};
+lastReplId = replicateAndVerify(replPolicy, null, lastReplId, null,
+null, bootstrapTables, replicatedTables);
+
+primary.run("use " + primaryDbName)
+.run("alter table in3 rename to in7")
+.run("alter table in7 rename to in8") // Double rename, both
satisfying the filter, so no bootstrap.
+.run("alter table in4 rename to out9") // out9 does not match the
filter so in4 should be dropped.
+.run("alter table in5 rename to out10") // Rename from satisfying
name to not satisfying name.
+.run("alter table out10 rename to in11");// from non satisfying to
satisfying, should be bootstrapped
+
+replicatedTables = new String[]{"in1", "in2", "in8", "in11"};
+bootstrapTables = new String[]{"in11"};
+lastReplId = replicateAndVerify(replPolicy, null, lastReplId, null,
+null, bootstrapTables, replicatedTables);
+
+primary.run("use " + primaryDbName)
+.run("alter table in8 rename to in12") // table is renamed from
satisfying to satisfying, no bootstrap
+.run("alter table out9 rename to in13") // out9 does not match the
filter so in13 should be bootstrapped.
+.run("alter table in13 rename to in14") // table is renamed from
satisfying to satisfying
+.run("drop table in14"); // table in14 is dropped, so no
bootstrap should happen.
Review comment:
Need to add a scenario for out9 -> in13 -> in14 without drop (in14). It
should bootstrap in14.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264385)
Time Spent: 1h 10m (was: 1h)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264387=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264387
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 06:27
Start Date: 21/Jun/19 06:27
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296101528
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -82,14 +90,49 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
+ // return true, if event needs to be dumped, else return false.
+ private boolean handleRenameDuringTableLevelReplication(Context
withinContext) {
+String oldName = before.getTableName();
+String newName = after.getTableName();
+
+if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
+ // If old table satisfies the filter, but the new table does not, then
the old table should be dropped.
+ if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope,
newName)) {
+scenario = Scenario.DROP;
+LOG.info("Table " + oldName + " will be dropped as the table is
renamed to " + newName);
+return true;
+ }
+
+ // If both old and new table satisfies the filter, then dump the rename
event.
+ return true;
+} else {
+ // if the old table does not satisfies the filter, but the new one
satisfies, then the new table should be
+ // added to the list of tables to be bootstrapped and don't dump the
event.
+ if (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
newName)) {
+LOG.info("Table " + newName + " is added for bootstrap " + " during
rename from " + oldName);
+withinContext.tablesForBootstrap.add(newName);
+return false;
+ }
+
+ // if both old and new table does not satisfies the filter, then don't
dump the event.
+ return false;
+}
+ }
+
@Override
public void handle(Context withinContext) throws Exception {
LOG.info("Processing#{} ALTER_TABLE message : {}", fromEventId(),
eventMessageAsJSON);
Table qlMdTableBefore = new Table(before);
+if (Scenario.RENAME == scenario) {
+ // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
+ // list of tables to bne bootstrapped.
Review comment:
Typo error: "be"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264387)
Time Spent: 1.5h (was: 1h 20m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264388=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264388
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 06:27
Start Date: 21/Jun/19 06:27
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296098970
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/Utils.java
##
@@ -214,29 +215,35 @@ public static boolean shouldReplicate(ReplicationSpec
replicationSpec, Table tab
return false;
}
-// Skip dumping events related to ACID tables if bootstrap is enabled
on it.
-// Also, skip if current table is included only in new policy but not
in old policy.
-if (isEventDump) {
- return
!hiveConf.getBoolVar(HiveConf.ConfVars.REPL_BOOTSTRAP_ACID_TABLES)
- && ReplUtils.tableIncludedInReplScope(oldReplScope,
tableHandle.getTableName());
+// Skip dumping events related to ACID tables if bootstrap is enabled
for ACID tables.
+if (isEventDump &&
hiveConf.getBoolVar(HiveConf.ConfVars.REPL_BOOTSTRAP_ACID_TABLES)) {
+ return false;
}
}
- // If replication policy is replaced with new included/excluded tables
list, then events
- // corresponding to tables which are not included in old policy but
included in new policy
- // should be skipped. Those tables would be bootstrapped along with the
current incremental
- // replication dump.
- // Note: If any event dump reaches here, it means, table is included in
new replication policy.
- if (isEventDump && !ReplUtils.tableIncludedInReplScope(oldReplScope,
tableHandle.getTableName())) {
-return false;
+ // Tables which are selected for bootstrap should be skipped. Those
tables would be bootstrapped
+ // along with the current incremental replication dump and thus no need
to dump events for them.
+ // Note: If any event (other than alter table with table level
replication) dump reaches here, it means, table is
+ // included in new replication policy.
+ if (isEventDump) {
+// If replication policy is replaced with new included/excluded tables
list, then events
+// corresponding to tables which are not included in old policy but
included in new policy
+// should be skipped.
+if (!ReplUtils.tableIncludedInReplScope(oldReplScope,
tableHandle.getTableName())) {
+ return false;
+}
+
+// Tables in the list of tables to be bootstrapped should be skipped.
+return (!bootstrapTableList.contains(tableHandle.getTableName()));
Review comment:
Should we convert to lower case?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264388)
Time Spent: 1h 40m (was: 1.5h)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264380=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264380
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 06:27
Start Date: 21/Jun/19 06:27
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296107333
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/Utils.java
##
@@ -214,29 +215,35 @@ public static boolean shouldReplicate(ReplicationSpec
replicationSpec, Table tab
return false;
}
-// Skip dumping events related to ACID tables if bootstrap is enabled
on it.
-// Also, skip if current table is included only in new policy but not
in old policy.
-if (isEventDump) {
- return
!hiveConf.getBoolVar(HiveConf.ConfVars.REPL_BOOTSTRAP_ACID_TABLES)
- && ReplUtils.tableIncludedInReplScope(oldReplScope,
tableHandle.getTableName());
+// Skip dumping events related to ACID tables if bootstrap is enabled
for ACID tables.
+if (isEventDump &&
hiveConf.getBoolVar(HiveConf.ConfVars.REPL_BOOTSTRAP_ACID_TABLES)) {
+ return false;
}
}
- // If replication policy is replaced with new included/excluded tables
list, then events
- // corresponding to tables which are not included in old policy but
included in new policy
- // should be skipped. Those tables would be bootstrapped along with the
current incremental
- // replication dump.
- // Note: If any event dump reaches here, it means, table is included in
new replication policy.
- if (isEventDump && !ReplUtils.tableIncludedInReplScope(oldReplScope,
tableHandle.getTableName())) {
-return false;
+ // Tables which are selected for bootstrap should be skipped. Those
tables would be bootstrapped
+ // along with the current incremental replication dump and thus no need
to dump events for them.
+ // Note: If any event (other than alter table with table level
replication) dump reaches here, it means, table is
+ // included in new replication policy.
+ if (isEventDump) {
+// If replication policy is replaced with new included/excluded tables
list, then events
+// corresponding to tables which are not included in old policy but
included in new policy
+// should be skipped.
+if (!ReplUtils.tableIncludedInReplScope(oldReplScope,
tableHandle.getTableName())) {
+ return false;
+}
+
+// Tables in the list of tables to be bootstrapped should be skipped.
Review comment:
Comment can explicitly say that events on tables to be bootstrapped can be
skipped.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264380)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264379=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264379
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 06:27
Start Date: 21/Jun/19 06:27
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296097381
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestTableLevelReplicationScenarios.java
##
@@ -585,4 +585,143 @@ public void
testReplacePolicyOnBootstrapExternalTablesIncrementalPhase() throws
.run("show tables")
.verifyResults(incrementalReplicatedTables);
}
+
+ @Test
+ public void testRenameTableScenariosBasic() throws Throwable {
+String replPolicy = primaryDbName + ".['in[0-9]+'].['out[0-9]+']";
+String lastReplId = replicateAndVerify(replPolicy, null, null, null,
+null, new String[]{}, new String[]{});
+
+String[] originalNonAcidTables = new String[]{"in1", "in2", "out3",
"out4", "out5", "out6"};
+createTables(originalNonAcidTables, CreateTableType.NON_ACID);
+
+// Replicate and verify if only 2 tables are replicated to target.
+String[] replicatedTables = new String[]{"in1", "in2"};
+String[] bootstrapTables = new String[]{};
+lastReplId = replicateAndVerify(replPolicy, null, lastReplId, null,
+null, bootstrapTables, replicatedTables);
+
+// Rename tables to make them satisfy the filter.
+primary.run("use " + primaryDbName)
+.run("alter table out3 rename to in3")
+.run("alter table out4 rename to in4")
+.run("alter table out5 rename to in5");
+
+replicatedTables = new String[]{"in1", "in2", "in3", "in4", "in5"};
+bootstrapTables = new String[]{"in3", "in4", "in5"};
+lastReplId = replicateAndVerify(replPolicy, null, lastReplId, null,
+null, bootstrapTables, replicatedTables);
+
+primary.run("use " + primaryDbName)
+.run("alter table in3 rename to in7")
+.run("alter table in7 rename to in8") // Double rename, both
satisfying the filter, so no bootstrap.
+.run("alter table in4 rename to out9") // out9 does not match the
filter so in4 should be dropped.
+.run("alter table in5 rename to out10") // Rename from satisfying
name to not satisfying name.
+.run("alter table out10 rename to in11");// from non satisfying to
satisfying, should be bootstrapped
+
+replicatedTables = new String[]{"in1", "in2", "in8", "in11"};
+bootstrapTables = new String[]{"in11"};
+lastReplId = replicateAndVerify(replPolicy, null, lastReplId, null,
+null, bootstrapTables, replicatedTables);
+
+primary.run("use " + primaryDbName)
+.run("alter table in8 rename to in12") // table is renamed from
satisfying to satisfying, no bootstrap
+.run("alter table out9 rename to in13") // out9 does not match the
filter so in13 should be bootstrapped.
+.run("alter table in13 rename to in14") // table is renamed from
satisfying to satisfying
+.run("drop table in14"); // table in14 is dropped, so no
bootstrap should happen.
+
+replicatedTables = new String[]{"in1", "in2", "in12", "in12"};
+bootstrapTables = new String[]{};
+replicateAndVerify(replPolicy, null, lastReplId, null,
+null, bootstrapTables, replicatedTables);
+ }
+
+ @Test
+ public void testRenameTableScenariosAcidTable() throws Throwable {
+String replPolicy = primaryDbName + ".['in[0-9]+'].['out[0-9]+']";
+List dumpWithClause = Arrays.asList("'" +
HiveConf.ConfVars.REPL_BOOTSTRAP_ACID_TABLES.varname + "'='false'",
+"'" + ReplUtils.REPL_DUMP_INCLUDE_ACID_TABLES + "'='false'");
+String lastReplId = replicateAndVerify(replPolicy, null, null,
dumpWithClause,
+null, new String[]{}, new String[]{});
+
+String[] originalNonAcidTables = new String[]{"in1", "out4"};
+String[] originalFullAcidTables = new String[]{"in2", "out5"};
+String[] originalMMAcidTables = new String[]{"out3", "out6"};
+createTables(originalNonAcidTables, CreateTableType.NON_ACID);
+createTables(originalFullAcidTables, CreateTableType.FULL_ACID);
+createTables(originalMMAcidTables, CreateTableType.MM_ACID);
+
+// Replicate and verify if only 1 tables are replicated to target. Acid
tables are not dumped.
+String[] replicatedTables = new String[]{"in1"};
+String[] bootstrapTables = new String[]{};
+lastReplId = replicateAndVerify(replPolicy, null, lastReplId,
dumpWithClause,
+null, bootstrapTables,
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264376=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264376
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 06:27
Start Date: 21/Jun/19 06:27
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296104789
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -101,6 +144,14 @@ public void handle(Context withinContext) throws
Exception {
}
}
+// If the tables are filtered based on name, then needs to handle the
rename scenarios.
+if (withinContext.replScope != null &&
!withinContext.replScope.includeAllTables()) {
+ if (!handleRenameDuringTableLevelReplication(withinContext)) {
Review comment:
This can be done only if scenario = Scenario.RENAME.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264376)
Time Spent: 20m (was: 10m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[ https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=263495=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-263495 ] ASF GitHub Bot logged work on HIVE-21764: - Author: ASF GitHub Bot Created on: 20/Jun/19 02:28 Start Date: 20/Jun/19 02:28 Worklog Time Spent: 10m Work Description: maheshk114 commented on pull request #679: HIVE-21764 : REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included. URL: https://github.com/apache/hive/pull/679 … This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 263495) Time Spent: 10m Remaining Estimate: 0h > REPL DUMP should detect and bootstrap any rename table events where old table > was excluded but renamed table is included. > - > > Key: HIVE-21764 > URL: https://issues.apache.org/jira/browse/HIVE-21764 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Sankar Hariappan >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular > expression + inclusion/exclusion list. So, in case of rename table event, the > event will be ignored if old table doesn't match the pattern but the new > table should be bootstrapped. REPL DUMP should have a mechanism to detect > such tables and automatically bootstrap with incremental replication.Also, if > renamed table is excluded from replication policy, then need to drop the old > table at target as well. > There are 4 scenarios that needs to be handled. > # Both new name and old name satisfies the table name pattern filter. > ## No need to do anything. The incremental event for rename should take care > of the replication. > # Both the names does not satisfy the table name pattern filter. > ## Both the names are not in the scope of the policy and thus nothing needs > to be done. > # New name satisfies the pattern but the old name does not. > ## The table will not be present at the target. > ## Rename event handler for dump should detect this case and add the new > table name to the list of table for bootstrap. > ## All the events related to the table (new name) should be ignored. > ## If there is a drop event for the table (with new name), then remove the > table from the list of tables to be bootstrapped. > ## In case of rename (double rename) > ### If the new name satisfies the table pattern, then add the new name to > the list of tables to be bootstrapped and remove the old name from the list > of tables to be bootstrapped. > ### If the new name does not satisfies then just removed the table name from > the list of tables to be bootstrapped. > # New name does not satisfies the pattern but the old name satisfies. > ## Change the rename event to a drop event. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
