keith-turner commented on a change in pull request #2185:
URL: https://github.com/apache/accumulo/pull/2185#discussion_r682676983
##########
File path:
server/manager/src/main/java/org/apache/accumulo/manager/upgrade/Upgrader9to10.java
##########
@@ -726,4 +734,37 @@ static Path resolveRelativeDelete(String oldDelete, String
upgradeProperty) {
}
return new Path(upgradeProperty,
VolumeManager.FileType.TABLE.getDirectory() + oldDelete);
}
+
+ /**
+ * Remove old temporary map files to prevent problems during recovery.
+ */
+ static void dropSortedMapWALFiles(VolumeManager vm) {
+ Path recoveryDir = new Path("/accumulo/recovery");
Review comment:
Seems like this should loop over the set of configured volumes instead
of this.
##########
File path:
server/manager/src/main/java/org/apache/accumulo/manager/upgrade/Upgrader9to10.java
##########
@@ -142,6 +148,8 @@ public void upgradeMetadata(ServerContext ctx) {
upgradeRelativePaths(ctx, Ample.DataLevel.USER);
upgradeDirColumns(ctx, Ample.DataLevel.USER);
upgradeFileDeletes(ctx, Ample.DataLevel.USER);
+ // special case where old files need to be deleted
Review comment:
Did you consider calling this new method in upgradeZookeeper instead? I
think that is called before the root tablet is loaded, which would allow
deleting any old sorted logs the root tablet may reference.
##########
File path:
server/manager/src/main/java/org/apache/accumulo/manager/upgrade/Upgrader9to10.java
##########
@@ -142,6 +148,8 @@ public void upgradeMetadata(ServerContext ctx) {
upgradeRelativePaths(ctx, Ample.DataLevel.USER);
upgradeDirColumns(ctx, Ample.DataLevel.USER);
upgradeFileDeletes(ctx, Ample.DataLevel.USER);
+ // special case where old files need to be deleted
Review comment:
I think the upgrade methods are run as follows.
* `upgradeZookeeper()` is run before the root tablet is loaded.
* `upgradeRoot()` is run after the root tablet is loaded and before the
metadata table is loaded.
* `upgradeMetadata` is run after the metadata table is loaded and before
loading user tablets.
##########
File path:
server/manager/src/main/java/org/apache/accumulo/manager/upgrade/Upgrader9to10.java
##########
@@ -726,4 +734,37 @@ static Path resolveRelativeDelete(String oldDelete, String
upgradeProperty) {
}
return new Path(upgradeProperty,
VolumeManager.FileType.TABLE.getDirectory() + oldDelete);
}
+
+ /**
+ * Remove old temporary map files to prevent problems during recovery.
+ */
+ static void dropSortedMapWALFiles(VolumeManager vm) {
+ Path recoveryDir = new Path("/accumulo/recovery");
Review comment:
Yeah, could look through all of those. Could narrow it by calling :
https://github.com/apache/accumulo/blob/a9100ac35f3d72677b767d7670dfb032c7e7d733/server/base/src/main/java/org/apache/accumulo/server/fs/VolumeManager.java#L180
But I don't think it hurts to just look through all volumes. Maybe could
use
https://github.com/apache/accumulo/blob/a9100ac35f3d72677b767d7670dfb032c7e7d733/server/base/src/main/java/org/apache/accumulo/server/fs/VolumeManager.java#L196
to get the list of volumes
##########
File path:
server/manager/src/main/java/org/apache/accumulo/manager/upgrade/Upgrader9to10.java
##########
@@ -726,4 +734,37 @@ static Path resolveRelativeDelete(String oldDelete, String
upgradeProperty) {
}
return new Path(upgradeProperty,
VolumeManager.FileType.TABLE.getDirectory() + oldDelete);
}
+
+ /**
+ * Remove old temporary map files to prevent problems during recovery.
+ */
+ static void dropSortedMapWALFiles(VolumeManager vm) {
+ Path recoveryDir = new Path("/accumulo/recovery");
Review comment:
> o you think using the volumes configured in instance.volumes will be
enough?
Yeah, could look through all of those. Could narrow it by calling :
https://github.com/apache/accumulo/blob/a9100ac35f3d72677b767d7670dfb032c7e7d733/server/base/src/main/java/org/apache/accumulo/server/fs/VolumeManager.java#L180
But I don't think it hurts to just look through all volumes. Maybe could
use
https://github.com/apache/accumulo/blob/a9100ac35f3d72677b767d7670dfb032c7e7d733/server/base/src/main/java/org/apache/accumulo/server/fs/VolumeManager.java#L196
to get the list of volumes
##########
File path:
server/manager/src/main/java/org/apache/accumulo/manager/upgrade/Upgrader9to10.java
##########
@@ -726,4 +734,37 @@ static Path resolveRelativeDelete(String oldDelete, String
upgradeProperty) {
}
return new Path(upgradeProperty,
VolumeManager.FileType.TABLE.getDirectory() + oldDelete);
}
+
+ /**
+ * Remove old temporary map files to prevent problems during recovery.
+ */
+ static void dropSortedMapWALFiles(VolumeManager vm) {
+ Path recoveryDir = new Path("/accumulo/recovery");
Review comment:
> Do you think using the volumes configured in instance.volumes will be
enough?
Yeah, could look through all of those. Could narrow it by calling :
https://github.com/apache/accumulo/blob/a9100ac35f3d72677b767d7670dfb032c7e7d733/server/base/src/main/java/org/apache/accumulo/server/fs/VolumeManager.java#L180
But I don't think it hurts to just look through all volumes. Maybe could
use
https://github.com/apache/accumulo/blob/a9100ac35f3d72677b767d7670dfb032c7e7d733/server/base/src/main/java/org/apache/accumulo/server/fs/VolumeManager.java#L196
to get the list of volumes
##########
File path:
server/manager/src/main/java/org/apache/accumulo/manager/upgrade/Upgrader9to10.java
##########
@@ -726,4 +734,37 @@ static Path resolveRelativeDelete(String oldDelete, String
upgradeProperty) {
}
return new Path(upgradeProperty,
VolumeManager.FileType.TABLE.getDirectory() + oldDelete);
}
+
+ /**
+ * Remove old temporary map files to prevent problems during recovery.
+ */
+ static void dropSortedMapWALFiles(VolumeManager vm) {
+ Path recoveryDir = new Path("/accumulo/recovery");
Review comment:
Would probably be best to avoid choosable() as that will give the
volumes configured for new WALs. The config could change and an old WAL could
be on a volume that choosable() no longer returns. So probably best to inspect
all volumes looking for old sorted wals to nuke.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
