>From Hussain Towaileb <hussai...@gmail.com>: Hussain Towaileb has submitted this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19783 )
Change subject: [ASTERIXDB-3420][STO] Account for file changes on directory traversal ...................................................................... [ASTERIXDB-3420][STO] Account for file changes on directory traversal - user model changes: no - storage format changes: no - interface changes: no Details: - Replace FileUtils.sizeOfDirectory with custom directory size function that accounts for file changes while traversing a directory. Ext-ref: MB-66673 Change-Id: I660233e6b74429c75b9dd63103db1248846054a0 Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18343 Reviewed-by: Murtadha Hubail <mhub...@apache.org> Tested-by: Murtadha Hubail <mhub...@apache.org> Integration-Tests: Murtadha Hubail <mhub...@apache.org> (cherry picked from commit 8b1003bd931136520f15f6eac103d0616026d229) Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19783 Tested-by: Jenkins <jenk...@fulliautomatix.ics.uci.edu> Reviewed-by: Michael Blow <mb...@apache.org> --- M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-nc/src/main/java/org/apache/hyracks/control/nc/io/IOManager.java M hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/util/IoUtil.java 2 files changed, 60 insertions(+), 2 deletions(-) Approvals: Michael Blow: Looks good to me, approved Jenkins: Verified diff --git a/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/util/IoUtil.java b/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/util/IoUtil.java index ae49cb6..9c29206 100644 --- a/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/util/IoUtil.java +++ b/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/util/IoUtil.java @@ -24,13 +24,17 @@ import java.io.FilenameFilter; import java.io.IOException; import java.nio.channels.FileChannel; +import java.nio.file.FileVisitResult; import java.nio.file.Files; import java.nio.file.NoSuchFileException; import java.nio.file.Path; +import java.nio.file.SimpleFileVisitor; import java.nio.file.StandardOpenOption; +import java.nio.file.attribute.BasicFileAttributes; import java.util.ArrayList; import java.util.Collection; import java.util.Objects; +import java.util.concurrent.atomic.AtomicLong; import org.apache.commons.io.FileUtils; import org.apache.hyracks.api.exceptions.ErrorCode; @@ -189,4 +193,31 @@ } } } + + public static long sizeOfDirectory(final Path path) { + final AtomicLong size = new AtomicLong(0); + try { + Files.walkFileTree(path, new SimpleFileVisitor<>() { + @Override + public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) { + size.addAndGet(attrs.size()); + return FileVisitResult.CONTINUE; + } + + @Override + public FileVisitResult visitFileFailed(Path file, IOException exc) { + return FileVisitResult.CONTINUE; + } + + @Override + public FileVisitResult postVisitDirectory(Path dir, IOException exc) { + return FileVisitResult.CONTINUE; + } + }); + } catch (IOException e) { + // This should never happen + throw new IllegalStateException("Cannot get the size of directory " + path); + } + return size.get(); + } } diff --git a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-nc/src/main/java/org/apache/hyracks/control/nc/io/IOManager.java b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-nc/src/main/java/org/apache/hyracks/control/nc/io/IOManager.java index dee61a4..8d058e5 100644 --- a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-nc/src/main/java/org/apache/hyracks/control/nc/io/IOManager.java +++ b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-nc/src/main/java/org/apache/hyracks/control/nc/io/IOManager.java @@ -37,7 +37,6 @@ import java.util.concurrent.Executors; import java.util.concurrent.TimeUnit; -import org.apache.commons.io.FileUtils; import org.apache.hyracks.api.exceptions.ErrorCode; import org.apache.hyracks.api.exceptions.HyracksDataException; import org.apache.hyracks.api.io.FileReference; @@ -441,7 +440,7 @@ public long getTotalDiskUsage() { long totalSize = 0; for (IODeviceHandle handle : ioDevices) { - totalSize += FileUtils.sizeOfDirectory(handle.getMount()); + totalSize += IoUtil.sizeOfDirectory(handle.getMount().toPath()); } return totalSize; } -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19783 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: neo Gerrit-Change-Id: I660233e6b74429c75b9dd63103db1248846054a0 Gerrit-Change-Number: 19783 Gerrit-PatchSet: 4 Gerrit-Owner: Hussain Towaileb <hussai...@gmail.com> Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Hussain Towaileb <hussai...@gmail.com> Gerrit-Reviewer: Jenkins <jenk...@fulliautomatix.ics.uci.edu> Gerrit-Reviewer: Michael Blow <mb...@apache.org> Gerrit-Reviewer: Murtadha Hubail <mhub...@apache.org> Gerrit-CC: Savyasach Reddy <savyasach.enuko...@couchbase.com> Gerrit-MessageType: merged
