Github user hyunsik commented on a diff in the pull request:
https://github.com/apache/tajo/pull/161#discussion_r18433154
--- Diff:
tajo-core/src/main/java/org/apache/tajo/master/querymaster/Query.java ---
@@ -486,6 +532,65 @@ public Path commitOutputData(Query query) {
return finalOutputDir;
}
+ /**
+ * This method sets a a rename map which includes renamed staging
directory to final output directory recursively.
+ * If there exists some data files, this delete it for duplicate data.
+ *
+ *
+ * @param fs
+ * @param stagingPath
+ * @param outputPath
+ * @param stagingParentPathString
+ * @throws IOException
+ */
+ private void visitPartitionedDirectory(FileSystem fs, Path
stagingPath, Path outputPath,
+ String stagingParentPathString,
+ Map<Path, Path> renameDirs, Path
oldTableDir) throws IOException {
+ FileStatus[] files = fs.listStatus(stagingPath);
+
+ for(FileStatus eachFile : files) {
+ if (eachFile.isDirectory()) {
+ Path oldPath = eachFile.getPath();
+
+ // Make recover directory.
+ String recoverPathString =
oldPath.toString().replaceAll(stagingParentPathString,
+ oldTableDir.toString());
+ Path recoveryPath = new Path(recoverPathString);
+ if (!fs.exists(recoveryPath)) {
+ fs.mkdirs(recoveryPath);
+ }
+
+ visitPartitionedDirectory(fs, eachFile.getPath(), outputPath,
stagingParentPathString,
+ renameDirs, oldTableDir);
+ // Find last order partition for renaming
+ String newPathString =
oldPath.toString().replaceAll(stagingParentPathString,
+ outputPath.toString());
+ Path newPath = new Path(newPathString);
+ if (!hasDirectory(fs, eachFile.getPath())) {
--- End diff --
In my view, ```isLeafDirectory()``` would be more intuitive name than
```!hasDirectory()```.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
