jt2594838 commented on code in PR #201:
URL: https://github.com/apache/tsfile/pull/201#discussion_r1716197989
##########
java/tools/src/main/java/org/apache/tsfile/tools/TsFileTool.java:
##########
@@ -158,20 +167,25 @@ private static boolean writeTsFile(String fileName,
List<String> lineList) {
return false;
}
} catch (Exception e) {
- e.printStackTrace();
- LOGGER.error("Failed to write file: " + tsFile);
+ LOGGER.error("Failed to write file" + tsFile.getAbsolutePath() + ":{} ",
e.getMessage());
return false;
} finally {
if (writer != null) {
try {
writer.close();
} catch (IOException e) {
- e.printStackTrace();
+ LOGGER.error("Failed to close file" + tsFile.getAbsolutePath() +
":{}", e.getMessage());
Review Comment:
Are you sure the exception stack is unnecessary? If not, may use
LOGGER.error("Failed to close file: " + tsFile.getAbsolutePath(), e);
##########
java/tools/src/main/java/org/apache/tsfile/tools/TsFileTool.java:
##########
@@ -158,20 +167,25 @@ private static boolean writeTsFile(String fileName,
List<String> lineList) {
return false;
}
} catch (Exception e) {
- e.printStackTrace();
- LOGGER.error("Failed to write file: " + tsFile);
+ LOGGER.error("Failed to write file" + tsFile.getAbsolutePath() + ":{} ",
e.getMessage());
return false;
} finally {
if (writer != null) {
try {
writer.close();
} catch (IOException e) {
- e.printStackTrace();
+ LOGGER.error("Failed to close file" + tsFile.getAbsolutePath() +
":{}", e.getMessage());
Review Comment:
Please also check other places.
##########
java/tools/src/test/java/org/apache/tsfile/tools/TsfiletoolsTest.java:
##########
Review Comment:
The new version seems to be able to move erroneous input files to the target
directory. How about adding associated test cases?
##########
java/tools/src/main/java/org/apache/tsfile/tools/TsFileTool.java:
##########
@@ -188,6 +202,7 @@ private static Tablet genTablet(
map.put(column.name, i);
}
try {
+ sortByColumnIndex(lineList);
for (int i = 0; i < num; i++) {
String line = lineList.get(i);
String[] lineArray = line.split(schema.separator);
Review Comment:
The lines are split twice, which is unnecessary.
You may first split them, sort them, and then iterate over them.
##########
java/tools/src/main/java/org/apache/tsfile/tools/TsFileTool.java:
##########
@@ -266,15 +296,57 @@ private static void processDirectory(File directory,
ExecutorService executor) {
private static void cpFile(String sourceFilePath, String
targetDirectoryPath) {
try {
- Files.createDirectories(Paths.get(targetDirectoryPath));
+ String inputFileAbsolutePath = new
File(inputDirectoryStr).getAbsolutePath();
+ String soureFlieName = new File(sourceFilePath).getName();
+ String fileOutPutDirStr =
+ targetDirectoryPath
+ + sourceFilePath.replace(inputFileAbsolutePath,
"").replace(soureFlieName, "");
+ Files.createDirectories(Paths.get(fileOutPutDirStr));
Path sourcePath = Paths.get(sourceFilePath);
- Path targetPath = Paths.get(targetDirectoryPath,
sourcePath.getFileName().toString());
+ Path targetPath = Paths.get(fileOutPutDirStr,
sourcePath.getFileName().toString());
Files.copy(sourcePath, targetPath, StandardCopyOption.REPLACE_EXISTING);
} catch (IOException e) {
LOGGER.error("Failed to copy file: " + sourceFilePath, e);
}
}
+ public static void writeToNewCSV(String fileAbsolutePath, List<String> data,
String newFileName) {
+
+ if (schema.hasHeader) {
+ try (BufferedReader reader =
+ new BufferedReader(
+ new InputStreamReader(
+ Files.newInputStream(Paths.get(fileAbsolutePath)),
StandardCharsets.UTF_8))) {
+ String line;
+ if ((line = reader.readLine()) != null) {
+ data.add(0, line);
+ }
+ } catch (Exception e) {
+ LOGGER.error("Error reading file: " + e.getMessage());
+ }
+ }
Review Comment:
First, how about caching the header line as a member field in the previous
process.
Second, I would assume the size of data to be quite large, so inserting an
element to the head may not be efficient if using an ArrayList, and a
LinkedList is also inefficient due to the list item overhead. You may just
store the header line in a local variable to avoid this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
