github-code-scanning[bot] commented on code in PR #13298:
URL:
https://github.com/apache/dolphinscheduler/pull/13298#discussion_r1058685086
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/FileUtils.java:
##########
@@ -264,4 +267,38 @@
}
}
+ /**
+ * Calculate file checksum with CRC32 algorithm
+ *
+ * @param pathName
+ * @return
+ * @throws IOException
+ */
+ public static String getFileChecksum(String pathName) {
+ CRC32 crc32 = new CRC32();
+ File file = new File(pathName);
+ String crcString = "";
+ if (file.isDirectory()) {
+ // file system interface remains the same order
+ String[] subPaths = file.list();
+ for (String subPath : subPaths) {
+ crcString += getFileChecksum(pathName + FOLDER_SEPARATOR +
subPath);
+ }
+ } else {
+ FileInputStream fileInputStream = null;
+ try {
+ fileInputStream = new FileInputStream(pathName);
Review Comment:
## Potential input resource leak
This FileInputStream is not always closed on method exit.
[Show more
details](https://github.com/apache/dolphinscheduler/security/code-scanning/2455)
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/FileUtils.java:
##########
@@ -264,4 +267,38 @@
}
}
+ /**
+ * Calculate file checksum with CRC32 algorithm
+ *
+ * @param pathName
+ * @return
+ * @throws IOException
Review Comment:
## Javadoc has impossible 'throws' tag
Javadoc for getFileChecksum claims to throw IOException but this is
impossible.
[Show more
details](https://github.com/apache/dolphinscheduler/security/code-scanning/2454)
##########
dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/utils/TaskCacheUtils.java:
##########
@@ -139,9 +161,40 @@
.filter(property -> propertyInSet.contains(property.getProp()))
.sorted(Comparator.comparing(Property::getProp))
.collect(Collectors.toList());
+
+ varPool.forEach(property -> {
+ if (property.getType() == DataType.FILE) {
+ property.setValue(fileCheckSumMap.get(property.getValue()));
+ }
+ });
return JSONUtils.toJsonString(varPool);
}
+ /**
+ * get checksum from crc32 file of file property in varPool
+ * cache can be used if content of upstream output files are the same
+ * @param fileProperty
+ * @param context
+ * @param storageOperate
+ */
+ public static String getValCheckSum(Property fileProperty,
TaskExecutionContext context,
+ StorageOperate storageOperate) {
+ String resourceCRCPath = fileProperty.getValue() + CRC_SUFFIX;
+ String resourceCRCWholePath =
storageOperate.getResourceFileName(context.getTenantCode(), resourceCRCPath);
+ String targetPath = String.format("%s/%s", context.getExecutePath(),
resourceCRCPath);
+ logger.info("{} --- Remote:{} to Local:{}", "CRC file",
resourceCRCWholePath, targetPath);
+ String crcString = "";
+ try {
+ storageOperate.download(context.getTenantCode(),
resourceCRCWholePath, targetPath, false,
+ true);
+ crcString = FileUtils.readFile2Str(new
FileInputStream(targetPath));
Review Comment:
## Potential input resource leak
This FileInputStream is not always closed on method exit.
[Show more
details](https://github.com/apache/dolphinscheduler/security/code-scanning/2456)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
