AHeise commented on a change in pull request #10952:
[FLINK-15777][filesystem]Determine Hadoop version with fs classpath.
URL: https://github.com/apache/flink/pull/10952#discussion_r371301769
##########
File path:
flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableFsDataOutputStream.java
##########
@@ -175,44 +172,39 @@ private static void safelyTruncateFile(
}
}
- private static void ensureTruncateInitialized() throws
FlinkRuntimeException {
- if (HadoopUtils.isMinHadoopVersion(2, 7) && truncateHandle ==
null) {
- Method truncateMethod;
+ private static Method findTrunacteMethod(FileSystem fs) throws
FlinkRuntimeException {
+ Method truncateMethod = null;
+ if
(HadoopUtils.isMinHadoopVersion(fs.getClass().getClassLoader(), 2, 7)) {
try {
- truncateMethod =
FileSystem.class.getMethod("truncate", Path.class, long.class);
- }
- catch (NoSuchMethodException e) {
+ truncateMethod =
fs.getClass().getMethod("truncate", Path.class, long.class);
+ } catch (NoSuchMethodException e) {
throw new FlinkRuntimeException("Could not find
a public truncate method on the Hadoop File System.");
}
-
- if (!Modifier.isPublic(truncateMethod.getModifiers())) {
- throw new FlinkRuntimeException("Could not find
a public truncate method on the Hadoop File System.");
- }
-
- truncateHandle = truncateMethod;
}
+
+ return truncateMethod;
}
- private static boolean truncate(final FileSystem hadoopFs, final Path
file, final long length) throws IOException {
- if (!HadoopUtils.isMinHadoopVersion(2, 7)) {
- throw new IllegalStateException("Truncation is not
available in hadoop version < 2.7 , You are on Hadoop " +
VersionInfo.getVersion());
+ private static boolean truncate(
Review comment:
Yes, that's what I'd wanted to do but unfortunately,
`HadoopRecoverableWriter` accesses the static
`HadoopRecoverableFsDataOutputStream.HadoopFsCommitter` which in turn invokes
the truncate.
We could factor out a common TruncateWrapper though.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
