Copilot commented on code in PR #11823:
URL: https://github.com/apache/seatunnel/pull/11823#discussion_r3811674103
##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/hadoop/HadoopFileSystemProxy.java:
##########
@@ -469,6 +469,69 @@ private void initializeWithKerberosLogin() throws
IOException, InterruptedExcept
log.info("Create FileSystem success with Kerberos: {}.",
hadoopConf.getKerberosPrincipal());
}
+ /**
+ * Creates the Hadoop {@link FileSystem} for the given configuration,
rewriting a classpath
+ * mismatch into a diagnosable failure.
+ *
+ * <p>{@code FileSystem.get()} constructs a {@code DFSClient}, which
reaches into Hadoop
+ * internals (e.g. {@code FsTracer}) whose method signatures have changed
across Hadoop
+ * releases. If the Hadoop client jars actually resolved at runtime (this
dependency is {@code
+ * provided}, so it comes from the deployment environment, not this
project's shaded jar) don't
+ * match what SeaTunnel was built against, that surfaces as a raw {@link
LinkageError} deep in
+ * Hadoop's own code with no indication of the real cause.
+ *
+ * <p>Nothing about that failure is specific to one authentication mode,
so all three {@code
+ * FileSystem.get()} call sites in this class -- the plain path, the
Kerberos path and the
+ * remote-user path -- are routed through here.
+ *
+ * @param configuration Hadoop configuration to build the filesystem from
+ * @return the filesystem for {@code configuration}
+ * @throws IOException if the filesystem cannot be created, including the
diagnostic rewrite of
+ * a Hadoop client version mismatch
+ */
+ private static FileSystem getFileSystemWithDiagnostics(Configuration
configuration)
+ throws IOException {
+ return wrapClasspathMismatch(() -> FileSystem.get(configuration));
+ }
+
+ @FunctionalInterface
+ interface FileSystemSupplier {
+ FileSystem get() throws IOException;
+ }
+
+ /**
+ * Invokes {@code supplier} and rewrites a {@link LinkageError} raised
while loading or linking
+ * Hadoop classes into an {@link IOException} that names the likely Hadoop
client version
+ * mismatch, keeping the original error as the cause.
+ *
+ * <p>Only {@code LinkageError} is caught -- {@code NoSuchMethodError},
{@code
+ * NoClassDefFoundError}, {@code NoSuchFieldError} and {@code
IncompatibleClassChangeError} are
+ * all plausible symptoms of the same jar mismatch -- so unrelated errors
such as {@code
+ * OutOfMemoryError} still propagate untouched, as does an {@link
IOException} raised by the
+ * supplier itself.
+ *
+ * @param supplier filesystem acquisition to guard
+ * @return whatever {@code supplier} returns
+ * @throws IOException the supplier's own {@code IOException}, or a
diagnostic one wrapping a
+ * {@code LinkageError}
+ */
+ static FileSystem wrapClasspathMismatch(FileSystemSupplier supplier)
throws IOException {
+ try {
+ return supplier.get();
+ } catch (LinkageError e) {
+ throw new IOException(
+ "Failed to initialize Hadoop FileSystem, likely due to a
Hadoop client "
+ + "version mismatch between SeaTunnel and the
Hadoop jars resolved "
+ + "at runtime (the Hadoop dependency here is
`provided`, so it comes "
+ + "from your deployment environment, not a bundled
version). This "
+ + "connector is built and tested against Hadoop
3.1.4 (see "
+ + "seatunnel-hadoop3-3.1.4-uber in the reactor
pom); check that the "
+ + "Hadoop client jars on the classpath are
compatible with that "
+ + "version.",
+ e);
Review Comment:
The message hardcodes the target Hadoop version (`3.1.4`) and an internal
artifact name; this will become stale the next time the connector’s Hadoop
baseline changes. Consider centralizing the target Hadoop version into a single
constant (or a build-time injected property) and referencing that constant here
so updates don’t require hunting through strings.
##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/hadoop/HadoopFileSystemProxy.java:
##########
@@ -469,6 +469,69 @@ private void initializeWithKerberosLogin() throws
IOException, InterruptedExcept
log.info("Create FileSystem success with Kerberos: {}.",
hadoopConf.getKerberosPrincipal());
}
+ /**
+ * Creates the Hadoop {@link FileSystem} for the given configuration,
rewriting a classpath
+ * mismatch into a diagnosable failure.
+ *
+ * <p>{@code FileSystem.get()} constructs a {@code DFSClient}, which
reaches into Hadoop
+ * internals (e.g. {@code FsTracer}) whose method signatures have changed
across Hadoop
+ * releases. If the Hadoop client jars actually resolved at runtime (this
dependency is {@code
+ * provided}, so it comes from the deployment environment, not this
project's shaded jar) don't
+ * match what SeaTunnel was built against, that surfaces as a raw {@link
LinkageError} deep in
+ * Hadoop's own code with no indication of the real cause.
+ *
+ * <p>Nothing about that failure is specific to one authentication mode,
so all three {@code
+ * FileSystem.get()} call sites in this class -- the plain path, the
Kerberos path and the
+ * remote-user path -- are routed through here.
+ *
+ * @param configuration Hadoop configuration to build the filesystem from
+ * @return the filesystem for {@code configuration}
+ * @throws IOException if the filesystem cannot be created, including the
diagnostic rewrite of
+ * a Hadoop client version mismatch
+ */
+ private static FileSystem getFileSystemWithDiagnostics(Configuration
configuration)
+ throws IOException {
+ return wrapClasspathMismatch(() -> FileSystem.get(configuration));
+ }
+
+ @FunctionalInterface
+ interface FileSystemSupplier {
+ FileSystem get() throws IOException;
+ }
+
+ /**
+ * Invokes {@code supplier} and rewrites a {@link LinkageError} raised
while loading or linking
+ * Hadoop classes into an {@link IOException} that names the likely Hadoop
client version
+ * mismatch, keeping the original error as the cause.
+ *
+ * <p>Only {@code LinkageError} is caught -- {@code NoSuchMethodError},
{@code
+ * NoClassDefFoundError}, {@code NoSuchFieldError} and {@code
IncompatibleClassChangeError} are
+ * all plausible symptoms of the same jar mismatch -- so unrelated errors
such as {@code
+ * OutOfMemoryError} still propagate untouched, as does an {@link
IOException} raised by the
+ * supplier itself.
+ *
+ * @param supplier filesystem acquisition to guard
+ * @return whatever {@code supplier} returns
+ * @throws IOException the supplier's own {@code IOException}, or a
diagnostic one wrapping a
+ * {@code LinkageError}
+ */
+ static FileSystem wrapClasspathMismatch(FileSystemSupplier supplier)
throws IOException {
+ try {
+ return supplier.get();
+ } catch (LinkageError e) {
+ throw new IOException(
+ "Failed to initialize Hadoop FileSystem, likely due to a
Hadoop client "
+ + "version mismatch between SeaTunnel and the
Hadoop jars resolved "
+ + "at runtime (the Hadoop dependency here is
`provided`, so it comes "
+ + "from your deployment environment, not a bundled
version). This "
+ + "connector is built and tested against Hadoop
3.1.4 (see "
+ + "seatunnel-hadoop3-3.1.4-uber in the reactor
pom); check that the "
+ + "Hadoop client jars on the classpath are
compatible with that "
+ + "version.",
Review Comment:
The rewritten `IOException` message doesn’t include the specific linkage
failure (error class + original message). Including something like the
`LinkageError`’s concrete type and message (while still keeping it as the
cause) would make the diagnostic more actionable from logs where the cause
stacktrace might be truncated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
