[
https://issues.apache.org/jira/browse/FLINK-36694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18016384#comment-18016384
]
Mohsen Rezaei edited comment on FLINK-36694 at 8/26/25 9:18 PM:
----------------------------------------------------------------
With Hadoop 3.4 moving to the new AWS SDK APIs with breaking changes without
bumping major, updating minor versions became more challenging, blocking
updates for bringing in new features and improvements to the file system
extensions.
I started a new PR with a new pattern to allow file systems to move to newer
Hadoop versions when possible, and with Azure's latest implementation of Hadoop
becoming more and more the standard out there, e.g. WI Auth mechanism, I've
migrated its FS extension to Hadoop 3.4.
[~martijnvisser] ptal when you get a chance:
[https://github.com/apache/flink/pull/26944]
was (Author: JIRAUSER298550):
With Hadoop 3.4 moving to new AWS SDK APIs with breaking changes without
bumping major, updating minor versions became more challenging, blocking
updates for bringing in new features and improvements to the file system
extensions.
I started a new PR with a new pattern to allow file systems to move to newer
Hadoop versions when possible, and with Azure's latest implementation of Hadoop
becoming more and more the standard out there, e.g. WI Auth mechanism, I've
migrated its FS extension to Hadoop 3.4.
[~martijnvisser] ptal when you get a chance:
https://github.com/apache/flink/pull/26944
> Update Hadoop shaded from 3.3.4 to 3.4.1
> ----------------------------------------
>
> Key: FLINK-36694
> URL: https://issues.apache.org/jira/browse/FLINK-36694
> Project: Flink
> Issue Type: Improvement
> Components: FileSystems
> Affects Versions: 1.20.0
> Reporter: Rasmus Thygesen
> Priority: Major
> Labels: pull-request-available
>
> I believe the Hadoop version for the filesystem plugins is currently 3.3.4
> [https://github.com/apache/flink/blob/master/flink-filesystems/pom.xml#L37]
> I would like to update it to the latest version, 3.4.1, with the following
> release notes
> [https://hadoop.apache.org/docs/r3.4.1/hadoop-project-dist/hadoop-common/release/3.4.1/RELEASENOTES.3.4.1.html]
>
> I would be more than happy to do it myself, but then i would need to know if
> theres anything i need to change apart from the version in
> [https://hadoop.apache.org/docs/r3.4.1/hadoop-project-dist/hadoop-common/release/3.4.1/RELEASENOTES.3.4.1.html]
> Otherwise, anyone is welcome to fix this issue
>
> I am using the filesystem through docker which gets the filesystem plugins
> from
> [https://dlcdn.apache.org/flink/flink-1.20.0/flink-1.20.0-bin-scala_2.12.tgz]
> does anything else need to be done to get that working? Im guessing a new
> release will update the TGZ automatically
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
