CVE-2023-26031: Privilege escalation in Apache Haoop Yarn container-executor binary on Linux systems
Severity: critical Affected versions: - Apache Hadoop 3.3.1 before 3.3.5 Description: Relative library resolution in linux container-executor binary in Apache Hadoop 3.3.1-3.3.4 on Linux allows local user to gain root privileges. If the YARN cluster is accepting work from remote (authenticated) users, this MAY permit remote users to gain root privileges. Hadoop 3.3.0 updated the " YARN Secure Containers https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/SecureContainer.html " to add a feature for executing user-submitted applications in isolated linux containers. The native binary HADOOP_HOME/bin/container-executor is used to launch these containers; it must be owned by root and have the suid bit set in order for the YARN processes to run the containers as the specific users submitting the jobs. The patch " YARN-10495 https://issues.apache.org/jira/browse/YARN-10495 . make the rpath of container-executor configurable" modified the library loading path for loading .so files from "$ORIGIN/" to ""$ORIGIN/:../lib/native/". This is the a path through which libcrypto.so is located. Thus it is is possible for a user with reduced privileges to install a malicious libcrypto library into a path to which they have write access, invoke the container-executor command, and have their modified library executed as root. If the YARN cluster is accepting work from remote (authenticated) users, and these users' submitted job are executed in the physical host, rather than a container, then the CVE permits remote users to gain root privileges. The fix for the vulnerability is to revert the change, which is done in YARN-11441 https://issues.apache.org/jira/browse/YARN-11441 , "Revert YARN-10495". This patch is in hadoop-3.3.5. To determine whether a version of container-executor is vulnerable, use the readelf command. If the RUNPATH or RPATH value contains the relative path "./lib/native/" then it is at risk $ readelf -d container-executor|grep 'RUNPATH\|RPATH' 0x001d (RUNPATH)Library runpath: [$ORIGIN/:../lib/native/] If it does not, then it is safe: $ readelf -d container-executor|grep 'RUNPATH\|RPATH' 0x001d (RUNPATH)Library runpath: [$ORIGIN/] For an at-risk version of container-executor to enable privilege escalation, the owner must be root and the suid bit must be set $ ls -laF /opt/hadoop/bin/container-executor ---Sr-s---. 1 root hadoop 802968 May 9 20:21 /opt/hadoop/bin/container-executor A safe installation lacks the suid bit; ideally is also not owned by root. $ ls -laF /opt/hadoop/bin/container-executor -rwxr-xr-x. 1 yarn hadoop 802968 May 9 20:21 /opt/hadoop/bin/container-executor This configuration does not support Yarn Secure Containers, but all other hadoop services, including YARN job execution outside secure containers continue to work. This issue is being tracked as YARN-11441 Required Configurations: The owner of the container-executor binary must be set to "root" and suid set bit such that callers would execute the binary as root. These operations are a requirement for "YARN Secure Containers". In an installation using the hadoop.tar.gz file the binary's owner is that of the installing user, and without the suid permission is not at risk. However, Apache BIgtop installations set the owner and permissions such that installations may be vulnerable The container-executor binary is only vulnerable on some Hadoop/Bigtop releases. It is possible to verify whether a version is vulnerable using the readelf command. Work Arounds: * Upgrade to Apache Hadoop 3.3.5 * If Yarn Secure Containers are not required, remove all execute permissions on bin/container-executor ; change its owner from root, or simply delete it. * If Yarn Secure Containers are required on a vulnerable release and upgrade is not possible, replace the container-executor binary with that of the 3.3.5 release. As most Hadoop installations do not use Yarn Secure Containers, removing execute permissions from the container-executor binary a is sufficient to secure the systems; deletion ensures that no security scanners will report the issue. Credit: Esa Hiltunen (finder) Mikko Kortelainen (finder) The Teragrep Project (sponsor) References: https://issues.apache.org/jira/browse/YARN-11441 https://hadoop.apache.org/ https://www.cve.org/CVERecord?id=CVE-2023-26031 - To unsubscribe, e-mail: general-unsubscr...@hadoop.apache.org For additional commands, e-mail: general-h...@hadoop.apache.org
CVE-2021-25642: Apache Hadoop YARN remote code execution in ZKConfigurationStore of capacity scheduler
Severity: important Versions affected: 2.9.0 to 2.10.1, 3.0.0-alpha to 3.2.3, 3.3.0 to 3.3.3 Description: ZKConfigurationStore which is optionally used by CapacityScheduler of Apache Hadoop YARN deserializes data obtained from ZooKeeper without validation. An attacker having access to ZooKeeper can run arbitrary commands as YARN user by exploiting this. Mitigation: Users should upgrade to Apache Hadoop 2.10.2, 3.2.4, 3.3.4 or later (containing YARN-11126) if ZKConfigurationStore is used. Credit: Apache Hadoop would like to thank Liu Ximing for reporting this issue. - To unsubscribe, e-mail: general-unsubscr...@hadoop.apache.org For additional commands, e-mail: general-h...@hadoop.apache.org
CVE-2022-25168: Apache Hadoop: Command injection in org.apache.hadoop.fs.FileUtil.unTarUsingTar
Severity: important Versions affected: 2.0.0 to 2.10.1, 3.0.0-alpha to 3.2.3, 3.3.0 to 3.3.2 Description: Apache Hadoop's FileUtil.unTar(File, File) API does not escape the input file name before being passed to the shell. An attacker can inject arbitrary commands. This is only used in Hadoop 3.3 InMemoryAliasMap.completeBootstrapTransfer, which is only ever run by a local user. It has been used in Hadoop 2.x for yarn localization, which does enable remote code execution. It is used in Apache Spark, from the SQL command ADD ARCHIVE. As the ADD ARCHIVE command adds new binaries to the classpath, being able to execute shell scripts does not confer new permissions to the caller. SPARK-38305. "Check existence of file before untarring/zipping", which is included in 3.3.0, 3.1.4, 3.2.2, prevents shell commands being executed, regardless of which version of the hadoop libraries are in use. Mitigation: Users should upgrade to Apache Hadoop 2.10.2, 3.2.4, 3.3.3 or upper (including HADOOP-18136). Credit: Apache Hadoop would like to thank Kostya Kortchinsky for reporting this issue. - To unsubscribe, e-mail: general-unsubscr...@hadoop.apache.org For additional commands, e-mail: general-h...@hadoop.apache.org
[ANNOUNCE] Apache Hadoop 3.2.4 release
On behalf of the Apache Hadoop Project Management Committee, I'm pleased to announce that Hadoop 3.2.4 has been released. This is the third stable release of Apache Hadoop 3.2 line. It contains 153 bug fixes, improvements and enhancements since 3.2.3. Users are encouraged to read the overview of major changes[1] since 3.2.3. For details of 153 bug fixes, improvements, and other enhancements since the previous 3.2.3 release, please check release notes[2] and changelog[3]. [1]: https://hadoop.apache.org/docs/r3.2.4/index.html [2]: https://hadoop.apache.org/docs/r3.2.4/hadoop-project-dist/hadoop-common/release/3.2.4/RELEASENOTES.3.2.4.html [3]: https://hadoop.apache.org/docs/r3.2.4/hadoop-project-dist/hadoop-common/release/3.2.4/CHANGELOG.3.2.4.html As the release notes highlight, this release contains HADOOP-18088 "Replace log4j 1.x with reload4j" https://issues.apache.org/jira/browse/HADOOP-18088 This ensures that the version of log4j shipped is free of known CVEs. the standard log4j 1.2.17 has some known CVEs in classes which were never uses; reload4j cuts them out. Audit scanning tools should stop highlighting perceived risks here. If you are using maven exclusions to manage logging libraries, or were otherwise replacing the log4j artifacts in deployments, note the different library/artifact names which need to be handled. Many thanks to everyone who helped in this release by supplying patches, reviewing them, helping get this release building and testing reviewing the final artifacts. Masatake Iwasaki - To unsubscribe, e-mail: general-unsubscr...@hadoop.apache.org For additional commands, e-mail: general-h...@hadoop.apache.org
CVE-2021-37404: Apache Hadoop: Heap buffer overflow in libhdfs native library
Severity: important Versions affected: 2.9.0 to 2.10.1, 3.0.0 to 3.1.4, 3.2.0 to 3.2.2, 3.3.0 to 3.3.1 Description: There is a potential heap buffer overflow in libhdfs native code. Opening a file path provided by user without validation may result in a denial of service or arbitrary code execution. Mitigation: Users should upgrade to Apache Hadoop 2.10.2, 3.2.3, 3.3.2 or upper. Credit: This issue was discovered by Igor Chervatyuk. - To unsubscribe, e-mail: general-unsubscr...@hadoop.apache.org For additional commands, e-mail: general-h...@hadoop.apache.org
[ANNOUNCE] Apache Hadoop 2.10.2 release
On behalf of the Apache Hadoop Project Management Committee, I'm pleased to announce that Hadoop 2.10.2 has been released. This is the second stable release of Apache Hadoop 2.10 line. It contains 211 bug fixes, improvements and enhancements since 2.10.1. Users are encouraged to read the overview of major changes[1] since 2.10.1. For details of 211 bug fixes, improvements, and other enhancements since the previous 2.10.1 release, please check release notes[2] and changelog[3]. [1]: https://hadoop.apache.org/docs/r2.10.2/index.html [2]: https://hadoop.apache.org/docs/r2.10.2/hadoop-project-dist/hadoop-common/release/2.10.2/RELEASENOTES.2.10.2.html [3]: https://hadoop.apache.org/docs/r2.10.2/hadoop-project-dist/hadoop-common/release/2.10.2/CHANGELOG.2.10.2.html As the release notes highlight, this release contains HADOOP-18088 "Replace log4j 1.x with reload4j" https://issues.apache.org/jira/browse/HADOOP-18088 This ensures that the version of log4j shipped is free of known CVEs. the standard log4j 1.2.17 has some known CVEs in classes which were never uses; reload4j cuts them out. Audit scanning tools should stop highlighting perceived risks here. If you are using maven exclusions to manage logging libraries, or were otherwise replacing the log4j artifacts in deployments, note the different library/artifact names which need to be handled. Many thanks to everyone who helped in this release by supplying patches, reviewing them, helping get this release building and testing reviewing the final artifacts. Masatake Iwasaki - To unsubscribe, e-mail: general-unsubscr...@hadoop.apache.org For additional commands, e-mail: general-h...@hadoop.apache.org
Re: [ANNOUNCE] CVE-2022-26612: Apache Hadoop: Arbitrary file write in FileUtil#unpackEntries on Windows
Hi Gautham,
Let me know about versions affected.
Is the version prior to 3.1.0 affected?
The cause seems to be the code introduced by HADOOP-15170.
I want to know this for releasing 2.10.2.
If branch-2.10 too is affected, I need to backport HADOOP-18155.
Thanks,
Masatake Iwasaki
On Mon, Apr 11, 2022 at 5:13 PM Gautham Banasandra wrote:
>
> Hi Akira,
>
>> What are the fixed versions?
>
> 3.2.3, all releases 3.4 onwards contain the fix.
>
>> What's the mitigation?
>
> Here are the possible mitigations (mutually exclusive of each other) in the
> order of effectiveness -
> 1. Upgrade to the fixed versions.
> 2. Do not run any of the YARN daemons in admin mode on Windows.
> 3. Do not use symlinks in tar.
>
>> Would you update the CVE list in https://hadoop.apache.org/cve_list.html ?
>
> Sure, I'll do it shortly.
>
> Thanks,
> --Gautham
>
> On Sat, 9 Apr 2022 at 13:46, Akira Ajisaka wrote:
>>
>> HI Gautham,
>>
>> Thank you for your announcement.
>> What are the fixed versions? What's the mitigation? Would you update the CVE
>> list in https://hadoop.apache.org/cve_list.html ?
>>
>> -Akira
>>
>>
>>
>> On Thu, Apr 7, 2022 at 11:36 PM Gautham Banasandra
>> wrote:
>>>
>>> The unTar function [1] uses unTarUsingJava function on Windows and the
>>> built-in tar utility on Unix and other OSes:
>>>
>>> if(Shell.WINDOWS) {
>>> // Tar is not native to Windows. Use simple Java based implementation for
>>> // tests and simple tar archives
>>> unTarUsingJava(inFile, untarDir, gzipped);
>>> }
>>> else {
>>> // spawn tar utility to untar archive for full fledged unix behavior such
>>> // as resolving symlinks in tar archives
>>> unTarUsingTar(inFile, untarDir, gzipped);
>>> }
>>>
>>> The function verifies that the extracted TAR entry is under the expected
>>> targetDirPath[2]:
>>>
>>> if (!outputFile.getCanonicalPath().startsWith(targetDirPath)) {
>>> throw new IOException("expanding " + entry.getName()
>>> + " would create entry outside of " + outputDir);
>>> }
>>>
>>> However it doesn't apply the same restriction to the target of an extracted
>>> symlink[3]:
>>>
>>> if (entry.isSymbolicLink()) {
>>> // Create symbolic link relative to tar parent dir
>>> Files.createSymbolicLink(FileSystems.getDefault()
>>> .getPath(outputDir.getPath(), entry.getName()),
>>> FileSystems.getDefault().getPath(entry.getLinkName()));
>>> return;
>>> }
>>>
>>> As a result, a TAR entry may create a symlink under the expected extraction
>>> directory which points to an external directory. A subsequent TAR entry may
>>> extract an arbitrary file into the external directory using the symlink
>>> name. This however would be caught by the same targetDirPath[4] check on
>>> Unix because of the getCanonicalPath call. However on Windows,
>>> getCanonicalPath doesn't resolve symbolic links, which bypasses the check.
>>>
>>> unpackEntries during TAR extraction follows symbolic links which allows
>>> writing outside expected base directory on Windows.
>>>
>>> [1]=
>>> https://github.com/apache/hadoop/blob/125e3b616040b4f98956aa946cc51e99f7d596c2/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java#L850
>>> [2]=
>>> https://github.com/apache/hadoop/blob/125e3b616040b4f98956aa946cc51e99f7d596c2/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java#L964-L967
>>> [3]=
>>> https://github.com/apache/hadoop/blob/125e3b616040b4f98956aa946cc51e99f7d596c2/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java#L983-L989
>>> [4]=
>>> https://github.com/apache/hadoop/blob/125e3b616040b4f98956aa946cc51e99f7d596c2/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java#L964-L967
>>>
>>> Credit:
>>>
>>> This issue was reported by a member of GitHub Security Lab, Jaroslav
>>> Lobačevski (https://github.com/JarLob).
>>>
>>> References:
>>>
>>> https://lists.apache.org/thread/hslo7wzw2449gv1jyjk8g6ttd7935fyz
-
To unsubscribe, e-mail: general-unsubscr...@hadoop.apache.org
For additional commands, e-mail: general-h...@hadoop.apache.org
[ANNOUNCE] Apache Hadoop 3.2.3 release
Hi all, It gives me great pleasure to announce that the Apache Hadoop community has released Apache Hadoop 3.2.3. This is the third stable release of Apache Hadoop 3.2 line. It contains 328 bug fixes, improvements and enhancements since 3.2.2. For details of bug fixes, improvements, and other enhancements since the previous 3.2.2 release, please check release notes[1] and changelog[2]. [1]: https://hadoop.apache.org/docs/r3.2.3/hadoop-project-dist/hadoop-common/release/3.2.3/RELEASENOTES.3.2.3.html [2]: https://hadoop.apache.org/docs/r3.2.3/hadoop-project-dist/hadoop-common/release/3.2.3/CHANGELOG.3.2.3.html Best Regards, Masatake Iwasaki - To unsubscribe, e-mail: general-unsubscr...@hadoop.apache.org For additional commands, e-mail: general-h...@hadoop.apache.org
Re: [ANNOUNCE] Apache Hadoop 2.10.1 release
Hi all, It was pointed out that the https://hadoop.apache.org/docs/r2.10.1/index.html contains old contents due to missing updates on my creation of the RC. # Thanks, Jonathan Hung. I would like to leave it as is since the docs are part of signed artifacts. Other pages including RELEASENOTES and CHANGES have valid contents. Thanks, Masatake Iwasaki On 2020/09/23 5:17, Masatake Iwasaki wrote: Hi all, I am happy to announce that the Apache Hadoop 2.10.1 has been released. Apache Hadoop 2.10.1 is the second release in the Apache Hadoop 2.10 line. You can download the release from the Download page [1]. The release details, including links to list of major features, release notes, and changelog, are on the 2.10.1 announcement page [2]. [1] https://hadoop.apache.org/releases.html [2] https://hadoop.apache.org/release/2.10.1.html Thanks, Masatake Iwasaki - To unsubscribe, e-mail: general-unsubscr...@hadoop.apache.org For additional commands, e-mail: general-h...@hadoop.apache.org - To unsubscribe, e-mail: general-unsubscr...@hadoop.apache.org For additional commands, e-mail: general-h...@hadoop.apache.org
[ANNOUNCE] Apache Hadoop 2.10.1 release
Hi all, I am happy to announce that the Apache Hadoop 2.10.1 has been released. Apache Hadoop 2.10.1 is the second release in the Apache Hadoop 2.10 line. You can download the release from the Download page [1]. The release details, including links to list of major features, release notes, and changelog, are on the 2.10.1 announcement page [2]. [1] https://hadoop.apache.org/releases.html [2] https://hadoop.apache.org/release/2.10.1.html Thanks, Masatake Iwasaki - To unsubscribe, e-mail: general-unsubscr...@hadoop.apache.org For additional commands, e-mail: general-h...@hadoop.apache.org
Re: [ANNOUNCE] New Hadoop Committer - Masatake Iwasaki
Thank you, everyone!
Masatake
On 1/29/16 03:19, Neelesh Salian wrote:
Congratulations Masatake!
On Wed, Jan 27, 2016 at 6:37 PM, Tsuyoshi OZAWA
<ozawa.tsuyo...@lab.ntt.co.jp <mailto:ozawa.tsuyo...@lab.ntt.co.jp>>
wrote:
On behalf of the Apache Hadoop PMC, I am pleased to announce that
Masatake Iwasaki has been elected as a committer on the Apache Hadoop
project. We appreciate all of Masatake's hard work thus far, and we
look forward to his continued contributions.
Welcome Masatake!
Regards,
- Tsuyoshi
--
Neelesh Srinivas Salian
Customer Operations Engineer
*
*
*
*
Re: [ANNOUNCE] New Hadoop Committer - Sangjin Lee
Congratulations, Sanjin Lee! Masatake Iwasaki On 9/30/15 20:17, Karthik Kambatla wrote: [bcc: general@, please bcc: (and not cc:) general@ if you want to include] On behalf of the Apache Hadoop PMC, I am very pleased to announce that Sanjin Lee has been elected a committer on the Apache Hadoop project recognizing his continued contributions to the project. Sangjin is already a branch committer on YARN-2928 and can hit the ground running :) Welcome Sangjin! Cheers!
Re: [ANNOUNCE] New Hadoop Committer - Ming Ma
Congratulations, Ming Ma! On 6/18/15 12:55, Chris Nauroth wrote: On behalf of the Apache Hadoop PMC, I am pleased to announce that Ming Ma has been elected as a committer on the Apache Hadoop project. We appreciate all of Ming's hard work thus far, and we look forward to his continued contributions. Welcome, Ming! --Chris Nauroth
Re: [ANNOUNCE] New Hadoop committer - Varun Vasudev
Congratulations, Varun Vasudev! On 6/16/15 09:54, Vinod Kumar Vavilapalli wrote: Hi all, It gives me great pleasure to announce that the Apache Hadoop PMC recently invited Varun Vasudev to become a committer in the project, to which he accepted. We deeply appreciate his efforts in the project so far, specifically in the areas of YARN and MapReduce. Here’s looking forward to his continued contributions going into the future! Welcome aboard and congratulations, Varun! +Vinod On behalf of the Apache Hadoop PMC
Re: [ANNOUNCE] New Hadoop committer - Lei (Eddy) Xu
Congratulations, Lei Xu! On 6/15/15 14:50, Andrew Wang wrote: Hello all, It is my pleasure to announce that Lei Xu, also known as Eddy, has accepted the Apache Hadoop PMC's invitation to become a committer. We appreciate all of Eddy's hard work thus far, and look forward to his continued contributions. Welcome and congratulations, Eddy! Best, Andrew, on behalf of the Apache Hadoop PMC
Re: [ANNOUNCE] New Hadoop Committers - Akira Ajisaka and Tsuyoshi Ozawa
Congratulations, Akira and Tsuyoshi! Masatake Iwasaki (12/18/14, 0:55), Steve Loughran wrote: On behalf of the Apache Hadoop PMC, I am pleased to announce that Akira Ajisaka and Tsuyoshi Ozawa have been elected as committers on the Apache Hadoop project. We appreciate all the work they have put into the project so far, and look forward to their future contributions. Welcome, Akira and Tsuyoshi! -Steve
