[
https://issues.apache.org/jira/browse/HADOOP-20031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18135344#comment-18135344
]
ASF GitHub Bot commented on HADOOP-20031:
-----------------------------------------
GauthamBanasandra commented on code in PR #8844:
URL: https://github.com/apache/hadoop/pull/8844#discussion_r4426617820
##########
dev-support/docker/Dockerfile_windows_10:
##########
@@ -50,54 +50,56 @@ RUN powershell Copy-Item -Recurse -Path 'C:\Program
Files\Git' -Destination C:\G
RUN powershell git clone https://github.com/microsoft/vcpkg.git \
&& cd vcpkg \
&& git fetch --all \
- && git checkout 2025.03.19 \
+ && git checkout 2026.07.29 \
&& .\bootstrap-vcpkg.bat
ADD vcpkg/vcpkg.json .
RUN powershell .\vcpkg\vcpkg.exe install --x-install-root .\vcpkg\installed
-# Install Azul Java 8 JDK.
-RUN powershell Invoke-WebRequest -URI
https://cdn.azul.com/zulu/bin/zulu8.62.0.19-ca-jdk8.0.332-win_x64.zip -OutFile
$Env:TEMP\zulu8.62.0.19-ca-jdk8.0.332-win_x64.zip
-RUN powershell Expand-Archive -Path
$Env:TEMP\zulu8.62.0.19-ca-jdk8.0.332-win_x64.zip -DestinationPath "C:\Java"
+# Install Microsoft Build of OpenJDK 17.
+RUN powershell Invoke-WebRequest -URI
https://aka.ms/download-jdk/microsoft-jdk-17.0.20.1-windows-x64.msi -OutFile
C:\jdk17.msi
+RUN powershell Start-Process msiexec.exe -ArgumentList
'/i','C:\jdk17.msi','INSTALLDIR=C:\Java\jdk17\','/quiet','/norestart' -Wait
+RUN powershell if (-not (Test-Path 'C:\Java\jdk17\bin\javac.exe')) { throw
'JDK 17 installation failed' }
+RUN powershell Remove-Item C:\jdk17.msi -Force
# Install Apache Maven.
RUN powershell Invoke-WebRequest -URI
https://archive.apache.org/dist/maven/maven-3/3.9.15/binaries/apache-maven-3.9.15-bin.zip
-OutFile $Env:TEMP\apache-maven-3.9.15-bin.zip
RUN powershell Expand-Archive -Path $Env:TEMP\apache-maven-3.9.15-bin.zip
-DestinationPath "C:\Maven"
-# Install CMake 3.19.0.
-RUN powershell Invoke-WebRequest -URI
https://cmake.org/files/v3.19/cmake-3.19.0-win64-x64.zip -OutFile
$Env:TEMP\cmake-3.19.0-win64-x64.zip
-RUN powershell Expand-Archive -Path $Env:TEMP\cmake-3.19.0-win64-x64.zip
-DestinationPath "C:\CMake"
+# Install CMake 3.30.2.
+RUN powershell Invoke-WebRequest -URI
https://cmake.org/files/v3.30/cmake-3.30.2-windows-x86_64.zip -OutFile
$Env:TEMP\cmake-3.30.2-windows-x86_64.zip
+RUN powershell Expand-Archive -Path $Env:TEMP\cmake-3.30.2-windows-x86_64.zip
-DestinationPath "C:\CMake"
# Install zstd 1.5.4.
RUN powershell Invoke-WebRequest -Uri
https://github.com/facebook/zstd/releases/download/v1.5.4/zstd-v1.5.4-win64.zip
-OutFile $Env:TEMP\zstd-v1.5.4-win64.zip
RUN powershell Expand-Archive -Path $Env:TEMP\zstd-v1.5.4-win64.zip
-DestinationPath "C:\ZStd"
RUN setx PATH "%PATH%;C:\ZStd"
-# Install libopenssl 3.5.2-1 needed for rsync 3.2.7.
+# Install libopenssl 3.5.2-1 needed for rsync 3.5.1.
RUN powershell Invoke-WebRequest -Uri
https://repo.msys2.org/msys/x86_64/libopenssl-3.5.2-1-x86_64.pkg.tar.zst
-OutFile $Env:TEMP\libopenssl-3.5.2-1-x86_64.pkg.tar.zst
RUN powershell zstd -d $Env:TEMP\libopenssl-3.5.2-1-x86_64.pkg.tar.zst -o
$Env:TEMP\libopenssl-3.5.2-1-x86_64.pkg.tar
RUN powershell mkdir "C:\LibOpenSSL"
RUN powershell tar -xvf $Env:TEMP\libopenssl-3.5.2-1-x86_64.pkg.tar -C
"C:\LibOpenSSL"
-# Install libxxhash 0.8.3 needed for rsync 3.2.7.
+# Install libxxhash 0.8.3 needed for rsync 3.5.1.
RUN powershell Invoke-WebRequest -Uri
https://repo.msys2.org/msys/x86_64/libxxhash-0.8.3-1-x86_64.pkg.tar.zst
-OutFile $Env:TEMP\libxxhash-0.8.3-1-x86_64.pkg.tar.zst
RUN powershell zstd -d $Env:TEMP\libxxhash-0.8.3-1-x86_64.pkg.tar.zst -o
$Env:TEMP\libxxhash-0.8.3-1-x86_64.pkg.tar
RUN powershell mkdir "C:\LibXXHash"
RUN powershell tar -xvf $Env:TEMP\libxxhash-0.8.3-1-x86_64.pkg.tar -C
"C:\LibXXHash"
-# Install libzstd 1.5.4 needed for rsync 3.2.7.
-RUN powershell Invoke-WebRequest -Uri
https://repo.msys2.org/msys/x86_64/libzstd-1.5.5-1-x86_64.pkg.tar.zst -OutFile
$Env:TEMP\libzstd-1.5.5-1-x86_64.pkg.tar.zst
-RUN powershell zstd -d $Env:TEMP\libzstd-1.5.5-1-x86_64.pkg.tar.zst -o
$Env:TEMP\libzstd-1.5.5-1-x86_64.pkg.tar
+# Install libzstd 1.5.7 needed for rsync 3.5.1.
+RUN powershell Invoke-WebRequest -Uri
https://repo.msys2.org/msys/x86_64/libzstd-1.5.7-1-x86_64.pkg.tar.zst -OutFile
$Env:TEMP\libzstd-1.5.7-1-x86_64.pkg.tar.zst
Review Comment:
Thanks for the review @pan3793.
You're right about HADOOP-19855, and it's already on trunk beneath this
branch — there's no
zstd left in any CMakeLists.txt, config.h.cmake or pom other than
`zstd-jni`, and nothing
consumes `-Drequire.zstd` any more. This PR doesn't re-add it: no zstd in
`vcpkg.json`, and
the mvn command in BUILDING.txt only passes `-Drequire.openssl
-Drequire.test.libhadoop`.
This line is a different thing though — it's the MSYS2 `libzstd` package,
which ships just
`/usr/bin/msys-zstd-1.dll` and is a declared runtime dependency of MSYS2's
`rsync.exe`
(packages.msys2.org lists rsync 3.5.1-2's deps as libiconv, libidn2, liblz4,
libopenssl,
libxxhash, libzstd, popt). Line 105 copies it next to `rsync.exe` for that
reason, and
`create-release` depends on rsync working.
The separate `C:\ZStd` CLI has to stay too: `zstd -d` unpacks all four
`.pkg.tar.zst`
archives, including libopenssl and libxxhash, which aren't zstd-related at
all.
These lines are only in the diff because repo.msys2.org serves only the
current version of
each package, so trunk's pinned URLs now 404 and the image can't build at
all. I've added a
comment at the zstd step noting it's an unpacking tool rather than the
codec, to make this
less confusing next time.
> Fix the Windows native build: Docker image, libhdfspp CMake and YARN UI
> -----------------------------------------------------------------------
>
> Key: HADOOP-20031
> URL: https://issues.apache.org/jira/browse/HADOOP-20031
> Project: Hadoop Common
> Issue Type: Bug
> Components: hadoop-common
> Affects Versions: 3.5.0
> Environment: Windows 10/11
> Reporter: Gautham Banasandra
> Assignee: Gautham Banasandra
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0
>
>
> h2. Problem
> {{mvn clean install}} with {{-Pnative-win}} cannot complete on Windows. The
> Docker image described in BUILDING.txt no longer builds at all, and even with
> a
> working image the native build fails in libhdfspp and again in the YARN
> capacity-scheduler UI. There is no single cause -- eight separate defects have
> accumulated, each of which independently stops the build.
> h2. Root causes
> h3. The build image no longer builds
> # {{Dockerfile_windows_10}} installs Azul JDK 8. Hadoop requires JDK 17, so
> the
> enforcer rejects the build immediately. This line has not been touched since
> 2022.
> # {{vcpkg install}} fails because the pinned {{protobuf 3.25.5}} cannot be
> resolved from vcpkg's version database. The pin was added by HADOOP-19605 on
> 2025-11-10, so the Windows image has been unbuildable for roughly eleven
> months.
> # The pinned vcpkg tag {{2025.03.19}} references MSYS2 packages
> ({{msys2-runtime-3.5.4-2}}) that have since been purged from the upstream
> mirror. MSYS2 keeps only current versions.
> # The {{libzstd}} and {{rsync}} MSYS2 pins in the Dockerfile have been purged
> from the same mirror for the same reason.
> h3. The native build fails
> # Boost is not pinned explicitly -- its version comes from
> {{builtin-baseline}}, so it floats as the baseline moves. Boost 1.87 removed
> {{boost/asio/io_service.hpp}}, which five libhdfspp files still include
> (ioservice.h, ioservice_impl.h, mock_connection.{h,cc},
> remote_block_reader_test.cc).
> # The protobuf/protoc compatibility check in
> {{libhdfspp/CMakeLists.txt}} leaves {{CMAKE_REQUIRED_LIBRARIES}} pointing at
> the Abseil targets. Abseil carries C++ feature requirements that a later
> C-language {{check_symbol_exists()}} cannot satisfy under the Visual Studio
> generator, so that check fails spuriously.
> # The legacy {{PROTOBUF_LIBRARY}}, {{PROTOBUF_PROTOC_LIBRARY}} and
> {{PROTOBUF_LIBRARIES}} variables resolve to import DLLs under vcpkg, which
> MSVC rejects with {{LNK1107}}.
> # {{hadoop-yarn-capacity-scheduler-ui}} sets {{packageManagerScript}} to
> {{node_modules/.bin/react-router}}. On Windows that path is an {{sh}} shim,
> which {{node}} cannot execute.
> h2. Why Linux is unaffected
> The Linux build *pins* its native dependencies inside the repository --
> {{dev-support/docker/install-boost.sh}} hard-codes Boost 1.86.0 and
> {{install-protobuf.sh}} hard-codes Protobuf 3.25.5, both built from source.
> The
> Windows build instead delegates to vcpkg and lets the versions float with the
> baseline. That asymmetry is the root cause of defects 2, 5 and arguably 3: the
> Windows build has no equivalent of the version lock the Linux build has had
> all
> along.
> h2. Fix
> * Dockerfile: install the Microsoft Build of OpenJDK 17; move to vcpkg tag
> {{2026.07.29}}; refresh the {{libzstd}} and {{rsync}} pins; refresh CMake to
> 3.30.2. Switch the base image to {{windows/servercore}}, which carries
> everything the build needs at a fraction of the size, and relax the password
> policy that Server Core enables by default so the existing passwordless
> {{HadoopBuilder}} account can still be created.
> * {{vcpkg.json}}: narrow the Boost dependency from the {{boost}} metapackage
> to
> the four ports libhdfspp actually includes (asio, date-time,
> program-options,
> system), and pin all 55 transitively required Boost ports to 1.86.0 -- the
> same version {{install-boost.sh}} pins for Linux. Pin protobuf to a version
> vcpkg publishes.
> * {{libhdfspp}}: reset {{CMAKE_REQUIRED_INCLUDES}} and
> {{CMAKE_REQUIRED_LIBRARIES}} after the protoc compatibility check, and
> migrate from the legacy {{PROTOBUF_*}} variables to the
> {{protobuf::libprotobuf}} and {{protobuf::libprotoc}} imported targets.
> * {{hadoop-yarn-capacity-scheduler-ui}}: point {{packageManagerScript}} at the
> real entry point, {{node_modules/@react-router/dev/bin.js}}.
> * BUILDING.txt: correct the Windows Docker section. It omitted that Docker
> Desktop must be switched to Windows containers; it documented the Maven
> repository mount under {{ContainerAdministrator}} although the Dockerfile
> ends with {{USER HadoopBuilder}}, so the documented mount is silently
> ignored; it recommended {{docker cp}}, which fails against a running Hyper-V
> isolated container; it omitted the container memory the build needs; and the
> manual vcpkg route still pinned the purged {{2025.03.19}} tag.
> h2. Testing
> Verified with a single clean {{mvn clean install}} inside the Windows
> container,
> from a fresh trunk clone with these changes applied:
> * *120 of 120 modules SUCCESS*, no failures and no skipped modules
> * 48:59 min, against a warm Maven repository
> * Produces {{winutils.exe}}, {{hadoop.dll}} and {{hdfs.dll}}, present both in
> {{hadoop-dist/target/hadoop-3.6.0-SNAPSHOT/bin}} and inside the distribution
> tarball
> * {{winutils.exe systeminfo}}, {{ls}}, {{chmod}} and {{hardlink}} all run
> correctly against the produced binary
> No new tests are added: these are build, packaging and documentation changes,
> and the test is that the build completes and emits working native binaries.
> h2. Notes
> While investigating, {{-Dcmake.prefix.path}} was found to be read by nothing.
> The only reader in the tree is {{${windows.cmake.prefix.path}}} in
> {{hadoop-hdfs-native-client/pom.xml}}, and neither property is defined in any
> pom, so the flag documented in BUILDING.txt left {{CMAKE_PREFIX_PATH}} holding
> an unexpanded literal. BUILDING.txt is corrected here; the same misspelling
> also
> appears in {{dev-support/bin/hadoop.sh}} and may deserve a separate issue.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
