Gautham Banasandra created HADOOP-20087:
-------------------------------------------
Summary: Pin the Windows build image's base to ltsc2022 and slim
it for CI
Key: HADOOP-20087
URL: https://issues.apache.org/jira/browse/HADOOP-20087
Project: Hadoop Common
Issue Type: Task
Components: hadoop-common
Affects Versions: 3.5.0
Reporter: Gautham Banasandra
Assignee: Gautham Banasandra
Fix For: 3.5.0
{{dev-support/docker/Dockerfile_windows_10}} builds from
{{servercore:ltsc2019}} by tag, and the tag moves with every monthly update.
GitHub's windows-2022 and windows-2025 runners can run ltsc2019 only under
Hyper-V isolation. In fork runs, Hyper-V isolation made the Maven build 1.07x
to 1.46x slower than process isolation, depending on the runner's VM type
(measured by forcing Hyper-V on the ltsc2022 image). This change prepares the
image for a GitHub Actions Windows build (HADOOP-19857):
* Make the base a build argument, defaulting to {{servercore:ltsc2022}} pinned
by digest, so the default base changes only when the Dockerfile does. The pin
is 10.0.20348.5761 (October), the base that the fork runs below tested. The
Dockerfile notes to bump the digest monthly, no sooner than 7 days after
Microsoft publishes the image (SECURITY.md requires a cooldown;
{{.github/dependabot.yml}} uses 7 days).
* Check that the Visual Studio Build Tools install worked: {{vcvars64.bat}}
runs, {{cl.exe}} is found and Windows SDK 10.0.19041 is present. Today the
install step passes even when the installer fails: {{start /w}} reports success
to {{||}} whatever the installer returns (it only sets {{%ERRORLEVEL%}}), so
its {{IF "%ERRORLEVEL%"=="3010"}} test never runs.
* Delete vcpkg's build trees, downloads and packages in the same layer as the
install, and turn off its binary cache, which kept a second copy of every
package. On windows-2022 with the ltsc2022 base, the image shrinks from about
24.4 GB to 16.9 GB, and from about 8.8 GB to 6.6 GB compressed. Recent runs of
the Jenkins Windows nightly {{hadoop-qbt-trunk-java8-win10-x86_64}} (#1471,
#1477 to #1479) failed in {{docker build}} with "There is not enough space on
the disk"; #1479 ran out while writing vcpkg's build trees into the image layer.
* Create {{C:\tmp}}, writable by the Users group. Tests that use
{{file:///tmp}} resolve it to {{C:\tmp}}; on a host without it,
{{TestPathData#testWithStringAndConfForBuggyPath}} fails, as it did in a fork
run.
* {{dev-support/bin/hadoop.sh}} passes {{\-Dcmake.prefix.path}} on Windows, but
hadoop-hdfs-native-client reads {{windows.cmake.prefix.path}}. Pass the
property it reads. This affects only the Yetus build that the Jenkins Windows
jobs run, and matches what BUILDING.txt documents.
* Stop changes to the Windows Dockerfile and {{dev-support/docker/vcpkg}} from
rebuilding the Linux image caches (the {{build_image_cache.yml}} path filter).
No Linux Dockerfile reads them.
* Update the Windows section of BUILDING.txt: which hosts need the ltsc2019
build argument, Docker Engine on Windows Server, which isolation to use on
client and server hosts, when {{\-m 8g}} is needed, and the expected image size
(about 17 GB, was 23 GB).
*Behaviour change*
Building without the {{WINDOWS_BASE_IMAGE}} build argument now needs a Windows
11 or Windows Server 2022 (or newer) host. Windows 10 and Windows Server 2019
hosts pass {{\--build-arg
WINDOWS_BASE_IMAGE=mcr.microsoft.com/windows/servercore:ltsc2019}} and keep
following that tag, as before; {{dev-support/jenkinsfile-windows-10}} now
passes it. The Jenkins nightly {{hadoop-qbt-trunk-java8-win10-x86_64}} keeps
its pipeline script in the job configuration rather than reading that file, so
someone with access to the job needs to add the same argument to its {{docker
build}} before this change is merged, or the nightly will switch to the
ltsc2022 base. Docker only warns about an unused build argument, so the job can
be updated first.
*Testing*
These runs come from a trial GitHub Actions workflow in my fork; that workflow
is not part of this change. GitHub keeps their logs for 90 days. Sizes here are
in decimal GB; the run summaries print GiB.
* [Fork run
42493777552|https://github.com/GauthamBanasandra/hadoop/actions/runs/42493777552]
tested this change; since then only comments and a Linux-cache path filter
have changed. It shows as failed because of its ltsc2019 job, explained below.
The default (ltsc2022) image built on windows-2022: 16.9 GB, 6.6 GB compressed
(15.75 and 6.13 GiB in the run summary). It then passed full {{\-Pnative-win}}
builds (120/120 modules, with {{\-DskipTests}}) and a native smoke test
({{TestHdfsNativeCodeLoader}} with {{\-Drequire.test.libhadoop}}) on
windows-2022 and windows-2025 under process isolation, and on windows-2022
under Hyper-V isolation.
* The Windows-specific unit tests ran on windows-2022 only: 12 classes in
hadoop-common, hadoop-hdfs and hadoop-yarn-common, 275 tests, no failures, 33
skipped. That run excluded six methods that fail both on windows-2022 and in
Jenkins nightly #1470 (this change does not exclude them):
{{TestFileUtil#testUnZip}}, {{#testFailFullyDeleteDirSymlinks}},
{{#testSymlinkSameFile}} and {{#testGetDU}}, {{TestNativeIO#testFstat}}, and
{{TestFsShellCopy#testPutSrcFileNoPerm}} (HADOOP-19599).
* In the same run, the ltsc2019 image built under Hyper-V on windows-2022
(10.0.17763.9378, 16.0 GB) and its entry point ran. That job shows as failed
only because the fork workflow did not clear the expected exit code 3 of its
entry point check: {{docker build}} exited 0, and the log shows "The entry
point returned 3 (expected 3)". This ltsc2019 image was not put through a full
build. On trunk, the Jenkins nightly is the only CI job that builds the
ltsc2019 path, and by 12 November none of its runs since HADOOP-20031 (#1470 to
#1481) had completed a full build. #1470 built the image and 117 of the 120
modules, but failed to compile hadoop-yarn-applications-catalog-webapp
({{\-endorseddirs}} cannot be used with {{\--release}}). #1471 and #1475 to
#1479 stopped in {{docker build}} (agent out of disk, layer errors). #1472 to
#1474, #1480 and #1481 had not started (agents offline).
* [Fork run
43083636020|https://github.com/GauthamBanasandra/hadoop/actions/runs/43083636020]
started the 10 November ltsc2022 base (10.0.20348.5870) under process
isolation (a {{cmd /c exit 3}} probe) on three windows-2022 hosts still at
10.0.20348.5773, and on a windows-2025 host. Earlier runs built Hadoop under
process isolation from the older 5761 base on 5773 and windows-2025 hosts. This
agrees with Microsoft's documentation that from Windows Server 2019 on, the
base need not match the host's revision, though only a base older than the host
has been through a full build.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
