[
https://issues.apache.org/jira/browse/HDDS-17275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17275:
------------------------------
Description:
{{dist-tar-stitching}} creates {{ozone-X.tar.gz}} with plain {{tar}} and
{{gzip}}, so the archive records metadata specific to the build: entry mtimes,
permissions following the umask, owner and group names of the builder, and the
mtime and file name in the gzip header. Entry order follows the directory
listing order of the filesystem.
Two runs of {{mvn clean package -DskipTests -DskipDocs -Pdist -Psrc}} on the
same machine, the second with {{TZ=Asia/Tokyo LC_ALL=de_DE.UTF-8}} and {{umask
002}}: all jars (including javadoc, sources and shaded jars), {{bom.json}},
{{bom.xml}} and the source tarball were identical. The binary tarball had
identical file contents and entry order, but different archive metadata.
Proposed: when GNU tar is available (as in the Linux release build
environment), create the archive with {{--format=gnu --sort=name
--mtime=<project.build.outputTimestamp> --owner=0 --group=0 --numeric-owner
--mode=go+u,go-w}}, and compress with {{gzip -n}}. Other tar implementations
keep creating a plain archive.
Other gaps found during the investigation are tracked separately: HDDS-17283
(downloaded inputs), HDDS-17284 ({{.buildinfo}} and release build environment),
HDDS-17285 (native library build path), HDDS-17286 (CI check for tarballs).
was:
h3. Problem
The {{repro}} check runs {{artifact:compare}} in the same job, on the same
runner, path, JDK, user, locale and umask as the reference build. It covers
attached Maven artifacts, but not the release tarballs ({{ozone-X.tar.gz}} is
produced by {{dist-tar-stitching}}, {{ozone-X-src.tar.gz}} has
{{attach=false}}), and it cannot detect differences caused by the build
environment. Verifying a release independently (for example with Apache Trusted
Releases, see HADOOP-19976) needs the tarballs to be identical when built
elsewhere.
h3. Observation
Two runs of {{mvn clean package -DskipTests -DskipDocs -Pdist -Psrc}} on the
same machine, the second with {{TZ=Asia/Tokyo LC_ALL=de_DE.UTF-8}} and {{umask
002}}:
* all jars (including javadoc, sources and shaded jars), {{bom.json}},
{{bom.xml}} and the source tarball were identical
* the binary tarball had identical file contents and entry order, but different
archive metadata: entry mtimes, permissions (following umask), owner and group
names of the builder, and the mtime and file name in the gzip header
h3. Gaps
# *Binary tarball metadata.* {{dist-tar-stitching}} runs plain {{tar}} and
{{gzip}}. Entry order also depends on the directory listing order of the
filesystem (sorted on APFS, not on ext4).
# *Unpinned inputs.* The {{hadoop-native-lib}} profile downloads libhadoop from
the {{master}} branch of {{ozone-thirdparty}} without a checksum. The RocksDB
source archive for {{hdds-rocks-native}} is downloaded without a checksum.
# *Optional content depending on the environment.* Docs are included only if
Hugo is available, and {{--enableGitInfo}} is used only in a git checkout, so
building from the source tarball gives different docs than building from git.
libhadoop is included only if it was downloaded. The Hugo docs are being
removed by HDDS-15923 ([PR #10829|https://github.com/apache/ozone/pull/10829]),
which resolves the docs part.
# *Build environment not pinned or recorded.* The release guide uses the
floating {{rockylinux:9}} image and unpinned JDK, cmake and gcc packages. No
{{.buildinfo}} is produced, so a verifier cannot know which toolchain to use.
# *Native library build path.* {{libozone_rocksdb_tools.so}} is built without
{{-ffile-prefix-map}}, so it may embed the build directory. Not verified.
# *rpm and deb packages* (HDDS-13439) may embed build host and build time. Not
verified; only relevant if they become release artifacts.
h3. Proposed changes
* Create the binary tarball with sorted entries, mtime from
{{project.build.outputTimestamp}}, numeric owner and group 0, normalized
permissions, and {{gzip -n}}.
* Pin downloaded inputs to a tag or commit and verify them with {{sha256}}.
* Make tarball content independent of optional downloads such as libhadoop
(include it consistently or fail).
* Generate {{.buildinfo}} ({{artifact:buildinfo}}) and document a pinned build
environment for releases.
* Extend the {{repro}} check (or add a separate one) to compare both tarballs
from an independent build with a different path, user, TZ, locale and umask.
These can be split into separate subtasks if preferred. Nondeterminism within a
single build is tracked separately in HDDS-16309 and HDDS-12862.
Summary: Make binary tarball archive metadata reproducible (was: Make
release tarballs reproducible across build environments)
> Make binary tarball archive metadata reproducible
> -------------------------------------------------
>
> Key: HDDS-17275
> URL: https://issues.apache.org/jira/browse/HDDS-17275
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Attachments: HDDS-17275.001.diff
>
>
> {{dist-tar-stitching}} creates {{ozone-X.tar.gz}} with plain {{tar}} and
> {{gzip}}, so the archive records metadata specific to the build: entry
> mtimes, permissions following the umask, owner and group names of the
> builder, and the mtime and file name in the gzip header. Entry order follows
> the directory listing order of the filesystem.
> Two runs of {{mvn clean package -DskipTests -DskipDocs -Pdist -Psrc}} on the
> same machine, the second with {{TZ=Asia/Tokyo LC_ALL=de_DE.UTF-8}} and
> {{umask 002}}: all jars (including javadoc, sources and shaded jars),
> {{bom.json}}, {{bom.xml}} and the source tarball were identical. The binary
> tarball had identical file contents and entry order, but different archive
> metadata.
> Proposed: when GNU tar is available (as in the Linux release build
> environment), create the archive with {{--format=gnu --sort=name
> --mtime=<project.build.outputTimestamp> --owner=0 --group=0 --numeric-owner
> --mode=go+u,go-w}}, and compress with {{gzip -n}}. Other tar implementations
> keep creating a plain archive.
> Other gaps found during the investigation are tracked separately: HDDS-17283
> (downloaded inputs), HDDS-17284 ({{.buildinfo}} and release build
> environment), HDDS-17285 (native library build path), HDDS-17286 (CI check
> for tarballs).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
