Reproducible MacOS Codesign'ed builds?
Hi, Has anyone looked into reproducible builds for codesign'd MacOS builds? Basically Apple codesigning is non-deterministic, which is not surprisingly I guess, so naturally makes reproducible builds a bit tricky. The general theme for this sort of issue seems to be to remove the signature before comparing (codesign --remove-signature X.dylib). Which i've attempted, and works to a degree. The single stumbling block being the signing of jpackageapplauncher in jdk.jpackage, which then gets placed in the jmod's classes resource section, leading to different module "hash" in java.base/../module-info.class, and also a different "modules" file. Has anyone else tried to tackle this problem? Could we store jpackageapplauncher somewhere that would not end up in the jmod classes...but still be securely loadable? ( https://github.com/openjdk/jdk/blob/646c8aaeeccb494c72ff84c6e0f303f790be0ba9/src/jdk.jpackage/macosx/classes/jdk/jpackage/internal/MacAppImageBuilder.java#L284 ) Thanks Andrew
Integrated: 8284539: Configure --with-source-date=version fails on MacOS
On Thu, 14 Apr 2022 16:13:59 GMT, Andrew Leonard wrote: > JDK-8282769 added support for more ISO-8601 formats, but remove handling of > just a date "-MM-DD" being present, which is the case for a configure > using --with-source-date=version which uses the date string from > version-numbers.conf. > Also, the first date parse had an invalid format string "%FZ %TZ", with too > many Zs. > This PR corrects the first date parse to parse a standard ISO-8601 Zulu > date: "%FT%TZ" > Then it adds the final check for no time being specified. > > Signed-off-by: Andrew Leonard This pull request has now been integrated. Changeset: da3d8b1d Author:Andrew Leonard URL: https://git.openjdk.java.net/jdk/commit/da3d8b1d1ea132e670d5629af3e98d958f2b56f7 Stats: 6 lines in 1 file changed: 5 ins; 0 del; 1 mod 8284539: Configure --with-source-date=version fails on MacOS Reviewed-by: erikj, ihse - PR: https://git.openjdk.java.net/jdk/pull/8247
Re: RFR: 8284539: Configure --with-source-date=version fails on MacOS
On Fri, 15 Apr 2022 12:32:50 GMT, Erik Joelsson wrote: >> JDK-8282769 added support for more ISO-8601 formats, but remove handling of >> just a date "-MM-DD" being present, which is the case for a configure >> using --with-source-date=version which uses the date string from >> version-numbers.conf. >> Also, the first date parse had an invalid format string "%FZ %TZ", with too >> many Zs. >> This PR corrects the first date parse to parse a standard ISO-8601 Zulu >> date: "%FT%TZ" >> Then it adds the final check for no time being specified. >> >> Signed-off-by: Andrew Leonard > > make/autoconf/util.m4 line 243: > >> 241: # BSD date >> 242: # ISO-8601 date in Zulu 'date'T'time'Z >> 243: timestamp=$($DATE -u -j -f "%FT%TZ" "$2" "+%s" 2> /dev/null) > > You are removing the space between FT and TZ, I'm just curious why and if > that is significant. > EDIT: Never mind me, this looks good. yes, this is supposed to be parsing an ISO8601 eg.2022-02-09T14:47:36Z, the previous JDK-8282769 had an error in this string - PR: https://git.openjdk.java.net/jdk/pull/8247
Re: RFR: 8284539: Configure --with-source-date=version fails on MacOS
On Thu, 14 Apr 2022 16:13:59 GMT, Andrew Leonard wrote: > JDK-8282769 added support for more ISO-8601 formats, but remove handling of > just a date "-MM-DD" being present, which is the case for a configure > using --with-source-date=version which uses the date string from > version-numbers.conf. > Also, the first date parse had an invalid format string "%FZ %TZ", with too > many Zs. > This PR corrects the first date parse to parse a standard ISO-8601 Zulu > date: "%FT%TZ" > Then it adds the final check for no time being specified. > > Signed-off-by: Andrew Leonard @magicus let me know when you're good for this one? and i'll then integrate - PR: https://git.openjdk.java.net/jdk/pull/8247
Re: RFR: 8284539: Configure --with-source-date=version fails on MacOS
On Thu, 14 Apr 2022 16:13:59 GMT, Andrew Leonard wrote: > JDK-8282769 added support for more ISO-8601 formats, but remove handling of > just a date "-MM-DD" being present, which is the case for a configure > using --with-source-date=version which uses the date string from > version-numbers.conf. > Also, the first date parse had an invalid format string "%FZ %TZ", with too > many Zs. > This PR corrects the first date parse to parse a standard ISO-8601 Zulu > date: "%FT%TZ" > Then it adds the final check for no time being specified. > > Signed-off-by: Andrew Leonard @magicus Hi Magnus, when you get a moment can you take a look at this one please? cheers - PR: https://git.openjdk.java.net/jdk/pull/8247
Re: RFR: 8284661: Reproducible assembly builds without relative linking [v7]
> This PR removes the need for relative object file linking introduced by > JDK-8284437 for linux libraries, by specifying > .file directives in the linux .S source files. The > source files specify a .file ASSEMBLY_SRC_FILE > where ASSEMBLY_SRC_FILE is defined by the NativeCompliation.gmk. > > Signed-off-by: Andrew Leonard Andrew Leonard has updated the pull request incrementally with one additional commit since the last revision: Trigger checks - Changes: - all: https://git.openjdk.java.net/jdk/pull/8177/files - new: https://git.openjdk.java.net/jdk/pull/8177/files/37d1c3d2..e2dc99cb Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk=8177=06 - incr: https://webrevs.openjdk.java.net/?repo=jdk=8177=05-06 Stats: 0 lines in 0 files changed: 0 ins; 0 del; 0 mod Patch: https://git.openjdk.java.net/jdk/pull/8177.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/8177/head:pull/8177 PR: https://git.openjdk.java.net/jdk/pull/8177
Re: RFR: 8284539: Configure --with-source-date=version fails on MacOS
On Thu, 14 Apr 2022 16:13:59 GMT, Andrew Leonard wrote: > JDK-8282769 added support for more ISO-8601 formats, but remove handling of > just a date "-MM-DD" being present, which is the case for a configure > using --with-source-date=version which uses the date string from > version-numbers.conf. > Also, the first date parse had an invalid format string "%FZ %TZ", with too > many Zs. > This PR corrects the first date parse to parse a standard ISO-8601 Zulu > date: "%FT%TZ" > Then it adds the final check for no time being specified. > > Signed-off-by: Andrew Leonard Verified formatting on MacOS: 1st parse: % date -u -j -f "%FT%TZ" "2022-09-22T12:56:04Z" "+%s" 1663851364 2nd parse trimming milli-seconds: % date -u -j -f "%Y-%m-%dT%H:%M:%S" "2022-09-22T12:56:04.456Z" "+%s" Warning: Ignoring 5 extraneous characters in date string (.456Z) 1663851364 3rd parse with T00:00:00Z time added: % date -u -j -f "%FT%TZ" "2022-09-22T00:00:00Z" "+%s" 1663804800 - PR: https://git.openjdk.java.net/jdk/pull/8247
RFR: 8284539: Configure --with-source-date=version fails on MacOS
JDK-8282769 added support for more ISO-8601 formats, but remove handling of just a date "-MM-DD" being present, which is the case for a configure using --with-source-date=version which uses the date string from version-numbers.conf. Also, the first date parse had an invalid format string "%FZ %TZ", with too many Zs. This PR corrects the first date parse to parse a standard ISO-8601 Zulu date: "%FT%TZ" Then it adds the final check for no time being specified. Signed-off-by: Andrew Leonard - Commit messages: - 8284539: Configure --with-source-date=version fails on MacOS Changes: https://git.openjdk.java.net/jdk/pull/8247/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk=8247=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8284539 Stats: 6 lines in 1 file changed: 5 ins; 0 del; 1 mod Patch: https://git.openjdk.java.net/jdk/pull/8247.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/8247/head:pull/8247 PR: https://git.openjdk.java.net/jdk/pull/8247
Re: RFR: 8284661: Reproducible assembly builds without relative linking [v5]
On Thu, 14 Apr 2022 11:13:20 GMT, Andrew Leonard wrote: >> make/data/autoheaders/assemblyprefix.h line 25: >> >>> 23: >>> 24: // ASSEMBLY_SRC_FILE gets replaced by relative or absolute file path >>> 25: // in NativeCompilation.gmk, this ensures reproducible .debuginfo >> >> Actually, this ensures reproducible object file (`.o`) rather than the >> contents of `.debuginfo`. As far as I remember, `gcc` uses the "value" of >> `.file` in `.debuginfo`, but `clang` does not. Which is why the path mapping >> option is still necessary even with `.file` present and with a value stable >> across builds. >> >> Maybe phrase this something like >> >> ... this ensures a reproducible object file through a predictable value of >> the STT_FILE symbol. >> >> ? > > @mkartashev thanks. So at the moment we are only using this assembly header > for gcc tooling, see NativeCompliation.gmk change. > So maybe I change this to specifically state gcc and it's effect on linux > debuginfo ? updated - PR: https://git.openjdk.java.net/jdk/pull/8177
Re: RFR: 8284661: Reproducible assembly builds without relative linking [v5]
On Wed, 13 Apr 2022 13:55:59 GMT, Andrew Leonard wrote: >> This PR removes the need for relative object file linking introduced by >> JDK-8284437 for linux libraries, by specifying >> .file directives in the linux .S source files. The >> source files specify a .file ASSEMBLY_SRC_FILE >> where ASSEMBLY_SRC_FILE is defined by the NativeCompliation.gmk. >> >> Signed-off-by: Andrew Leonard > > Andrew Leonard has updated the pull request incrementally with two additional > commits since the last revision: > > - 8284661: Reproducible assembly builds without relative linking > >Signed-off-by: Andrew Leonard > - 8284661: Reproducible assembly builds without relative linking > >Signed-off-by: Andrew Leonard I have successfully tested this PR on all platforms running test/jdk/build/AbsPathsInImage. - PR: https://git.openjdk.java.net/jdk/pull/8177
Re: RFR: 8284661: Reproducible assembly builds without relative linking [v6]
> This PR removes the need for relative object file linking introduced by > JDK-8284437 for linux libraries, by specifying > .file directives in the linux .S source files. The > source files specify a .file ASSEMBLY_SRC_FILE > where ASSEMBLY_SRC_FILE is defined by the NativeCompliation.gmk. > > Signed-off-by: Andrew Leonard Andrew Leonard has updated the pull request incrementally with one additional commit since the last revision: 8284661: Reproducible assembly builds without relative linking Signed-off-by: Andrew Leonard - Changes: - all: https://git.openjdk.java.net/jdk/pull/8177/files - new: https://git.openjdk.java.net/jdk/pull/8177/files/4245e704..37d1c3d2 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk=8177=05 - incr: https://webrevs.openjdk.java.net/?repo=jdk=8177=04-05 Stats: 3 lines in 1 file changed: 2 ins; 0 del; 1 mod Patch: https://git.openjdk.java.net/jdk/pull/8177.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/8177/head:pull/8177 PR: https://git.openjdk.java.net/jdk/pull/8177
Re: RFR: 8284661: Reproducible assembly builds without relative linking [v5]
On Thu, 14 Apr 2022 10:13:32 GMT, Maxim Kartashev wrote: >> Andrew Leonard has updated the pull request incrementally with two >> additional commits since the last revision: >> >> - 8284661: Reproducible assembly builds without relative linking >> >>Signed-off-by: Andrew Leonard >> - 8284661: Reproducible assembly builds without relative linking >> >>Signed-off-by: Andrew Leonard > > make/data/autoheaders/assemblyprefix.h line 25: > >> 23: >> 24: // ASSEMBLY_SRC_FILE gets replaced by relative or absolute file path >> 25: // in NativeCompilation.gmk, this ensures reproducible .debuginfo > > Actually, this ensures reproducible object file (`.o`) rather than the > contents of `.debuginfo`. As far as I remember, `gcc` uses the "value" of > `.file` in `.debuginfo`, but `clang` does not. Which is why the path mapping > option is still necessary even with `.file` present and with a value stable > across builds. > > Maybe phrase this something like > > ... this ensures a reproducible object file through a predictable value of > the STT_FILE symbol. > > ? @mkartashev thanks. So at the moment we are only using this assembly header for gcc tooling, see NativeCompliation.gmk change. So maybe I change this to specifically state gcc and it's effect on linux debuginfo ? - PR: https://git.openjdk.java.net/jdk/pull/8177
Re: RFR: 8284661: Reproducible assembly builds without relative linking [v4]
On Wed, 13 Apr 2022 11:37:36 GMT, Magnus Ihse Bursie wrote: > The latest version looks much better! Fix the test to check for linux, and > the indentation in the new header file, and I'd say you're good to go! @magicus update ready, with test fix, indentation and remove .file form the linux incubator files. - PR: https://git.openjdk.java.net/jdk/pull/8177
Re: RFR: 8284661: Reproducible assembly builds without relative linking [v5]
> This PR removes the need for relative object file linking introduced by > JDK-8284437 for linux libraries, by specifying > .file directives in the linux .S source files. The > source files specify a .file ASSEMBLY_SRC_FILE > where ASSEMBLY_SRC_FILE is defined by the NativeCompliation.gmk. > > Signed-off-by: Andrew Leonard Andrew Leonard has updated the pull request incrementally with two additional commits since the last revision: - 8284661: Reproducible assembly builds without relative linking Signed-off-by: Andrew Leonard - 8284661: Reproducible assembly builds without relative linking Signed-off-by: Andrew Leonard - Changes: - all: https://git.openjdk.java.net/jdk/pull/8177/files - new: https://git.openjdk.java.net/jdk/pull/8177/files/f9a53868..4245e704 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk=8177=04 - incr: https://webrevs.openjdk.java.net/?repo=jdk=8177=03-04 Stats: 79 lines in 39 files changed: 1 ins; 36 del; 42 mod Patch: https://git.openjdk.java.net/jdk/pull/8177.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/8177/head:pull/8177 PR: https://git.openjdk.java.net/jdk/pull/8177
Re: RFR: 8284661: Reproducible assembly builds without relative linking [v4]
On Wed, 13 Apr 2022 11:36:41 GMT, Magnus Ihse Bursie wrote: >> Andrew Leonard has updated the pull request incrementally with one >> additional commit since the last revision: >> >> 8284661: Reproducible assembly builds without relative linking >> >> Signed-off-by: Andrew Leonard > > make/data/autoheaders/assemblyprefix.h line 24: > >> 22: # >> 23: >> 24: // ASSEMBLY_SRC_FILE gets replaced by relative or absolute file >> path > > While not critical, the indentation here looks surprising and odd. I was keeping it inline with what's in the other .S files - PR: https://git.openjdk.java.net/jdk/pull/8177
Re: RFR: 8284661: Reproducible assembly builds without relative linking [v4]
On Wed, 13 Apr 2022 11:41:14 GMT, Andrew Leonard wrote: >> make/data/autoheaders/assemblyprefix.h line 24: >> >>> 22: # >>> 23: >>> 24: // ASSEMBLY_SRC_FILE gets replaced by relative or absolute file >>> path >> >> While not critical, the indentation here looks surprising and odd. > > I was keeping it inline with what's in the other .S files i'll change - PR: https://git.openjdk.java.net/jdk/pull/8177
Re: RFR: 8284661: Reproducible assembly builds without relative linking [v4]
> This PR removes the need for relative object file linking introduced by > JDK-8284437 for linux libraries, by specifying > .file directives in the linux .S source files. The > source files specify a .file ASSEMBLY_SRC_FILE > where ASSEMBLY_SRC_FILE is defined by the NativeCompliation.gmk. > > Signed-off-by: Andrew Leonard Andrew Leonard has updated the pull request incrementally with one additional commit since the last revision: 8284661: Reproducible assembly builds without relative linking Signed-off-by: Andrew Leonard - Changes: - all: https://git.openjdk.java.net/jdk/pull/8177/files - new: https://git.openjdk.java.net/jdk/pull/8177/files/748485d7..f9a53868 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk=8177=03 - incr: https://webrevs.openjdk.java.net/?repo=jdk=8177=02-03 Stats: 71 lines in 8 files changed: 41 ins; 27 del; 3 mod Patch: https://git.openjdk.java.net/jdk/pull/8177.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/8177/head:pull/8177 PR: https://git.openjdk.java.net/jdk/pull/8177
Re: RFR: 8284661: Reproducible assembly builds without relative linking [v3]
On Tue, 12 Apr 2022 10:27:30 GMT, Magnus Ihse Bursie wrote: > Actually, a better option might be to let GCC append the `.file` > automatically to all assembly files. I think this can be done by creating a > file like `make/data/autoheaders/assemblyprefix.h` with: > > ``` > // ASSEMBLY_SRC_FILE gets replaced by relative or absolute file path > // in NativeCompilation.gmk, this ensures reproducible .debuginfo > .file ASSEMBLY_SRC_FILE > ``` > > and then add `-include $TOPDIR/make/data/autoheaders/assemblyprefix.h` to the > gcc assembly command line. > > I have not verified this so it needs to be checked. > > I also think we should remove the `.file` keyword from the incubator files. > (Nothing good can come out of having two `.file` attributes.) @magicus this sounds good, i'll give it a try - PR: https://git.openjdk.java.net/jdk/pull/8177
Re: RFR: 8284661: Reproducible assembly builds without relative linking
On Tue, 12 Apr 2022 08:39:54 GMT, Andrew Haley wrote: > > I excluded all kinds of debuginfo files because I didn't know if they could > > be made free of absolute paths, and it was out of scope for what I was > > doing at the time. > > GCC, I believe, uses whatever pathname you give to the compiler in the > debuginfo. If you give GCC relative pathnames, that's what it emits. @theRealAph The current strategy is to retain full paths in cmdlines, so that they can be copy & pasted from logs and just invoked, if relative paths are used then you need to be in the right place. Hence instead to achieve relative paths in the debuginfo, we need to use --debug-prefix-map option. - PR: https://git.openjdk.java.net/jdk/pull/8177
Re: RFR: 8284661: Reproducible assembly builds without relative linking [v3]
On Mon, 11 Apr 2022 15:41:20 GMT, Andrew Leonard wrote: >> This PR removes the need for relative object file linking introduced by >> JDK-8284437 for linux libraries, by specifying >> .file directives in the linux .S source files. The >> source files specify a .file ASSEMBLY_SRC_FILE >> where ASSEMBLY_SRC_FILE is defined by the NativeCompliation.gmk. >> >> Signed-off-by: Andrew Leonard > > Andrew Leonard has updated the pull request incrementally with one additional > commit since the last revision: > > 8284661: Reproducible assembly builds without relative linking > > Signed-off-by: Andrew Leonard > Looks like there is way too few .S files in this patch. > > ``` > $ find . -name "*.S" > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_s_atan2_linux_x86.S > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_s_atan_linux_x86.S > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_s_tanh_linux_x86.S > ./jdk.incubator.vector/windows/native/libjsvml/jsvml_d_log10_windows_x86.S > ./jdk.incubator.vector/windows/native/libjsvml/jsvml_d_exp_windows_x86.S > ./jdk.incubator.vector/windows/native/libjsvml/jsvml_d_atan2_windows_x86.S > ./jdk.incubator.vector/windows/native/libjsvml/jsvml_s_expm1_windows_x86.S > ./hotspot/os_cpu/linux_arm/linux_arm_32.S > ./hotspot/os_cpu/linux_x86/linux_x86_32.S > ./hotspot/os_cpu/linux_x86/linux_x86_64.S > ./hotspot/os_cpu/linux_aarch64/atomic_linux_aarch64.S > ./hotspot/os_cpu/linux_aarch64/copy_linux_aarch64.S > ./hotspot/os_cpu/linux_aarch64/threadLS_linux_aarch64.S > ./hotspot/os_cpu/bsd_x86/bsd_x86_32.S > ./hotspot/os_cpu/bsd_x86/bsd_x86_64.S > ./hotspot/os_cpu/bsd_aarch64/copy_bsd_aarch64.S > ``` > > (You can leave out the windows files) @magicus all those incubator ones already have there own ".file " in them, so is already relative as it were.. I think these are pre-canned Intel/RedHat assembler maths functions taken as-is, so I thought best to leave the existing .file as-is. I did all the linux ones, and left Windows and Mac bsd ones. - PR: https://git.openjdk.java.net/jdk/pull/8177
Re: RFR: 8284661: Reproducible assembly builds without relative linking [v3]
On Mon, 11 Apr 2022 21:03:22 GMT, Magnus Ihse Bursie wrote: >> Andrew Leonard has updated the pull request incrementally with one >> additional commit since the last revision: >> >> 8284661: Reproducible assembly builds without relative linking >> >> Signed-off-by: Andrew Leonard > > Looks like there is way too few .S files in this patch. > > > $ find . -name "*.S" > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_s_atan2_linux_x86.S > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_s_atan_linux_x86.S > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_s_tanh_linux_x86.S > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_s_tan_linux_x86.S > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_d_pow_linux_x86.S > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_s_cbrt_linux_x86.S > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_d_log_linux_x86.S > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_d_log10_linux_x86.S > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_s_cosh_linux_x86.S > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_s_log1p_linux_x86.S > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_d_exp_linux_x86.S > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_d_hypot_linux_x86.S > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_s_expm1_linux_x86.S > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_d_sinh_linux_x86.S > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_d_cos_linux_x86.S > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_d_acos_linux_x86.S > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_d_asin_linux_x86.S > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_d_sin_linux_x86.S > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_s_exp_linux_x86.S > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_d_expm1_linux_x86.S > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_s_cos_linux_x86.S > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_s_asin_linux_x86.S > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_s_hypot_linux_x86.S > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_s_sinh_linux_x86.S > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_s_sin_linux_x86.S > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_s_acos_linux_x86.S > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_d_log1p_linux_x86.S > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_s_log10_linux_x86.S > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_d_tanh_linux_x86.S > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_d_atan_linux_x86.S > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_d_tan_linux_x86.S > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_s_log_linux_x86.S > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_s_pow_linux_x86.S > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_d_cosh_linux_x86.S > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_d_cbrt_linux_x86.S > ./jdk.incubator.vector/linux/native/libjsvml/jsvml_d_atan2_linux_x86.S > ./jdk.incubator.vector/windows/native/libjsvml/jsvml_s_tanh_windows_x86.S > ./jdk.incubator.vector/windows/native/libjsvml/jsvml_d_tanh_windows_x86.S > ./jdk.incubator.vector/windows/native/libjsvml/jsvml_s_log10_windows_x86.S > ./jdk.incubator.vector/windows/native/libjsvml/jsvml_s_exp_windows_x86.S > ./jdk.incubator.vector/windows/native/libjsvml/jsvml_s_log_windows_x86.S > ./jdk.incubator.vector/windows/native/libjsvml/jsvml_d_log10_windows_x86.S > ./jdk.incubator.vector/windows/native/libjsvml/jsvml_d_cos_windows_x86.S > ./jdk.incubator.vector/windows/native/libjsvml/jsvml_s_log1p_windows_x86.S > ./jdk.incubator.vector/windows/native/libjsvml/jsvml_d_sin_windows_x86.S > ./jdk.incubator.vector/windows/native/libjsvml/jsvml_d_tan_windows_x86.S > ./jdk.incubator.vector/windows/native/libjsvml/jsvml_d_pow_windows_x86.S > ./jdk.incubator.vector/windows/native/libjsvml/jsvml_d_cosh_windows_x86.S > ./jdk.incubator.vector/windows/native/libjsvml/jsvml_s_cosh_windows_x86.S > ./jdk.incubator.vector/windows/native/libjsvml/jsvml_d_log1p_windows_x86.S > ./jdk.incubator.vector/windows/native/libjsvml/jsvml_d_atan_windows_x86.S > ./jdk.incubator.vector/windows/native/libjsvml/jsvml_s_atan_windows_x86.S > ./jdk.incubator.vector/windows/native/libjsvml/jsvml_s_tan_windows_x86.S > ./jdk.incubator.vector/windows/native/libjsvml/jsvml_s_pow_windows_x86.S > ./jdk.incubator.vector/windows/native/libjsvml/jsvml_d_hypot_windows_x86.S > ./jdk.incubator.vector/windows/native/libjsvml/jsvml_s_sin_windows_x86.S > ./jdk.incubator.vector/windows/native/libjsvml/jsvml_s_hypot_windows_x86.S > ./jdk.incubator.vector/windows/native/libjsvml/jsvml_d_asin_windows_x86.S > ./jdk.incubator.vector/windows/native/libjsvml/jsvml_s_asin_windows_x86.S > ./jdk.incubator.vector/windows/na
Re: RFR: 8284661: Reproducible assembly builds without relative linking [v3]
On Mon, 11 Apr 2022 17:40:58 GMT, Erik Joelsson wrote:
>> @erikj79 As far as I can see only AIX is the exception of not being Linux
>> and using .debuginfo suffix, are there others?
>> I am currently testing AIX, so I will see if it passes, as you say, I would
>> have my doubts it will!
>
> Solaris used that extension as well. It's no longer supported in mainline,
> but if you backport this, which I assume you will want to, then someone may
> hit it.
ah right yes Solaris
I'll add a suitable System.getProperty("os.name").contains("Linux")
-
PR: https://git.openjdk.java.net/jdk/pull/8177
Re: RFR: 8284661: Reproducible assembly builds without relative linking
On Mon, 11 Apr 2022 16:58:38 GMT, Erik Joelsson wrote: > > thanks @erikj79 Not familiar with AbsPathsInImage, could you give some > > background please? wondering why it's not found these paths before, maybe > > it's not running on debugimages? What tier does it get run in? > > I wrote this test when introducing the --disable-absolute-paths-in-output > functionality, so we could verify that no absolute paths made it into any > part of the output that we eventually ship to customers. I excluded all kinds > of debuginfo files because I didn't know if they could be made free of > absolute paths, and it was out of scope for what I was doing at the time. > > I believe we run this test as part of our tier2 internally at Oracle. I think > generally it's an optional test as it relies on a specific build > configuration that all builders of OpenJDK may not want to use. Thanks for the background, we also run it as an "extended" test at Adoptium i've discovered. - PR: https://git.openjdk.java.net/jdk/pull/8177
Re: RFR: 8284661: Reproducible assembly builds without relative linking [v3]
On Mon, 11 Apr 2022 16:53:41 GMT, Erik Joelsson wrote:
>> Andrew Leonard has updated the pull request incrementally with one
>> additional commit since the last revision:
>>
>> 8284661: Reproducible assembly builds without relative linking
>>
>> Signed-off-by: Andrew Leonard
>
> test/jdk/build/AbsPathsInImage.java line 167:
>
>> 165: if (Files.isSymbolicLink(file)) {
>> 166: return super.visitFile(file, attrs);
>> 167: } else if (fileName.endsWith(".pdb")) {
>
> The .debuginfo suffix has at least historically been used on more OSes than
> just Linux. I think we should only include .debuginfo files in this test if
> the OS is also Linux.
@erikj79 As far as I can see only AIX is the exception of not being Linux and
using .debuginfo suffix, are there others?
I am currently testing AIX, so I will see if it passes, as you say, I would
have my doubts it will!
-
PR: https://git.openjdk.java.net/jdk/pull/8177
Re: RFR: 8284661: Reproducible assembly builds without relative linking [v3]
On Mon, 11 Apr 2022 15:41:20 GMT, Andrew Leonard wrote: >> This PR removes the need for relative object file linking introduced by >> JDK-8284437 for linux libraries, by specifying >> .file directives in the linux .S source files. The >> source files specify a .file ASSEMBLY_SRC_FILE >> where ASSEMBLY_SRC_FILE is defined by the NativeCompliation.gmk. >> >> Signed-off-by: Andrew Leonard > > Andrew Leonard has updated the pull request incrementally with one additional > commit since the last revision: > > 8284661: Reproducible assembly builds without relative linking > > Signed-off-by: Andrew Leonard @magicus @mkartashev For your reviews please. I'm also currently running a full platform build at Adoptium, so as to ensure validation on all Linux flavors (https://ci.adoptopenjdk.net/job/build-scripts/job/openjdk19-pipeline/88/parameters/), and plan to run AbsPathsInImage against them all. - PR: https://git.openjdk.java.net/jdk/pull/8177
Re: RFR: 8284661: Reproducible assembly builds without relative linking [v3]
> This PR removes the need for relative object file linking introduced by > JDK-8284437 for linux libraries, by specifying > .file directives in the linux .S source files. The > source files specify a .file ASSEMBLY_SRC_FILE > where ASSEMBLY_SRC_FILE is defined by the NativeCompliation.gmk. > > Signed-off-by: Andrew Leonard Andrew Leonard has updated the pull request incrementally with one additional commit since the last revision: 8284661: Reproducible assembly builds without relative linking Signed-off-by: Andrew Leonard - Changes: - all: https://git.openjdk.java.net/jdk/pull/8177/files - new: https://git.openjdk.java.net/jdk/pull/8177/files/d32473ba..748485d7 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk=8177=02 - incr: https://webrevs.openjdk.java.net/?repo=jdk=8177=01-02 Stats: 6 lines in 6 files changed: 0 ins; 0 del; 6 mod Patch: https://git.openjdk.java.net/jdk/pull/8177.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/8177/head:pull/8177 PR: https://git.openjdk.java.net/jdk/pull/8177
Re: RFR: 8284661: Reproducible assembly builds without relative linking [v2]
> This PR removes the need for relative object file linking introduced by > JDK-8284437 for linux libraries, by specifying > .file directives in the linux .S source files. The > source files specify a .file ASSEMBLY_SRC_FILE > where ASSEMBLY_SRC_FILE is defined by the NativeCompliation.gmk. > > Signed-off-by: Andrew Leonard Andrew Leonard has updated the pull request incrementally with one additional commit since the last revision: 8284661: Reproducible assembly builds without relative linking Signed-off-by: Andrew Leonard - Changes: - all: https://git.openjdk.java.net/jdk/pull/8177/files - new: https://git.openjdk.java.net/jdk/pull/8177/files/c73579e2..d32473ba Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk=8177=01 - incr: https://webrevs.openjdk.java.net/?repo=jdk=8177=00-01 Stats: 14 lines in 7 files changed: 12 ins; 0 del; 2 mod Patch: https://git.openjdk.java.net/jdk/pull/8177.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/8177/head:pull/8177 PR: https://git.openjdk.java.net/jdk/pull/8177
Re: RFR: 8284661: Reproducible assembly builds without relative linking
On Mon, 11 Apr 2022 14:50:37 GMT, Andrew Leonard wrote: >> Non-determinism in .debuginfo straight away makes .so libraries >> non-deterministic due to the embedded debuginfo CRC value. >> @erikj79 i'll try removing .debuginfo from the exceptions and try it... > >> @andrew-m-leonard This change leaves the pathmap options intact, right? >> Because `clang` leaves absolute path names in the debug info regardless of >> what's in `.file`. > > @mkartashev yes, that was in the already merged PR : > https://github.com/openjdk/jdk/commit/4451257b1432e4180a16757aafca6141b8063772 > > So clang / MacOS needs deeper investigation, the AbsPathInImage test > currently also excludes .dSYM folder. > Hi @andrew-m-leonard, I am currently working on moving SafeFetch to static > assembly: > > #7865 > > Would your change affect that, especially the ability to export > inner-function labels as jump-back-points? hi @tstuefe Thomas, this is only requiring .file directives in linux .S files, which will mean your PR will require the line: .file ASSEMBLY_SRC_FILE assuming they don't already have a .file... but I don't think it should affect anything else you're doing. This is for reproducible builds, and every platform has it's own issues with respect to this. In this particular Linux case, the linker embeds it's own .file symbol if one isn't present using the .o object path name. Other platforms will have there own quirks we need to fix > > Also, riscv, ppcle and s390 are missing, why not do it on all platforms? Which .S files am I missing for these flavors of Linux? - PR: https://git.openjdk.java.net/jdk/pull/8177
Re: RFR: 8284661: Reproducible assembly builds without relative linking
On Mon, 11 Apr 2022 14:05:54 GMT, Andrew Leonard wrote: >> This PR removes the need for relative object file linking introduced by >> JDK-8284437 for linux libraries, by specifying >> .file directives in the linux .S source files. The >> source files specify a .file ASSEMBLY_SRC_FILE >> where ASSEMBLY_SRC_FILE is defined by the NativeCompliation.gmk. >> >> Signed-off-by: Andrew Leonard > > Non-determinism in .debuginfo straight away makes .so libraries > non-deterministic due to the embedded debuginfo CRC value. > @erikj79 i'll try removing .debuginfo from the exceptions and try it... > @andrew-m-leonard This change leaves the pathmap options intact, right? > Because `clang` leaves absolute path names in the debug info regardless of > what's in `.file`. @mkartashev yes, that was in the already merged PR : https://github.com/openjdk/jdk/commit/4451257b1432e4180a16757aafca6141b8063772 So clang / MacOS needs deeper investigation, the AbsPathInImage test currently also excludes .dSYM folder. - PR: https://git.openjdk.java.net/jdk/pull/8177
Re: RFR: 8284661: Reproducible assembly builds without relative linking
On Mon, 11 Apr 2022 12:55:16 GMT, Erik Joelsson wrote: >> This PR removes the need for relative object file linking introduced by >> JDK-8284437 for linux libraries, by specifying >> .file directives in the linux .S source files. The >> source files specify a .file ASSEMBLY_SRC_FILE >> where ASSEMBLY_SRC_FILE is defined by the NativeCompliation.gmk. >> >> Signed-off-by: Andrew Leonard > > Maybe some short explanation should be added as comment in the src files? > > If this makes the debuginfo files free from absolute paths on Linux, could > you also adjust `open/test/jdk/build/AbsPathsInImage.java` to include those > files on Linux? Would be good to be able to catch any regression to this > behavior in the future. thanks @erikj79 Not familiar with AbsPathsInImage, could you give some background please? wondering why it's not found these paths before, maybe it's not running on debugimages? What tier does it get run in? @erikj79 this would be why: https://github.com/openjdk/jdk/blob/d442328bc2f2f4bc35dd054487a78552e3d9a759/test/jdk/build/AbsPathsInImage.java#L167 - PR: https://git.openjdk.java.net/jdk/pull/8177
Re: RFR: 8284661: Reproducible assembly builds without relative linking
On Mon, 11 Apr 2022 09:43:37 GMT, Andrew Leonard wrote: > This PR removes the need for relative object file linking introduced by > JDK-8284437 for linux libraries, by specifying > .file directives in the linux .S source files. The > source files specify a .file ASSEMBLY_SRC_FILE > where ASSEMBLY_SRC_FILE is defined by the NativeCompliation.gmk. > > Signed-off-by: Andrew Leonard Non-determinism in .debuginfo straight away makes .so libraries non-deterministic due to the embedded debuginfo CRC value. @erikj79 i'll try removing .debuginfo from the exceptions and try it... - PR: https://git.openjdk.java.net/jdk/pull/8177
Re: RFR: 8284661: Reproducible assembly builds without relative linking
On Mon, 11 Apr 2022 09:43:37 GMT, Andrew Leonard wrote: > This PR removes the need for relative object file linking introduced by > JDK-8284437 for linux libraries, by specifying > .file directives in the linux .S source files. The > source files specify a .file ASSEMBLY_SRC_FILE > where ASSEMBLY_SRC_FILE is defined by the NativeCompliation.gmk. > > Signed-off-by: Andrew Leonard dwarf debuginfo output for the assembly entries now looks good: < 0><0x000b> DW_TAG_compile_unit DW_AT_stmt_list 0x0022146d DW_AT_low_pc0x004707c0 DW_AT_high_pc 0x00470a40 DW_AT_name src/hotspot/os_cpu/linux_aarch64/atomic_linux_aarch64.S DW_AT_comp_dir make/hotspot DW_AT_producer GNU AS 2.27 DW_AT_language DW_LANG_Mips_Assembler - PR: https://git.openjdk.java.net/jdk/pull/8177
RFR: 8284661: Reproducible assembly builds without relative linking
This PR removes the need for relative object file linking introduced by JDK-8284437 for linux libraries, by specifying .file directives in the linux .S source files. The source files specify a .file ASSEMBLY_SRC_FILE where ASSEMBLY_SRC_FILE is defined by the NativeCompliation.gmk. Signed-off-by: Andrew Leonard - Commit messages: - 8284661: Reproducible assembly builds without relative linking Changes: https://git.openjdk.java.net/jdk/pull/8177/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk=8177=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8284661 Stats: 32 lines in 7 files changed: 15 ins; 13 del; 4 mod Patch: https://git.openjdk.java.net/jdk/pull/8177.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/8177/head:pull/8177 PR: https://git.openjdk.java.net/jdk/pull/8177
Re: RFR: 8284437: Building from different users/workspace is not always deterministic [v2]
On Fri, 8 Apr 2022 14:42:34 GMT, Andrew Leonard wrote: >> Andrew Leonard has updated the pull request incrementally with one >> additional commit since the last revision: >> >> Trigger checks > > I am not sure why without the explicit .file directive that the FILE symbol > in the ELF info contains an entry pointing to the .o object file, here's what > it was before: > > 31712: 0 FILELOCAL DEFAULT ABS > /home/andrew2/jdk/build/linux-aarch64-server-release/hotspot/variant-server/libjvm/objs/atomic_linux_aarch64.o > > It's as if the lack of the .file has caused some tooling (linker?) to create > this entry using the .o file path. > @andrew-m-leonard > > > So I am thinking adding a .file directive to the .S file specifying the > > "full source path" should do the trick thanks to -debug-prefix-map, I will > > do a test.. > > But then you'll end up with that absolute path name embedded into `STT_FILE` > in the object file. It's best to use the relative path name _and_ employ path > mapping. @mkartashev of course yes it would make the .o non-reproducible, so yes makes sense "relative path name _and_ employ path mapping." - PR: https://git.openjdk.java.net/jdk/pull/8124
Re: RFR: 8284437: Building from different users/workspace is not always deterministic [v2]
On Wed, 6 Apr 2022 13:30:28 GMT, Andrew Leonard wrote: >> This PR fixes the non-deterministic behavior when building on linux with >> different userids or within >> different workspace folders. >> It fixes the following issues: >> - MakeZipReproducible.java used to produce reproducible src.zip removes the >> optional zip "extra" field containing UID/GID. >> - When absolute output paths are not allowed, enable the use of >> -fdebug-prefix-map to ensure debug symbol info does not contain the top >> level workspace folder. >> - For reproducible builds ensure the gcc random symbol name generator is >> seeded using -frandom-seed. >> - For reproducible builds when producing debug symbols use relative object >> paths for library linking to remove absolute MASM object paths. >> >> Signed-off-by: Andrew Leonard > > Andrew Leonard has updated the pull request incrementally with one additional > commit since the last revision: > > Trigger checks I am not sure why without the explicit .file directive that the FILE symbol in the ELF info contains an entry pointing to the .o object file, here's what it was before: 31712: 0 FILELOCAL DEFAULT ABS /home/andrew2/jdk/build/linux-aarch64-server-release/hotspot/variant-server/libjvm/objs/atomic_linux_aarch64.o It's as if the lack of the .file has caused some tooling (linker?) to create this entry using the .o file path. - PR: https://git.openjdk.java.net/jdk/pull/8124
Re: RFR: 8284437: Building from different users/workspace is not always deterministic [v2]
On Fri, 8 Apr 2022 14:18:50 GMT, Maxim Kartashev wrote: >> @magicus >>> Is it possible to pass the ".file" directive on the command line? >> >> I don't think so. Your other idea works quite well, though. If you have this >> in the source >> >>.file THIS_FILE >> >> and you specify this on the command line >> >> -DTHIS_FILE='"src/hotspot/os_cpu/linux_x86/linux_x86_64.S"' >> >> you get the same reproducible and debuggable result. > >> @mkartashev Why don't you try building with clang as two different users, in >> two different directories, and see if the build is non-reproducible without >> this patch, but is reproducible with it (incl. your suggested change to >> include clang as well)? > > I'm going to do this on Mac, but perhaps not in the immediate future. If > someone has the time for this right now, please jump ahead of me in the line. @mkartashev yes you're right, here's the dwarf for the compilation unit that I changed to add the .file to: COMPILE_UNIT: < 0><0x000b> DW_TAG_compile_unit DW_AT_stmt_list 0x0022146d DW_AT_low_pc0x004707c0 DW_AT_high_pc 0x00470a40 DW_AT_name atomic_linux_aarch64.S DW_AT_comp_dir make/hotspot DW_AT_producer GNU AS 2.27 DW_AT_language DW_LANG_Mips_Assembler So it needs the full source path specifying as you said. Note however -debug-prefix-map has affected the DW_AT_comp_dir, removing the absolute workspace folder. Note, previously this is what the same entry had: COMPILE_UNIT: < 0><0x000b> DW_TAG_compile_unit DW_AT_stmt_list 0x00224f82 DW_AT_low_pc0x004707c0 DW_AT_high_pc 0x00470a40 DW_AT_name /home/andrew2/jdk/src/hotspot/os_cpu/linux_aarch64/atomic_linux_aarch64.S DW_AT_comp_dir /home/andrew2/jdk/make/hotspot DW_AT_producer GNU AS 2.27 DW_AT_language DW_LANG_Mips_Assembler So I am thinking adding a .file directive to the .S file specifying the "full source path" should do the trick thanks to -debug-prefix-map, I will do a test.. - PR: https://git.openjdk.java.net/jdk/pull/8124
Re: RFR: 8284437: Building from different users/workspace is not always deterministic [v2]
On Fri, 8 Apr 2022 13:11:32 GMT, Maxim Kartashev wrote: >>> While we're at it, let me repeat my question from the alias: >>> >>> I was wondering why were the changes in `make/autoconf/flags-cflags.m4` >>> made under: >>> >>> ``` >>> if test "x$TOOLCHAIN_TYPE" = xgcc; then ... >>> ``` >>> >>> but not also propagated to the clang branch down below >>> >>> ``` >>> elif test "x$TOOLCHAIN_TYPE" = xclang; then ... >>> ``` >>> >>> ? >>> >>> It seems like the same would be required to make builds by `clang` >>> reproducible and `clang` does support the same option, spelling and all. >> >> That is simply because I have not test full reproducibility on MacOS, the >> intention of this PR was for Linux platforms. >> Although MacOS is my next port of investigation for reproducibility as a >> whole.. > >> That is simply because I have not test full reproducibility on MacOS, the >> intention of this PR was for Linux platforms. >> Although MacOS is my next port of investigation for reproducibility as a >> whole.. > > @andrew-m-leonard Thank you for the clarification! > > Again, FWIW doing same on MacOS achieved the same reproducible result as on > Linux. The only thing I won't vouch for is how necessary it is on MacOS; > maybe some options can be omitted for `clang`, for example. I simply did the > exact same thing without much investigation as to why it worked. @mkartashev Maxim, just tried your solution on one of the assembly files, and it does resolve the absolute path for it. I did a readelf, and I can see the entry that was absolute before is now what I specified in the .S assembly: 31712: 0 FILELOCAL DEFAULT ABS atomic_linux_aarch64.S 31714: 0 FILELOCAL DEFAULT ABS /home/andrew3/jdk/build/l 31725: 0 FILELOCAL DEFAULT ABS /home/andrew3/jdk/build/l I can't seem to find any online assembly documentation(?) that states that a .file symbol pointing to the absolute object file name is produced, but it obviously does! In fact maybe it's to do with how the linker generates them, as using relative path for the link changed it...? @magicus I will create a new PR, with .file <...S> directives added to the linux platform assembly files, and undo the relative path linking. Does that seem reasonable? - PR: https://git.openjdk.java.net/jdk/pull/8124
Re: RFR: 8284437: Building from different users/workspace is not always deterministic [v2]
On Fri, 8 Apr 2022 12:43:46 GMT, Maxim Kartashev wrote: > While we're at it, let me repeat my question from the alias: > > I was wondering why were the changes in `make/autoconf/flags-cflags.m4` made > under: > > ``` > if test "x$TOOLCHAIN_TYPE" = xgcc; then ... > ``` > > but not also propagated to the clang branch down below > > ``` > elif test "x$TOOLCHAIN_TYPE" = xclang; then ... > ``` > > ? > > It seems like the same would be required to make builds by `clang` > reproducible and `clang` does support the same option, spelling and all. That is simply because I have not test full reproducibility on MacOS, the intention of this PR was for Linux platforms. Although MacOS is my next port of investigation for reproducibility as a whole.. - PR: https://git.openjdk.java.net/jdk/pull/8124
Re: RFR: 8284437: Building from different users/workspace is not always deterministic [v2]
On Fri, 8 Apr 2022 12:24:38 GMT, Maxim Kartashev wrote: > FWIW, I (locally) solved the problem of absolute path names in the compiled > assembly by adding the `.file` directive. For example: > > ``` > --- a/src/hotspot/os_cpu/linux_aarch64/atomic_linux_aarch64.S > +++ b/src/hotspot/os_cpu/linux_aarch64/atomic_linux_aarch64.S > @@ -19,7 +19,7 @@ > // or visit www.oracle.com if you need additional information or have any > // questions. > > - > +.file "atomic_linux_aarch64.S" > > .text > ``` @mkartashev Maxim, I think that might be worth a LOT :-) thank you, i'll have a look at that, that maybe the solution, I hadn't grasped the problem might be assembly compiled object path or something, cheers - PR: https://git.openjdk.java.net/jdk/pull/8124
Re: RFR: 8284437: Building from different users/workspace is not always deterministic [v2]
On Wed, 6 Apr 2022 13:30:28 GMT, Andrew Leonard wrote: >> This PR fixes the non-deterministic behavior when building on linux with >> different userids or within >> different workspace folders. >> It fixes the following issues: >> - MakeZipReproducible.java used to produce reproducible src.zip removes the >> optional zip "extra" field containing UID/GID. >> - When absolute output paths are not allowed, enable the use of >> -fdebug-prefix-map to ensure debug symbol info does not contain the top >> level workspace folder. >> - For reproducible builds ensure the gcc random symbol name generator is >> seeded using -frandom-seed. >> - For reproducible builds when producing debug symbols use relative object >> paths for library linking to remove absolute MASM object paths. >> >> Signed-off-by: Andrew Leonard > > Andrew Leonard has updated the pull request incrementally with one additional > commit since the last revision: > > Trigger checks @magicus now i've done a build without using -fdebug-prefix-map, and the debuginfo obviously contains lots of absolute paths to the "src" cpp/c/hpp/.. files, HOWEVER, if I grep for ".o" file names, I still only get the 3 hotspot assembly objects: andrew@andrew:~$ strings /home/andrew2/jdk/build/linux-aarch64-server-release/images/jdk/lib/server/libjvm.debuginfo | grep andrew | grep ".o" - PR: https://git.openjdk.java.net/jdk/pull/8124
Re: RFR: 8284437: Building from different users/workspace is not always deterministic [v2]
On Wed, 6 Apr 2022 13:30:28 GMT, Andrew Leonard wrote: >> This PR fixes the non-deterministic behavior when building on linux with >> different userids or within >> different workspace folders. >> It fixes the following issues: >> - MakeZipReproducible.java used to produce reproducible src.zip removes the >> optional zip "extra" field containing UID/GID. >> - When absolute output paths are not allowed, enable the use of >> -fdebug-prefix-map to ensure debug symbol info does not contain the top >> level workspace folder. >> - For reproducible builds ensure the gcc random symbol name generator is >> seeded using -frandom-seed. >> - For reproducible builds when producing debug symbols use relative object >> paths for library linking to remove absolute MASM object paths. >> >> Signed-off-by: Andrew Leonard > > Andrew Leonard has updated the pull request incrementally with one additional > commit since the last revision: > > Trigger checks @magicus Hmm, this is interesting, so with gcc 10.3, using -fdebug-prefix-map, I am seeing 3 absolute paths for the 3 hotspot assembly files, but note if I do a dwarfdump it shows no absolute paths, only the .FILE debuginfo strings have the path: andrew@andrew:~$ strings /home/andrew3/jdk/build/linux-aarch64-server-release/images/jdk/lib/server/libjvm.debuginfo | grep andrew - PR: https://git.openjdk.java.net/jdk/pull/8124
Re: RFR: 8284437: Building from different users/workspace is not always deterministic [v2]
On Fri, 8 Apr 2022 09:30:51 GMT, Andrew Leonard wrote: >> Actually, I think that the GNU assembler supports debug prefix mapping. See >> e.g. [this bug report](https://github.com/ocaml/ocaml/issues/10770), >> stating: "With GNU tools, to enable debug prefix map on C source we pass >> -fdebug-prefix-map to cc and --debug-prefix-map to as." >> >> @andrew-m-leonard Can you check if we can pass `--debug-prefix-map` to the >> assembler if on the gcc toolchain, and thus get rid of the relative linking? >> (And oh, speaking of that, the if statement here checks for target OS. It >> should, most likely, have checked for toolchain.) > > @magicus just saw your above post, interesting, i'll add I think gcc -v to my > above testing, to see what gcc is passing through to the actual compile, > apparently using verbose removes anything that gets removed for assembly, > ie.anything that is not relavent I did test with --debug-prefix-map on the assembly cmdline, it just seemed to be ignored, but I will try again - PR: https://git.openjdk.java.net/jdk/pull/8124
Re: RFR: 8284437: Building from different users/workspace is not always deterministic [v2]
On Fri, 8 Apr 2022 08:31:45 GMT, Magnus Ihse Bursie wrote: >> One way would be to check if we have either assembly source files, or >> anything in EXTRA_OBJ, and if so, do the relative linking. That would at >> least bring down collateral damage significantly, since the majority of libs >> have neither. >> >> It will still hurt the most important libs (like hotspot) but unless we >> can't find a way to get this working in some other way, it seems we'll have >> to accept that. >> >> @andrew-m-leonard Do you know if there is any bug on gcc tracking their >> inability to set debug prefix mapping for assembly files? > > Actually, I think that the GNU assembler supports debug prefix mapping. See > e.g. [this bug report](https://github.com/ocaml/ocaml/issues/10770), stating: > "With GNU tools, to enable debug prefix map on C source we pass > -fdebug-prefix-map to cc and --debug-prefix-map to as." > > @andrew-m-leonard Can you check if we can pass `--debug-prefix-map` to the > assembler if on the gcc toolchain, and thus get rid of the relative linking? > (And oh, speaking of that, the if statement here checks for target OS. It > should, most likely, have checked for toolchain.) @magicus thank you for looking at this, in hindsight I should have pinged you to review this as well. So a few thoughts, first one thing that crossed my mind which is the ifeq logic here should instead test ALLOW_ABSOLUTE_PATHS_IN_OUTPUT, exactly like the setting of -fdebug-prefix-map and -ffile-macro-prefix already do. In fact just changing to using that maybe sufficient? ie.only use relative paths here when ALLOW_ABSOLUTE_PATHS_IN_OUTPUT is false, ie. it is a "release" type build and not a developer/debug build, where CWD cmdlines are preferable. What's your thought on that? I did search google & bugzilla for any gcc bug, but didn't find anything. However, i've just found this issue: https://github.com/gcc-mirror/gcc/commit/d12153046816f955e74943af7089d30de6a00e19 which maybe related, although looks to do the opposite of what we want, although that maybe just because I don't understand it! However, it is the .file dwarf value for the assembly stripped debuginfo that has the full object path that isn't getting stripped, and I think we do use -flto. This fix is in gcc 11.2, i'm testing using gcc 10.3. I'm going to do two things, recreate the problem again, double check the assembly file compile options, with gcc 10.3, then try again with gcc 11.2.. @magicus just saw your above post, interesting, i'll add I think gcc -v to my above testing, to see what gcc is passing through to the actual compile, apparently using verbose removes anything that gets removed for assembly, ie.anything that is not relavent - PR: https://git.openjdk.java.net/jdk/pull/8124
Re: RFR: 8284437: Building from different users/workspace is not always deterministic [v2]
On Thu, 7 Apr 2022 15:51:30 GMT, Magnus Ihse Bursie wrote: >> Andrew Leonard has updated the pull request incrementally with one >> additional commit since the last revision: >> >> Trigger checks > > make/common/NativeCompilation.gmk line 1159: > >> 1157: ifeq ($(call isTargetOs, linux), true) >> 1158: ifeq ($$($1_COMPILE_WITH_DEBUG_SYMBOLS), true) >> 1159: $1_LINK_OBJS_RELATIVE := true > > I realize this PR has already been integrated, but I have some questions > about this. The comment talk about "ASM" objects (I assume that means output > of assembly files), but the code change is not restricted to assembly files. > > Afaict, this change means that all builds on linux with debug symbols and > reproducible builds will use relative paths! This is perhaps the most common > compilation scenario, and it would mean that our efforts to keep a CWD > neutral command line is basically in vain. :-( > > Or can anyone (@erikj79, @andrew-m-leonard) explain to me why this would not > be the case? @magicus So I did think about this, and the alternative I think would be to hard code a list of which object files are the output from assembly files, of which there is about a dozen. Unless we could capture somehow the list of assembled files as we go ...? As @erikj79 points out this is just for the "linking". As the problem is the assembly file objects don't have the equivalent of debug prefix mapping, and so when stripped the full file path of the object file ends up in the debuginfo file, and results in the libjvm.so non-determinism due to different debug CRC. - PR: https://git.openjdk.java.net/jdk/pull/8124
Re: RFR: 8284437: Building from different users/workspace is not always deterministic [v2]
On Thu, 7 Apr 2022 16:33:44 GMT, Erik Joelsson wrote: >> make/common/NativeCompilation.gmk line 1159: >> >>> 1157: ifeq ($(call isTargetOs, linux), true) >>> 1158: ifeq ($$($1_COMPILE_WITH_DEBUG_SYMBOLS), true) >>> 1159: $1_LINK_OBJS_RELATIVE := true >> >> I realize this PR has already been integrated, but I have some questions >> about this. The comment talk about "ASM" objects (I assume that means output >> of assembly files), but the code change is not restricted to assembly files. >> >> Afaict, this change means that all builds on linux with debug symbols and >> reproducible builds will use relative paths! This is perhaps the most common >> compilation scenario, and it would mean that our efforts to keep a CWD >> neutral command line is basically in vain. :-( >> >> Or can anyone (@erikj79, @andrew-m-leonard) explain to me why this would not >> be the case? > > You are correct for the linking command line. All the compilation command > lines are still handled with flags instead. What is CWD ? - PR: https://git.openjdk.java.net/jdk/pull/8124
Integrated: 8284437: Building from different users/workspace is not always deterministic
On Wed, 6 Apr 2022 10:27:40 GMT, Andrew Leonard wrote: > This PR fixes the non-deterministic behavior when building on linux with > different userids or within > different workspace folders. > It fixes the following issues: > - MakeZipReproducible.java used to produce reproducible src.zip removes the > optional zip "extra" field containing UID/GID. > - When absolute output paths are not allowed, enable the use of > -fdebug-prefix-map to ensure debug symbol info does not contain the top level > workspace folder. > - For reproducible builds ensure the gcc random symbol name generator is > seeded using -frandom-seed. > - For reproducible builds when producing debug symbols use relative object > paths for library linking to remove absolute MASM object paths. > > Signed-off-by: Andrew Leonard This pull request has now been integrated. Changeset: 4451257b Author:Andrew Leonard URL: https://git.openjdk.java.net/jdk/commit/4451257b1432e4180a16757aafca6141b8063772 Stats: 49 lines in 4 files changed: 47 ins; 0 del; 2 mod 8284437: Building from different users/workspace is not always deterministic Reviewed-by: erikj - PR: https://git.openjdk.java.net/jdk/pull/8124
Re: RFR: 8284437: Building from different users/workspace is not always deterministic [v2]
> This PR fixes the non-deterministic behavior when building on linux with > different userids or within > different workspace folders. > It fixes the following issues: > - MakeZipReproducible.java used to produce reproducible src.zip removes the > optional zip "extra" field containing UID/GID. > - When absolute output paths are not allowed, enable the use of > -fdebug-prefix-map to ensure debug symbol info does not contain the top level > workspace folder. > - For reproducible builds ensure the gcc random symbol name generator is > seeded using -frandom-seed. > - For reproducible builds when producing debug symbols use relative object > paths for library linking to remove absolute MASM object paths. > > Signed-off-by: Andrew Leonard Andrew Leonard has updated the pull request incrementally with one additional commit since the last revision: Trigger checks - Changes: - all: https://git.openjdk.java.net/jdk/pull/8124/files - new: https://git.openjdk.java.net/jdk/pull/8124/files/77edb59a..f2a842aa Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk=8124=01 - incr: https://webrevs.openjdk.java.net/?repo=jdk=8124=00-01 Stats: 0 lines in 0 files changed: 0 ins; 0 del; 0 mod Patch: https://git.openjdk.java.net/jdk/pull/8124.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/8124/head:pull/8124 PR: https://git.openjdk.java.net/jdk/pull/8124
RFR: 8284437: Building from different users/workspace is not always deterministic
This PR fixes the non-deterministic behavior when building on linux with different userids or within different workspace folders. It fixes the following issues: - MakeZipReproducible.java used to produce reproducible src.zip removes the optional zip "extra" field containing UID/GID. - When absolute output paths are not allowed, enable the use of -fdebug-prefix-map to ensure debug symbol info does not contain the top level workspace folder. - For reproducible builds ensure the gcc random symbol name generator is seeded using -frandom-seed. - For reproducible builds when producing debug symbols use relative object paths for library linking to remove absolute MASM object paths. Signed-off-by: Andrew Leonard - Commit messages: - 8284437: Building from different users/workspace is not always deterministic - 8284437: Building from different users/workspace is not always deterministic Changes: https://git.openjdk.java.net/jdk/pull/8124/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk=8124=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8284437 Stats: 49 lines in 4 files changed: 47 ins; 0 del; 2 mod Patch: https://git.openjdk.java.net/jdk/pull/8124.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/8124/head:pull/8124 PR: https://git.openjdk.java.net/jdk/pull/8124
Integrated: 8283315: jrt-fs.jar not always deterministically built
On Thu, 17 Mar 2022 11:09:24 GMT, Andrew Leonard wrote: > JarArchive.gmk uses an un-sorted jar @, thus depending on exactly > how that list gets ordered by relevant OS querys. Such queries can differ in > order on different CPU architectures (Intel vs AMD). > > This PR adds a "sort" to the jar @ contents. > > Signed-off-by: Andrew Leonard This pull request has now been integrated. Changeset: 85cc6f14 Author:Andrew Leonard URL: https://git.openjdk.java.net/jdk/commit/85cc6f1440aa7e073cab894cb9a72d9eed4e8369 Stats: 6 lines in 1 file changed: 3 ins; 0 del; 3 mod 8283315: jrt-fs.jar not always deterministically built Reviewed-by: ihse - PR: https://git.openjdk.java.net/jdk/pull/7852
Re: RFR: 8283315: jrt-fs.jar not always deterministically built [v5]
> JarArchive.gmk uses an un-sorted jar @, thus depending on exactly > how that list gets ordered by relevant OS querys. Such queries can differ in > order on different CPU architectures (Intel vs AMD). > > This PR adds a "sort" to the jar @ contents. > > Signed-off-by: Andrew Leonard Andrew Leonard has updated the pull request incrementally with one additional commit since the last revision: Retrigger checks - Changes: - all: https://git.openjdk.java.net/jdk/pull/7852/files - new: https://git.openjdk.java.net/jdk/pull/7852/files/3d97c595..b4613359 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk=7852=04 - incr: https://webrevs.openjdk.java.net/?repo=jdk=7852=03-04 Stats: 0 lines in 0 files changed: 0 ins; 0 del; 0 mod Patch: https://git.openjdk.java.net/jdk/pull/7852.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/7852/head:pull/7852 PR: https://git.openjdk.java.net/jdk/pull/7852
Re: RFR: 8283315: jrt-fs.jar not always deterministically built [v4]
> JarArchive.gmk uses an un-sorted jar @, thus depending on exactly > how that list gets ordered by relevant OS querys. Such queries can differ in > order on different CPU architectures (Intel vs AMD). > > This PR adds a "sort" to the jar @ contents. > > Signed-off-by: Andrew Leonard Andrew Leonard has updated the pull request incrementally with one additional commit since the last revision: Retrigger checks - Changes: - all: https://git.openjdk.java.net/jdk/pull/7852/files - new: https://git.openjdk.java.net/jdk/pull/7852/files/beab5888..3d97c595 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk=7852=03 - incr: https://webrevs.openjdk.java.net/?repo=jdk=7852=02-03 Stats: 0 lines in 0 files changed: 0 ins; 0 del; 0 mod Patch: https://git.openjdk.java.net/jdk/pull/7852.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/7852/head:pull/7852 PR: https://git.openjdk.java.net/jdk/pull/7852
Re: RFR: 8283315: jrt-fs.jar not always deterministically built [v3]
> JarArchive.gmk uses an un-sorted jar @, thus depending on exactly > how that list gets ordered by relevant OS querys. Such queries can differ in > order on different CPU architectures (Intel vs AMD). > > This PR adds a "sort" to the jar @ contents. > > Signed-off-by: Andrew Leonard Andrew Leonard has updated the pull request incrementally with one additional commit since the last revision: Retrigger checks - Changes: - all: https://git.openjdk.java.net/jdk/pull/7852/files - new: https://git.openjdk.java.net/jdk/pull/7852/files/897726be..beab5888 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk=7852=02 - incr: https://webrevs.openjdk.java.net/?repo=jdk=7852=01-02 Stats: 0 lines in 0 files changed: 0 ins; 0 del; 0 mod Patch: https://git.openjdk.java.net/jdk/pull/7852.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/7852/head:pull/7852 PR: https://git.openjdk.java.net/jdk/pull/7852
Re: RFR: 8283315: jrt-fs.jar not always deterministically built [v2]
> JarArchive.gmk uses an un-sorted jar @, thus depending on exactly > how that list gets ordered by relevant OS querys. Such queries can differ in > order on different CPU architectures (Intel vs AMD). > > This PR adds a "sort" to the jar @ contents. > > Signed-off-by: Andrew Leonard Andrew Leonard has updated the pull request incrementally with one additional commit since the last revision: 8283315: jrt-fs.jar not always deterministically built Signed-off-by: Andrew Leonard - Changes: - all: https://git.openjdk.java.net/jdk/pull/7852/files - new: https://git.openjdk.java.net/jdk/pull/7852/files/560c1425..897726be Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk=7852=01 - incr: https://webrevs.openjdk.java.net/?repo=jdk=7852=00-01 Stats: 2 lines in 1 file changed: 0 ins; 0 del; 2 mod Patch: https://git.openjdk.java.net/jdk/pull/7852.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/7852/head:pull/7852 PR: https://git.openjdk.java.net/jdk/pull/7852
Re: RFR: 8283315: jrt-fs.jar not always deterministically built [v2]
On Thu, 17 Mar 2022 11:28:55 GMT, Andrew Leonard wrote:
>> make/common/JarArchive.gmk line 196:
>>
>>> 194: if [ "`$(WC) -l $$($1_BIN)/_the.$$($1_JARNAME)_contents | $(AWK)
>>> '{ print 1 }'`" -gt "0" ]; then \
>>> 195: $(ECHO) " updating" `$(WC) -l
>>> $$($1_BIN)/_the.$$($1_JARNAME)_contents | $(AWK) '{ print 1 }'` files
>>> && \
>>> 196: $(CAT) $$($1_BIN)/_the.$$($1_JARNAME)_contents | $(SORT) >
>>> $$($1_BIN)/_the.$$($1_JARNAME)_contents_sorted && \
>>
>> I'm very grateful that you found this and fixed it, but this code could
>> really hand you a ["Useless Use of Cat
>> Award"](https://porkmail.org/era/unix/award#cat). :-D
>>
>> `sort in-file > out-file` is much better.
>
> I like to make my mark :-)
> i'm just so used to piping !
Fixed
-
PR: https://git.openjdk.java.net/jdk/pull/7852
Re: RFR: 8283315: jrt-fs.jar not always deterministically built
On Thu, 17 Mar 2022 11:20:36 GMT, Magnus Ihse Bursie wrote:
>> JarArchive.gmk uses an un-sorted jar @, thus depending on
>> exactly how that list gets ordered by relevant OS querys. Such queries can
>> differ in order on different CPU architectures (Intel vs AMD).
>>
>> This PR adds a "sort" to the jar @ contents.
>>
>> Signed-off-by: Andrew Leonard
>
> make/common/JarArchive.gmk line 196:
>
>> 194: if [ "`$(WC) -l $$($1_BIN)/_the.$$($1_JARNAME)_contents | $(AWK)
>> '{ print 1 }'`" -gt "0" ]; then \
>> 195: $(ECHO) " updating" `$(WC) -l
>> $$($1_BIN)/_the.$$($1_JARNAME)_contents | $(AWK) '{ print 1 }'` files &&
>> \
>> 196: $(CAT) $$($1_BIN)/_the.$$($1_JARNAME)_contents | $(SORT) >
>> $$($1_BIN)/_the.$$($1_JARNAME)_contents_sorted && \
>
> I'm very grateful that you found this and fixed it, but this code could
> really hand you a ["Useless Use of Cat
> Award"](https://porkmail.org/era/unix/award#cat). :-D
>
> `sort in-file > out-file` is much better.
I like to make my mark :-)
i'm just so used to piping !
-
PR: https://git.openjdk.java.net/jdk/pull/7852
RFR: 8283315: jrt-fs.jar not always deterministically built
JarArchive.gmk uses an un-sorted jar @, thus depending on exactly how that list gets ordered by relevant OS querys. Such queries can differ in order on different CPU architectures (Intel vs AMD). This PR adds a "sort" to the jar @ contents. Signed-off-by: Andrew Leonard - Commit messages: - 8283315: jrt-fs.jar not always deterministically built Changes: https://git.openjdk.java.net/jdk/pull/7852/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk=7852=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8283315 Stats: 6 lines in 1 file changed: 3 ins; 0 del; 3 mod Patch: https://git.openjdk.java.net/jdk/pull/7852.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/7852/head:pull/7852 PR: https://git.openjdk.java.net/jdk/pull/7852
Integrated: 8279834: Alpine Linux fails to build when --with-source-date enabled
On Tue, 11 Jan 2022 10:25:31 GMT, Andrew Leonard wrote: > The new reproducible build jar & jmod feature exposed a problem on Alpline > linux whereby when --with-source-date is specified the IS_GNU_DATE setting > was actually wrong. It was assuming the BusyBox "date" tool on Alpine linux > was BSD compatible, but it is actually GNU syntax compatible. > > This PR additionally checks for BusyBox when determining the date tool > version compatibility. > > Signed-off-by: Andrew Leonard This pull request has now been integrated. Changeset: 9e024476 Author:Andrew Leonard URL: https://git.openjdk.java.net/jdk/commit/9e0244762c0961b1bb3453d294531997b367e757 Stats: 4 lines in 1 file changed: 0 ins; 0 del; 4 mod 8279834: Alpine Linux fails to build when --with-source-date enabled Reviewed-by: erikj - PR: https://git.openjdk.java.net/jdk/pull/7025
Re: RFR: 8279834: Alpine Linux fails to build when --with-source-date enabled
On Tue, 11 Jan 2022 13:53:49 GMT, Erik Joelsson wrote: >> The new reproducible build jar & jmod feature exposed a problem on Alpline >> linux whereby when --with-source-date is specified the IS_GNU_DATE setting >> was actually wrong. It was assuming the BusyBox "date" tool on Alpine linux >> was BSD compatible, but it is actually GNU syntax compatible. >> >> This PR additionally checks for BusyBox when determining the date tool >> version compatibility. >> >> Signed-off-by: Andrew Leonard > > Marked as reviewed by erikj (Reviewer). thank you @erikj79 - PR: https://git.openjdk.java.net/jdk/pull/7025
RFR: 8279834: Alpine Linux fails to build when --with-source-date enabled
The new reproducible build jar & jmod feature exposed a problem on Alpline linux whereby when --with-source-date is specified the IS_GNU_DATE setting was actually wrong. It was assuming the BusyBox "date" tool on Alpine linux was BSD compatible, but it is actually GNU syntax compatible. This PR additionally checks for BusyBox when determining the date tool version compatibility. Signed-off-by: Andrew Leonard - Commit messages: - 8279834: Alpine Linux fails to build when --with-source-date enabled Changes: https://git.openjdk.java.net/jdk/pull/7025/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk=7025=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8279834 Stats: 4 lines in 1 file changed: 0 ins; 0 del; 4 mod Patch: https://git.openjdk.java.net/jdk/pull/7025.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/7025/head:pull/7025 PR: https://git.openjdk.java.net/jdk/pull/7025
Integrated: 8279182: MakeZipReproducible ZipEntry timestamps not localized to UTC
On Thu, 23 Dec 2021 09:39:19 GMT, Andrew Leonard wrote: > MakeZipReproducible was added to enable reproducible building of src.zip. > However, as ZipEntry timestamps are a "localized" date with no zone, the > specified epoch instant was getting localized in whatever the building > timezone was, hence src.zip built from the same source in different zones > would differ. > The timestamp should be localized to UTC (like for jar, jmod entries), this > PR ensures this. > > Signed-off-by: Andrew Leonard This pull request has now been integrated. Changeset: bc0466c7 Author:Andrew Leonard URL: https://git.openjdk.java.net/jdk/commit/bc0466c7ca57f14b1e6285e2a39755d57c8de376 Stats: 11 lines in 1 file changed: 6 ins; 0 del; 5 mod 8279182: MakeZipReproducible ZipEntry timestamps not localized to UTC Reviewed-by: erikj - PR: https://git.openjdk.java.net/jdk/pull/6926
Integrated: 8278766: Enable OpenJDK build support for reproducible jars and jmods using --date
On Fri, 17 Dec 2021 09:18:03 GMT, Andrew Leonard wrote: > If "reproducible build" is enabled, then utilize the new --date option in the > building of jar's and jmods. > Validating the boot jdk supports --date for the building of the jars. > > Signed-off-by: Andrew Leonard This pull request has now been integrated. Changeset: 214f98f6 Author:Andrew Leonard URL: https://git.openjdk.java.net/jdk/commit/214f98f6b07e312e6f4ded5364a94277114784e7 Stats: 65 lines in 7 files changed: 45 ins; 6 del; 14 mod 8278766: Enable OpenJDK build support for reproducible jars and jmods using --date Reviewed-by: erikj - PR: https://git.openjdk.java.net/jdk/pull/6878
RFR: 8279182: MakeZipReproducible ZipEntry timestamps not localized to UTC
MakeZipReproducible was added to enable reproducible building of src.zip. However, as ZipEntry timestamps are a "localized" date with no zone, the specified epoch instant was getting localized in whatever the building timezone was, hence src.zip built from the same source in different zones would differ. The timestamp should be localized to UTC (like for jar, jmod entries), this PR ensures this. Signed-off-by: Andrew Leonard - Commit messages: - 8279182: MakeZipReproducible ZipEntry timestamps not localized to UTC Changes: https://git.openjdk.java.net/jdk/pull/6926/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk=6926=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8279182 Stats: 11 lines in 1 file changed: 6 ins; 0 del; 5 mod Patch: https://git.openjdk.java.net/jdk/pull/6926.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/6926/head:pull/6926 PR: https://git.openjdk.java.net/jdk/pull/6926
Re: RFR: 8278766: Enable OpenJDK build support for reproducible jars and jmods using --date [v5]
> If "reproducible build" is enabled, then utilize the new --date option in the > building of jar's and jmods. > Validating the boot jdk supports --date for the building of the jars. > > Signed-off-by: Andrew Leonard Andrew Leonard has updated the pull request incrementally with one additional commit since the last revision: 8278766: Enable OpenJDK build support for reproducible jars and jmods using --date Signed-off-by: Andrew Leonard - Changes: - all: https://git.openjdk.java.net/jdk/pull/6878/files - new: https://git.openjdk.java.net/jdk/pull/6878/files/5fb4676b..add720cf Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk=6878=04 - incr: https://webrevs.openjdk.java.net/?repo=jdk=6878=03-04 Stats: 6 lines in 1 file changed: 4 ins; 0 del; 2 mod Patch: https://git.openjdk.java.net/jdk/pull/6878.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/6878/head:pull/6878 PR: https://git.openjdk.java.net/jdk/pull/6878
Re: RFR: 8278766: Enable OpenJDK build support for reproducible jars and jmods using --date [v4]
On Wed, 22 Dec 2021 17:30:36 GMT, Erik Joelsson wrote: >> Andrew Leonard has updated the pull request incrementally with three >> additional commits since the last revision: >> >> - 8278766: Enable OpenJDK build support for reproducible jars and jmods >> using --date >> >>Signed-off-by: Andrew Leonard >> - 8278766: Enable OpenJDK build support for reproducible jars and jmods >> using --date >> >>Signed-off-by: Andrew Leonard >> - 8278766: Enable OpenJDK build support for reproducible jars and jmods >> using --date >> >>Signed-off-by: Andrew Leonard > > make/InitSupport.gmk line 320: > >> 318: else >> 319: export SOURCE_DATE_ISO_8601 := $$(shell $$(DATE) -u -j -f "%s" >> "$$(SOURCE_DATE_EPOCH)" +"%Y-%m-%dT%H:%M:%SZ" 2> /dev/null) >> 320: endif > > This is starting to look pretty good, but I would ask you to try to break > these long lines. We try to keep the makefiles reasonably narrow for easier > future 3-way merges and diff viewing. ah yes, fixed - PR: https://git.openjdk.java.net/jdk/pull/6878
Re: RFR: 8278766: Enable OpenJDK build support for reproducible jars and jmods using --date [v4]
On Fri, 17 Dec 2021 20:04:20 GMT, Andrew Leonard wrote: >> make/common/MakeBase.gmk line 148: >> >>> 146: ifeq ($(SOURCE_DATE_ISO_8601), ) >>> 147: # GNU date format did not work, try BSD date options >>> 148: SOURCE_DATE_ISO_8601 := $(shell $(DATE) -u -j -f "%s" >>> "$(SOURCE_DATE_EPOCH)" +"%Y-%m-%dT%H:%M:%SZ" 2> /dev/null) >> >> Could we maybe figure out the date command line to use in configure instead >> to avoid the trial and error at build time? > > I based this trial and error based on util.m4 code and copied the method, so > yes I can store the result from then... Added IS_GNU_DATE variable - PR: https://git.openjdk.java.net/jdk/pull/6878
Re: RFR: 8278766: Enable OpenJDK build support for reproducible jars and jmods using --date [v3]
On Wed, 22 Dec 2021 15:28:52 GMT, Andrew Leonard wrote: >> make/InitSupport.gmk line 317: >> >>> 315: export SOURCE_DATE_EPOCH := $$(SOURCE_DATE) >>> 316: ifeq ($$(IS_GNU_DATE), yes) >>> 317: SOURCE_DATE_ISO_8601 := $(shell $(DATE) --utc >>> --date="@$(SOURCE_DATE_EPOCH)" +"%Y-%m-%dT%H:%M:%SZ" 2> /dev/null) >> >> Are you getting this to work without `export`? I would have expected that to >> be needed. > > @erikj79 fixed it I think, problems is above not enough $'s !! for > $(shell... resolved, thanks - PR: https://git.openjdk.java.net/jdk/pull/6878
Re: RFR: 8278766: Enable OpenJDK build support for reproducible jars and jmods using --date [v4]
On Fri, 17 Dec 2021 17:47:02 GMT, Erik Joelsson wrote: >> Andrew Leonard has updated the pull request incrementally with three >> additional commits since the last revision: >> >> - 8278766: Enable OpenJDK build support for reproducible jars and jmods >> using --date >> >>Signed-off-by: Andrew Leonard >> - 8278766: Enable OpenJDK build support for reproducible jars and jmods >> using --date >> >>Signed-off-by: Andrew Leonard >> - 8278766: Enable OpenJDK build support for reproducible jars and jmods >> using --date >> >>Signed-off-by: Andrew Leonard > > make/common/JarArchive.gmk line 234: > >> 232: endif >> 233: else >> 234: $1_JAR_OPTIONS := $$($1_JAR_SOURCE_DATE) > > Instead of adding $$($1_JAR_SOURCE_DATE) to every branch of this conditional, > you can use += to build the $1_JAR_OPTIONS variable one step at a time. I > don't think we need a separate variable JAR_SOURCE_DATE. reduced as stated - PR: https://git.openjdk.java.net/jdk/pull/6878
Re: RFR: 8278766: Enable OpenJDK build support for reproducible jars and jmods using --date [v4]
> If "reproducible build" is enabled, then utilize the new --date option in the > building of jar's and jmods. > Validating the boot jdk supports --date for the building of the jars. > > Signed-off-by: Andrew Leonard Andrew Leonard has updated the pull request incrementally with three additional commits since the last revision: - 8278766: Enable OpenJDK build support for reproducible jars and jmods using --date Signed-off-by: Andrew Leonard - 8278766: Enable OpenJDK build support for reproducible jars and jmods using --date Signed-off-by: Andrew Leonard - 8278766: Enable OpenJDK build support for reproducible jars and jmods using --date Signed-off-by: Andrew Leonard - Changes: - all: https://git.openjdk.java.net/jdk/pull/6878/files - new: https://git.openjdk.java.net/jdk/pull/6878/files/75c9d692..5fb4676b Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk=6878=03 - incr: https://webrevs.openjdk.java.net/?repo=jdk=6878=02-03 Stats: 3 lines in 2 files changed: 0 ins; 0 del; 3 mod Patch: https://git.openjdk.java.net/jdk/pull/6878.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/6878/head:pull/6878 PR: https://git.openjdk.java.net/jdk/pull/6878
Re: RFR: 8278766: Enable OpenJDK build support for reproducible jars and jmods using --date [v4]
On Fri, 17 Dec 2021 20:08:43 GMT, Andrew Leonard wrote: >> But I think the code in InitSupport will be executed always; Init.gmk is our >> "bootstrapper" / "trampoline" which wraps all calls to make (and >> InitSupport.gmk contains gory implementation details of Init.gmk). > > @magicus don't worry, enjoy your holiday, I think @erik has pointed out it > should work, so I probably made a mistake when I tried it... Moved export SOURCE_DATE_ISO_8601 to InitSupport as suggested, found the problem as to why it didn't work for me, I was missing a "$" ! - PR: https://git.openjdk.java.net/jdk/pull/6878
Re: RFR: 8278766: Enable OpenJDK build support for reproducible jars and jmods using --date [v3]
On Wed, 22 Dec 2021 13:41:53 GMT, Erik Joelsson wrote: >> Andrew Leonard has updated the pull request incrementally with one >> additional commit since the last revision: >> >> 8278766: Enable OpenJDK build support for reproducible jars and jmods >> using --date >> >> Signed-off-by: Andrew Leonard > > make/InitSupport.gmk line 317: > >> 315: export SOURCE_DATE_EPOCH := $$(SOURCE_DATE) >> 316: ifeq ($$(IS_GNU_DATE), yes) >> 317: SOURCE_DATE_ISO_8601 := $(shell $(DATE) --utc >> --date="@$(SOURCE_DATE_EPOCH)" +"%Y-%m-%dT%H:%M:%SZ" 2> /dev/null) > > Are you getting this to work without `export`? I would have expected that to > be needed. @erikj79 fixed it I think, problems is above not enough $'s !! for $(shell... - PR: https://git.openjdk.java.net/jdk/pull/6878
Re: RFR: 8278766: Enable OpenJDK build support for reproducible jars and jmods using --date [v3]
On Wed, 22 Dec 2021 13:41:53 GMT, Erik Joelsson wrote: >> Andrew Leonard has updated the pull request incrementally with one >> additional commit since the last revision: >> >> 8278766: Enable OpenJDK build support for reproducible jars and jmods >> using --date >> >> Signed-off-by: Andrew Leonard > > make/InitSupport.gmk line 317: > >> 315: export SOURCE_DATE_EPOCH := $$(SOURCE_DATE) >> 316: ifeq ($$(IS_GNU_DATE), yes) >> 317: SOURCE_DATE_ISO_8601 := $(shell $(DATE) --utc >> --date="@$(SOURCE_DATE_EPOCH)" +"%Y-%m-%dT%H:%M:%SZ" 2> /dev/null) > > Are you getting this to work without `export`? I would have expected that to > be needed. @erikj79 right, so I was using the wrong configuration and reproducible-build was not set, now I am, I can see I am where I was before in that SOURCE_DATE_ISO_8601 being set in InitSupport.gmk is not visible in JarArchive.gmk, i've tried both with and without the export...? - PR: https://git.openjdk.java.net/jdk/pull/6878
Re: RFR: 8278766: Enable OpenJDK build support for reproducible jars and jmods using --date [v3]
On Wed, 22 Dec 2021 13:50:32 GMT, Andrew Leonard wrote: >> make/InitSupport.gmk line 317: >> >>> 315: export SOURCE_DATE_EPOCH := $$(SOURCE_DATE) >>> 316: ifeq ($$(IS_GNU_DATE), yes) >>> 317: SOURCE_DATE_ISO_8601 := $(shell $(DATE) --utc >>> --date="@$(SOURCE_DATE_EPOCH)" +"%Y-%m-%dT%H:%M:%SZ" 2> /dev/null) >> >> Are you getting this to work without `export`? I would have expected that to >> be needed. > > just checking log now you're right ! - PR: https://git.openjdk.java.net/jdk/pull/6878
Re: RFR: 8278766: Enable OpenJDK build support for reproducible jars and jmods using --date [v3]
On Wed, 22 Dec 2021 13:41:53 GMT, Erik Joelsson wrote: >> Andrew Leonard has updated the pull request incrementally with one >> additional commit since the last revision: >> >> 8278766: Enable OpenJDK build support for reproducible jars and jmods >> using --date >> >> Signed-off-by: Andrew Leonard > > make/InitSupport.gmk line 317: > >> 315: export SOURCE_DATE_EPOCH := $$(SOURCE_DATE) >> 316: ifeq ($$(IS_GNU_DATE), yes) >> 317: SOURCE_DATE_ISO_8601 := $(shell $(DATE) --utc >> --date="@$(SOURCE_DATE_EPOCH)" +"%Y-%m-%dT%H:%M:%SZ" 2> /dev/null) > > Are you getting this to work without `export`? I would have expected that to > be needed. just checking log now - PR: https://git.openjdk.java.net/jdk/pull/6878
Re: RFR: 8278766: Enable OpenJDK build support for reproducible jars and jmods using --date [v3]
> If "reproducible build" is enabled, then utilize the new --date option in the > building of jar's and jmods. > Validating the boot jdk supports --date for the building of the jars. > > Signed-off-by: Andrew Leonard Andrew Leonard has updated the pull request incrementally with one additional commit since the last revision: 8278766: Enable OpenJDK build support for reproducible jars and jmods using --date Signed-off-by: Andrew Leonard - Changes: - all: https://git.openjdk.java.net/jdk/pull/6878/files - new: https://git.openjdk.java.net/jdk/pull/6878/files/e0fe5b75..75c9d692 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk=6878=02 - incr: https://webrevs.openjdk.java.net/?repo=jdk=6878=01-02 Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod Patch: https://git.openjdk.java.net/jdk/pull/6878.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/6878/head:pull/6878 PR: https://git.openjdk.java.net/jdk/pull/6878
Re: RFR: 8278766: Enable OpenJDK build support for reproducible jars and jmods using --date [v2]
> If "reproducible build" is enabled, then utilize the new --date option in the > building of jar's and jmods. > Validating the boot jdk supports --date for the building of the jars. > > Signed-off-by: Andrew Leonard Andrew Leonard has updated the pull request incrementally with three additional commits since the last revision: - 8278766: Enable OpenJDK build support for reproducible jars and jmods using --date Signed-off-by: Andrew Leonard - 8278766: Enable OpenJDK build support for reproducible jars and jmods using --date Signed-off-by: Andrew Leonard - 8278766: Enable OpenJDK build support for reproducible jars and jmods using --date Signed-off-by: Andrew Leonard - Changes: - all: https://git.openjdk.java.net/jdk/pull/6878/files - new: https://git.openjdk.java.net/jdk/pull/6878/files/b89ff538..e0fe5b75 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk=6878=01 - incr: https://webrevs.openjdk.java.net/?repo=jdk=6878=00-01 Stats: 35 lines in 6 files changed: 12 ins; 14 del; 9 mod Patch: https://git.openjdk.java.net/jdk/pull/6878.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/6878/head:pull/6878 PR: https://git.openjdk.java.net/jdk/pull/6878
Re: RFR: 8278766: Enable OpenJDK build support for reproducible jars and jmods using --date
On Fri, 17 Dec 2021 16:29:21 GMT, Magnus Ihse Bursie wrote: >> Oh, that's ... interesting. (I'm pretty sure I wrote that code myself :)) >> >> I still think it would be good to keep the new code close to the old. If we >> set SOURCE_DATE to "updated", I think that should reflect in >> SOURCE_DATE_ISO_8601 as well. Maybe it does by the current design, but if it >> does, it could be more obvious. >> >> I'm sorry I don't have any ready-made suggestion. :( I'm really on vacation >> now and can't really dive into this, so if you can't find any better >> solution, then this'll have to do. > > But I think the code in InitSupport will be executed always; Init.gmk is our > "bootstrapper" / "trampoline" which wraps all calls to make (and > InitSupport.gmk contains gory implementation details of Init.gmk). @magicus don't worry, enjoy your holiday, I think @erik has pointed out it should work, so I probably made a mistake when I tried it... - PR: https://git.openjdk.java.net/jdk/pull/6878
Re: RFR: 8278766: Enable OpenJDK build support for reproducible jars and jmods using --date
On Fri, 17 Dec 2021 17:53:12 GMT, Erik Joelsson wrote: >> But I think the code in InitSupport will be executed always; Init.gmk is our >> "bootstrapper" / "trampoline" which wraps all calls to make (and >> InitSupport.gmk contains gory implementation details of Init.gmk). > > SOURCE_DATE_EPOCH is initialized and exported in InitSupport.gmk so it's > always available in the environment. We did this because we want various > tools to pick this variable up from the environment, as this is a commonly > expected variable name for doing so. The new variable SOURCE_DATE_ISO_8601 is > a variant with a different format and there is no standard for reading this > from the environment, so it's not as obvious that we should just export it > the same way. > > On the other hand, we do not want to execute a shell expression every time we > import MakeBase.gmk, so I would still vote for doing this in InitSupport.gmk > and export SOURCE_DATE_ISO_8601. @erikj79 Interesting, I tried adding it to InitSupport.gmk, but it wasn't always set... odd, I will revisit, I probably did something wrong...! will give that another try, thanks - PR: https://git.openjdk.java.net/jdk/pull/6878
Re: RFR: 8278766: Enable OpenJDK build support for reproducible jars and jmods using --date
On Fri, 17 Dec 2021 17:54:46 GMT, Erik Joelsson wrote: >> If "reproducible build" is enabled, then utilize the new --date option in >> the building of jar's and jmods. >> Validating the boot jdk supports --date for the building of the jars. >> >> Signed-off-by: Andrew Leonard > > make/common/MakeBase.gmk line 148: > >> 146: ifeq ($(SOURCE_DATE_ISO_8601), ) >> 147: # GNU date format did not work, try BSD date options >> 148: SOURCE_DATE_ISO_8601 := $(shell $(DATE) -u -j -f "%s" >> "$(SOURCE_DATE_EPOCH)" +"%Y-%m-%dT%H:%M:%SZ" 2> /dev/null) > > Could we maybe figure out the date command line to use in configure instead > to avoid the trial and error at build time? I based this trial and error based on util.m4 code and copied the method, so yes I can store the result from then... - PR: https://git.openjdk.java.net/jdk/pull/6878
Re: RFR: 8278766: Enable OpenJDK build support for reproducible jars and jmods using --date
On Fri, 17 Dec 2021 11:18:10 GMT, Magnus Ihse Bursie wrote: >> If "reproducible build" is enabled, then utilize the new --date option in >> the building of jar's and jmods. >> Validating the boot jdk supports --date for the building of the jars. >> >> Signed-off-by: Andrew Leonard > > make/common/MakeBase.gmk line 145: > >> 143: # If reproducible builds are enabled then set SOURCE_DATE_ISO_8601 >> string variable >> 144: ifeq ($(ENABLE_REPRODUCIBLE_BUILD), true) >> 145: SOURCE_DATE_ISO_8601 := $(shell $(DATE) --utc >> --date="@$(SOURCE_DATE_EPOCH)" +"%Y-%m-%dT%H:%M:%SZ" 2> /dev/null) > > This should be set up by the configure script. Look for where we are > currently doing the SOURCE_DATE_EPOCH stuff. > > Also consider the possibility to *just* use this variable, and pass it on to > jar if it is present, and not if it's missing. That way you can get rid of > the redundant boolean variable. @magicus I looked at that, but the problem is SOURCE_DATE_EPOCH is set in InitSupport.gmk depending on whether SOURCE_DATE=="updated" : https://github.com/openjdk/jdk/blob/739769c8fc4b496f08a92225a12d07414537b6c0/make/InitSupport.gmk#L315 I also couldn't add it in InitSupport.gmk because that marco is not included from every place SetupJarArchive is resolved. Thoughts? - PR: https://git.openjdk.java.net/jdk/pull/6878
RFR: 8278766: Enable OpenJDK build support for reproducible jars and jmods using --date
If "reproducible build" is enabled, then utilize the new --date option in the building of jar's and jmods. Validating the boot jdk supports --date for the building of the jars. Signed-off-by: Andrew Leonard - Commit messages: - 8278766: Enable OpenJDK build support for reproducible jars and jmods using --date Changes: https://git.openjdk.java.net/jdk/pull/6878/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk=6878=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8278766 Stats: 55 lines in 6 files changed: 40 ins; 3 del; 12 mod Patch: https://git.openjdk.java.net/jdk/pull/6878.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/6878/head:pull/6878 PR: https://git.openjdk.java.net/jdk/pull/6878
Integrated: 8278163: --with-cacerts-src variable resolved after GenerateCacerts recipe setup
On Thu, 2 Dec 2021 21:07:30 GMT, Andrew Leonard wrote: > PR https://github.com/openjdk/jdk/pull/6647 resolved the GENDATA_CACERTS_SRC > fir --with-cacerts-src after the recipe, which meant the dependencies were > wrong. > This PR moves it before the recipe. > > Signed-off-by: Andrew Leonard This pull request has now been integrated. Changeset: 45da3aea Author:Andrew Leonard URL: https://git.openjdk.java.net/jdk/commit/45da3aea22fd85f214e661b2c98631cb91ddb55d Stats: 8 lines in 1 file changed: 4 ins; 3 del; 1 mod 8278163: --with-cacerts-src variable resolved after GenerateCacerts recipe setup Reviewed-by: ihse - PR: https://git.openjdk.java.net/jdk/pull/6680
RFR: 8278163: --with-cacerts-src variable resolved after GenerateCacerts recipe setup
PR https://github.com/openjdk/jdk/pull/6647 resolved the GENDATA_CACERTS_SRC fir --with-cacerts-src after the recipe, which meant the dependencies were wrong. This PR moves it before the recipe. Signed-off-by: Andrew Leonard - Commit messages: - 8278163: --with-cacerts-src variable resolved after GenerateCacerts recipe setup Changes: https://git.openjdk.java.net/jdk/pull/6680/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk=6680=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8278163 Stats: 8 lines in 1 file changed: 4 ins; 3 del; 1 mod Patch: https://git.openjdk.java.net/jdk/pull/6680.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/6680/head:pull/6680 PR: https://git.openjdk.java.net/jdk/pull/6680
Re: RFR: 8278080: Add --with-cacerts-src='user cacerts folder' to enable deterministic cacerts generation [v2]
On Thu, 2 Dec 2021 18:46:09 GMT, Erik Joelsson wrote: >> this was my understanding: >> https://www.gnu.org/software/make/manual/html_node/Variables-in-Recipes.html >> >> This occurs after make has finished reading all the makefiles and the target >> is determined to be out of date; so, the recipes for targets which are not >> rebuilt are never expanded. >> >> but i'm going to double check I was checking the resultant cacerts correctly >> in my tests > > Oh, I didn't expand the diff far enough to actually see the context correctly > when I reviewed this as I would never have imagined the conditional to be > placed after the rule. While this will work as so far as using the correct > files, incremental builds will not be correct, because the rules are defined > in the first pass. > > I very much agree with Magnus that this conditional belongs around line 63. yes, thanks, feeling rather stupid here! i'll raise an issue to fix - PR: https://git.openjdk.java.net/jdk/pull/6647
Re: RFR: 8278080: Add --with-cacerts-src='user cacerts folder' to enable deterministic cacerts generation [v2]
On Thu, 2 Dec 2021 17:48:04 GMT, Andrew Leonard wrote: >> you make a valid point, but i've tested this numerous times, but let me >> check again > > my assumption was the recipe gets resolved later this was my understanding: https://www.gnu.org/software/make/manual/html_node/Variables-in-Recipes.html This occurs after make has finished reading all the makefiles and the target is determined to be out of date; so, the recipes for targets which are not rebuilt are never expanded. but i'm going to double check I was checking the resultant cacerts correctly in my tests - PR: https://git.openjdk.java.net/jdk/pull/6647
Re: RFR: 8278080: Add --with-cacerts-src='user cacerts folder' to enable deterministic cacerts generation [v2]
On Thu, 2 Dec 2021 17:46:35 GMT, Andrew Leonard wrote: >> I would have expected to see something like: >> >> ifneq ($(CACERTS_SRC), ) >> GENDATA_CACERTS_SRC := $(CACERTS_SRC) >> else >> GENDATA_CACERTS_SRC := $(TOPDIR)/make/data/cacerts/ >> endif >> >> at line 63. > > you make a valid point, but i've tested this numerous times, but let me check > again my assumption was the recipe gets resolved later - PR: https://git.openjdk.java.net/jdk/pull/6647
Re: RFR: 8278080: Add --with-cacerts-src='user cacerts folder' to enable deterministic cacerts generation [v2]
On Thu, 2 Dec 2021 17:35:36 GMT, Magnus Ihse Bursie wrote: >> make/modules/java.base/Gendata.gmk line 76: >> >>> 74: ifneq ($(CACERTS_SRC), ) >>> 75: GENDATA_CACERTS_SRC := $(CACERTS_SRC) >>> 76: endif >> >> Does this even work?! You are reassigning the variable after it has been >> used. The := assignment means that it not a macro. > > I would have expected to see something like: > > ifneq ($(CACERTS_SRC), ) > GENDATA_CACERTS_SRC := $(CACERTS_SRC) > else > GENDATA_CACERTS_SRC := $(TOPDIR)/make/data/cacerts/ > endif > > at line 63. you make a valid point, but i've tested this numerous times, but let me check again - PR: https://git.openjdk.java.net/jdk/pull/6647
Integrated: 8278080: Add --with-cacerts-src='user cacerts folder' to enable deterministic cacerts generation
On Wed, 1 Dec 2021 18:30:06 GMT, Andrew Leonard wrote: > Addition of a configure option --with-cacerts-src='user cacerts folder' to > allow developers to specify their own cacerts PEM folder for generation of > the cacerts store using the deterministic openjdk GenerateCacerts tool. > > Signed-off-by: Andrew Leonard This pull request has now been integrated. Changeset: dc2abc9f Author: Andrew Leonard URL: https://git.openjdk.java.net/jdk/commit/dc2abc9f05c2b7c52aeb242082359c48963f9854 Stats: 22 lines in 3 files changed: 22 ins; 0 del; 0 mod 8278080: Add --with-cacerts-src='user cacerts folder' to enable deterministic cacerts generation Reviewed-by: erikj - PR: https://git.openjdk.java.net/jdk/pull/6647
Re: RFR: 8278080: Add --with-cacerts-src='user cacerts folder' to enable deterministic cacerts generation [v2]
On Thu, 2 Dec 2021 14:29:00 GMT, Sean Mullan wrote: > I don’t have any major concerns with this change, as long as the default > cacerts are still the ones that are in the JDK. As an aside, using Mozilla's > root certificates might be fine for TLS certificates, but if you need to > support code signing certificates you may run into issues with missing CAs as > Mozilla's Root program does not support that use case. Also, by overriding > the roots included in the JDK, you are taking on the responsibility (which is > significant, in my opinion) of ensuring that those roots are trusted and have > not been compromised, revoked, have weak keys, etc. @seanjmullan Thanks Sean, I'll pass your comment on, cheers Andrew - PR: https://git.openjdk.java.net/jdk/pull/6647
Re: RFR: 8278080: Add --with-cacerts-src='user cacerts folder' to enable deterministic cacerts generation [v2]
On Wed, 1 Dec 2021 18:47:41 GMT, Erik Joelsson wrote: >> Andrew Leonard has updated the pull request with a new target base due to a >> merge or a rebase. The incremental webrev excludes the unrelated changes >> brought in by the merge/rebase. The pull request contains four additional >> commits since the last revision: >> >> - 8278080: Add --with-cacerts-src='user cacerts folder' to enable >> deterministic cacerts generation >> >>Signed-off-by: Andrew Leonard >> - Merge branch 'master' of https://github.com/openjdk/jdk into cacertssrc >> - 8278080: Add --with-cacerts-src='user cacerts folder' to enable >> determinsitic cacerts generation >> >>Signed-off-by: Andrew Leonard >> - 8278080: Add --with-cacerts-src='user cacerts folder' to enable >> determinsitic cacerts generation >> >>Signed-off-by: Andrew Leonard > > make/autoconf/jdk-options.m4 line 176: > >> 174: [specify alternative cacerts source folder containing >> certificates])]) >> 175: AC_MSG_CHECKING([for cacerts source]) >> 176: if test "x$with_cacerts_src" == x; then > > Before this if block, please assign an empty value to CACERTS_SRC. Otherwise, > if the user happens to have that variable set in the environment, that value > will get recorded by configure, which is usually not something we want. done - PR: https://git.openjdk.java.net/jdk/pull/6647
Re: RFR: 8278080: Add --with-cacerts-src='user cacerts folder' to enable deterministic cacerts generation [v2]
> Addition of a configure option --with-cacerts-src='user cacerts folder' to > allow developers to specify their own cacerts PEM folder for generation of > the cacerts store using the deterministic openjdk GenerateCacerts tool. > > Signed-off-by: Andrew Leonard Andrew Leonard has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains four additional commits since the last revision: - 8278080: Add --with-cacerts-src='user cacerts folder' to enable deterministic cacerts generation Signed-off-by: Andrew Leonard - Merge branch 'master' of https://github.com/openjdk/jdk into cacertssrc - 8278080: Add --with-cacerts-src='user cacerts folder' to enable determinsitic cacerts generation Signed-off-by: Andrew Leonard - 8278080: Add --with-cacerts-src='user cacerts folder' to enable determinsitic cacerts generation Signed-off-by: Andrew Leonard - Changes: - all: https://git.openjdk.java.net/jdk/pull/6647/files - new: https://git.openjdk.java.net/jdk/pull/6647/files/1084c4e1..16c5ca6b Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk=6647=01 - incr: https://webrevs.openjdk.java.net/?repo=jdk=6647=00-01 Stats: 1879 lines in 62 files changed: 1045 ins; 297 del; 537 mod Patch: https://git.openjdk.java.net/jdk/pull/6647.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/6647/head:pull/6647 PR: https://git.openjdk.java.net/jdk/pull/6647
Re: RFR: 8278080: Add --with-cacerts-src='user cacerts folder' to enable deterministic cacerts generation
On Thu, 2 Dec 2021 00:09:31 GMT, Sergey Bylokhov wrote: > I have a question related to the custom cacerts which can be added to the > OpenJDK bundle. How do you pass the tests like > test/jdk/sun/security/lib/cacerts/VerifyCACerts.java using that custom jdk > bundle? Probably we can add an additional configuration to that test so it > will check the custom cacerts passed to the build as well? @mrserb So VerifyCACerts is specific to the make/data/cacerts certificates, the README specifically states there that when those are updated VerifyCACerts needs updating. It checks things like fingerprints etc.. If a developer or other provider decide to provide their own cacerts file, then it is up to them to have verified and trust those certificates. They won't run the VerifyCACerts which is specific to the openjdk certs. This is the case at Adoptium for example, which uses the Mozilla trusted CA certs. - PR: https://git.openjdk.java.net/jdk/pull/6647
RFR: 8278080: Add --with-cacerts-src='user cacerts folder' to enable determinsitic cacerts generation
Addition of a configure option --with-cacerts-src='user cacerts folder' to allow developers to specify their own cacerts PEM folder for generation of the cacerts store using the deterministic openjdk GenerateCacerts tool. Signed-off-by: Andrew Leonard - Commit messages: - 8278080: Add --with-cacerts-src='user cacerts folder' to enable determinsitic cacerts generation - 8278080: Add --with-cacerts-src='user cacerts folder' to enable determinsitic cacerts generation Changes: https://git.openjdk.java.net/jdk/pull/6647/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk=6647=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8278080 Stats: 21 lines in 3 files changed: 21 ins; 0 del; 0 mod Patch: https://git.openjdk.java.net/jdk/pull/6647.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/6647/head:pull/6647 PR: https://git.openjdk.java.net/jdk/pull/6647
Integrated: 8277762: Allow configuration of HOTSPOT_BUILD_USER
On Wed, 24 Nov 2021 16:49:31 GMT, Andrew Leonard wrote: > To better allow "reproducible builds", a new configure parameter is added to > set the USERNAME build variable, rather than always using the current user: > --with-build-user= > HOTSPOT_BUILD_USER is then set to a reproducible USERNAME if required. > > Signed-off-by: Andrew Leonard This pull request has now been integrated. Changeset: f41e768b Author:Andrew Leonard URL: https://git.openjdk.java.net/jdk/commit/f41e768bba2b2ce3b3cc5813ccb1ac4984dcefbd Stats: 17 lines in 2 files changed: 11 ins; 5 del; 1 mod 8277762: Allow configuration of HOTSPOT_BUILD_USER Reviewed-by: erikj - PR: https://git.openjdk.java.net/jdk/pull/6542
Re: RFR: 8277762: Allow configuration of HOTSPOT_BUILD_USER [v2]
On Wed, 1 Dec 2021 15:22:09 GMT, Andrew Leonard wrote: >> To better allow "reproducible builds", a new configure parameter is added to >> set the USERNAME build variable, rather than always using the current user: >> --with-build-user= >> HOTSPOT_BUILD_USER is then set to a reproducible USERNAME if required. >> >> Signed-off-by: Andrew Leonard > > Andrew Leonard has updated the pull request with a new target base due to a > merge or a rebase. The incremental webrev excludes the unrelated changes > brought in by the merge/rebase. The pull request contains five additional > commits since the last revision: > > - 8277762: Allow configuration of HOTSPOT_BUILD_USER > >Signed-off-by: Andrew Leonard > - Merge branch 'master' of https://github.com/openjdk/jdk into builduser > - 8277762: Allow configuration of HOTSPOT_BUILD_USER > >Signed-off-by: Andrew Leonard > - 8277762: Allow configuration of HOTSPOT_BUILD_USER > >Signed-off-by: Andrew Leonard > - 8277762: Allow configuration of HOTSPOT_BUILD_USER > >Signed-off-by: Andrew Leonard @erikj79 ready for review again please thanks - PR: https://git.openjdk.java.net/jdk/pull/6542
Re: RFR: 8277762: Allow configuration of HOTSPOT_BUILD_USER [v2]
> To better allow "reproducible builds", a new configure parameter is added to > set the USERNAME build variable, rather than always using the current user: > --with-build-user= > HOTSPOT_BUILD_USER is then set to a reproducible USERNAME if required. > > Signed-off-by: Andrew Leonard Andrew Leonard has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains five additional commits since the last revision: - 8277762: Allow configuration of HOTSPOT_BUILD_USER Signed-off-by: Andrew Leonard - Merge branch 'master' of https://github.com/openjdk/jdk into builduser - 8277762: Allow configuration of HOTSPOT_BUILD_USER Signed-off-by: Andrew Leonard - 8277762: Allow configuration of HOTSPOT_BUILD_USER Signed-off-by: Andrew Leonard - 8277762: Allow configuration of HOTSPOT_BUILD_USER Signed-off-by: Andrew Leonard - Changes: - all: https://git.openjdk.java.net/jdk/pull/6542/files - new: https://git.openjdk.java.net/jdk/pull/6542/files/f9fb274a..3896df55 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk=6542=01 - incr: https://webrevs.openjdk.java.net/?repo=jdk=6542=00-01 Stats: 6325 lines in 236 files changed: 4145 ins; 1326 del; 854 mod Patch: https://git.openjdk.java.net/jdk/pull/6542.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/6542/head:pull/6542 PR: https://git.openjdk.java.net/jdk/pull/6542
Re: RFR: 8277762: Allow configuration of HOTSPOT_BUILD_USER [v2]
On Wed, 24 Nov 2021 18:59:03 GMT, Erik Joelsson wrote: >> it's also used in make/hotspot/lib/CompileJvm.gmk, but agree moving to >> jdk-versions.m4 makes sense > > Right, I meant the only place it's used within configure. done, moved - PR: https://git.openjdk.java.net/jdk/pull/6542
Re: RFR: 8277762: Allow configuration of HOTSPOT_BUILD_USER
On Wed, 24 Nov 2021 17:36:03 GMT, Erik Joelsson wrote: >> To better allow "reproducible builds", a new configure parameter is added to >> set the USERNAME build variable, rather than always using the current user: >> --with-build-user= >> HOTSPOT_BUILD_USER is then set to a reproducible USERNAME if required. >> >> Signed-off-by: Andrew Leonard > > make/autoconf/basic.m4 line 99: > >> 97: >> 98: # Setup username (for use in adhoc version strings etc) >> 99: AC_ARG_WITH([build-user], [AS_HELP_STRING([--with-build-user], > > This variable is currently initiated in a rather weird location given what > it's used for. As it used to be a one liner, it didn't seem worth the effort > to move it to a more suitable location, but when expanding it with a --with > flag I think we should also move this to jdk-version.m4, which is the only > place where it's currently used. I think it makes sense to treat this as part > of the version variables as that's what it's used for. it's also used in make/hotspot/lib/CompileJvm.gmk, but agree moving to jdk-versions.m4 makes sense - PR: https://git.openjdk.java.net/jdk/pull/6542
RFR: 8277762: Allow configuration of HOTSPOT_BUILD_USER
To better allow "reproducible builds", a new configure parameter is added to set the USERNAME build variable, rather than always using the current user: --with-build-user= HOTSPOT_BUILD_USER is then set to a reproducible USERNAME if required. Signed-off-by: Andrew Leonard - Commit messages: - 8277762: Allow configuration of HOTSPOT_BUILD_USER - 8277762: Allow configuration of HOTSPOT_BUILD_USER - 8277762: Allow configuration of HOTSPOT_BUILD_USER Changes: https://git.openjdk.java.net/jdk/pull/6542/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk=6542=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8277762 Stats: 9 lines in 1 file changed: 6 ins; 0 del; 3 mod Patch: https://git.openjdk.java.net/jdk/pull/6542.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/6542/head:pull/6542 PR: https://git.openjdk.java.net/jdk/pull/6542
Integrated: 8276743: Make openjdk build Zip Archive generation "reproducible"
On Tue, 9 Nov 2021 12:59:17 GMT, Andrew Leonard wrote: > This PR adds a new openjdk build tool MakeZipReproducible, which if > ENABLE_REPRODUCIBLE_BUILD is enabled, generates a final "zip" file in a > deterministic way, ensuring ordering and timestamps are set as specified. > > Using this tool in ZipArchive.gmk will ensure src.zip is then created > deterministically. > > Signed-off-by: Andrew Leonard This pull request has now been integrated. Changeset: aeba6530 Author:Andrew Leonard URL: https://git.openjdk.java.net/jdk/commit/aeba65303479130d9bab74484accad5d7d116a40 Stats: 261 lines in 4 files changed: 254 ins; 0 del; 7 mod 8276743: Make openjdk build Zip Archive generation "reproducible" Reviewed-by: erikj, ihse - PR: https://git.openjdk.java.net/jdk/pull/6311
Re: RFR: 8276743: Make openjdk build Zip Archive generation "reproducible" [v5]
On Fri, 12 Nov 2021 14:34:50 GMT, Erik Joelsson wrote: >> Sorry, I'm mis-reading the lines here. It's of course for docs-zip. Then >> it's perfectly in order! :-) > > It's using JarArchive.gmk, which is a similar, but different and probably > also needs the same treatment. We should probably share more code between > them. I believe so, it's target docs-zip not jrtfs : https://github.com/openjdk/jdk/blob/51a5731d6dc4b6f6feac920a4b8b49c15fd6b34f/make/Docs.gmk#L712 - PR: https://git.openjdk.java.net/jdk/pull/6311
