Re: RFR: 8264449: Enable reproducible builds with SOURCE_DATE_EPOCH [v5]
On Sat, 18 Sep 2021 15:15:10 GMT, Kevin Rushforth wrote:
>> John Neffenger has updated the pull request with a new target base due to a
>> merge or a rebase. The pull request now contains seven commits:
>>
>> - Make build of SDK ZIP bundle reproducible
>> - Merge branch 'master' into allow-reproducible-builds
>> - Merge branch 'master' into allow-reproducible-builds
>> - Include WebKit shared library for Windows
>>
>>Enable reproducible builds of the native WebKit shared library for
>>Windows (jfxwebkit.dll) when SOURCE_DATE_EPOCH is defined.
>> - Include media shared libraries for Windows
>>
>>Enable reproducible builds of the native media shared libraries for
>>Windows when SOURCE_DATE_EPOCH is defined. The libraries are:
>>
>> fxplugins.dll
>> glib-lite.dll
>> gstreamer-lite.dll
>> jfxmedia.dll
>> - Enable reproducible builds with SOURCE_DATE_EPOCH
>> - 8238650: Allow to override buildDate with SOURCE_DATE_EPOCH
>
> Here are couple more observations, and then I'll need to put this on the back
> burner for a bit:
>
> 1. I did a CI build yesterday and again today on all three platforms and
> compared the sdk between the two builds for each platform. On all three
> platforms the results are the same: All files were identical except the
> native jfxwebkit library. So there is something in the WebKit build that is
> affected by an external input (perhaps the system date or similar).
> 2. On Mac, at least, there are several differences in the dylib files between
> a build on my local system and on our CI system. I was using the same devkit
> and boot JDK on both, and both were the same version of macOS (10.15.7).
> Likely some difference in the software installed on the two systems matters.
@kevinrushforth Thank you, Kevin, for your review comments back in September.
Your comment about the time zone was extremely helpful and indirectly helped
with [openjdk/jdk#6481][1] as well. I think I have addressed all of your
concerns now.
My more recent commits simply add the following line in preparation for
creating reproducible JMOD archives: `// args("--date", buildTimestamp)`. That
line needs to be commented out until JavaFX starts building with JDK 19.
I ran another round of 30 builds, 10 on each platform, with no surprises. The
only non-reproducible artifacts are the JMOD archives and the native WebKit
shared library, as before. See my GitHub repository [jgneff/jfxbuild][2] for
the build environment and shell scripts that I use to build and test JavaFX on
Linux, macOS, and Windows.
[1]: https://github.com/openjdk/jdk/pull/6481
[2]: https://github.com/jgneff/jfxbuild
-
PR: https://git.openjdk.java.net/jfx/pull/446
Re: RFR: 8264449: Enable reproducible builds with SOURCE_DATE_EPOCH [v7]
> This pull request allows for reproducible builds of JavaFX on Linux, macOS, > and Windows by defining the `SOURCE_DATE_EPOCH` environment variable. For > example, the following commands create a reproducible build: > > > $ export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) > $ bash gradlew sdk jmods javadoc > $ strip-nondeterminism -v -T $SOURCE_DATE_EPOCH build/jmods/*.jmod > > > The three commands: > > 1. set the build timestamp to the date of the latest source code change, > 2. build the JavaFX SDK libraries, JMOD archives, and API documentation, and > 3. recreate the JMOD files with stable file modification times and ordering. > > The third command won't be necessary once Gradle can build the JMOD archives > or the `jmod` tool itself has the required support. For more information on > the environment variable, see the [`SOURCE_DATE_EPOCH`][1] page. For more > information on the command to recreate the JMOD files, see the > [`strip-nondeterminism`][2] repository. I'd like to propose that we allow for > reproducible builds in JavaFX 17 and consider making them the default in > JavaFX 18. > > Fixes > > There are at least four sources of non-determinism in the JavaFX builds: > > 1. Build timestamp > > The class `com.sun.javafx.runtime.VersionInfo` in the JavaFX Base module > stores the time of the build. Furthermore, for builds that don't run on the > Hudson continuous integration tool, the class adds the build time to the > system property `javafx.runtime.version`. > > 2. Modification times > > The JAR, JMOD, and ZIP archives store the modification time of each file. > > 3. File ordering > > The JAR, JMOD, and ZIP archives store their files in the order returned > by the file system. The native shared libraries also store their object files > in the order returned by the file system. Most file systems, though, do not > guarantee the order of a directory's file listing. > > 4. Build path > > The class `com.sun.javafx.css.parser.Css2Bin` in the JavaFX Graphics > module stores the absolute path of its `.css` input file in the corresponding > `.bss` output file, which is then included in the JavaFX Controls module. > > This pull request modifies the Gradle and Groovy build files to fix the first > three sources of non-determinism. A later pull request can modify the Java > files to fix the fourth. > > [1]: https://reproducible-builds.org/docs/source-date-epoch/ > [2]: https://salsa.debian.org/reproducible-builds/strip-nondeterminism John Neffenger has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 14 commits: - Comment out 'jmod --date' until building on JDK 19 Support for the 'jmod --date' option was added to JDK 19 starting with the 19+2 early-access build, and it was backported to JDK 17 starting with release 17.0.3. It is not available in JDK 18. - Merge 'master' into allow-reproducible-builds - Make minimal changes for new '--date' option - Add new '--date' option to JMOD task - Add '--source-date' option to JMOD task - Get build timestamp in UTC and set ZIP timestamps Create the build timestamp as a zoned date and time in UTC instead of a local date and time. If SOURCE_DATE_EPOCH is defined, set the last modification time of ZIP and JAR entries to the local date and time in UTC of the instant defined by SOURCE_DATE_EPOCH. Add a comment as a reminder to make JMOD files deterministic when the following enhancements are complete: * Enable deterministic file content ordering for Jar and Jmod https://bugs.openjdk.java.net/browse/JDK-8276764 https://github.com/openjdk/jdk/pull/6395 * Enable jar and jmod to produce deterministic timestamped content https://bugs.openjdk.java.net/browse/JDK-8276766 https://github.com/openjdk/jdk/pull/6481 - Merge branch 'master' into allow-reproducible-builds - Make build of SDK ZIP bundle reproducible - Merge branch 'master' into allow-reproducible-builds - Merge branch 'master' into allow-reproducible-builds - ... and 4 more: https://git.openjdk.java.net/jfx/compare/d69a498c...3fd4449b - Changes: https://git.openjdk.java.net/jfx/pull/446/files Webrev: https://webrevs.openjdk.java.net/?repo=jfx=446=06 Stats: 147 lines in 7 files changed: 117 ins; 13 del; 17 mod Patch: https://git.openjdk.java.net/jfx/pull/446.diff Fetch: git fetch https://git.openjdk.java.net/jfx pull/446/head:pull/446 PR: https://git.openjdk.java.net/jfx/pull/446
Re: RFR: 8264449: Enable reproducible builds with SOURCE_DATE_EPOCH [v6]
On Mon, 22 Nov 2021 06:43:30 GMT, John Neffenger wrote: >> This pull request allows for reproducible builds of JavaFX on Linux, macOS, >> and Windows by defining the `SOURCE_DATE_EPOCH` environment variable. For >> example, the following commands create a reproducible build: >> >> >> $ export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) >> $ bash gradlew sdk jmods javadoc >> $ strip-nondeterminism -v -T $SOURCE_DATE_EPOCH build/jmods/*.jmod >> >> >> The three commands: >> >> 1. set the build timestamp to the date of the latest source code change, >> 2. build the JavaFX SDK libraries, JMOD archives, and API documentation, and >> 3. recreate the JMOD files with stable file modification times and ordering. >> >> The third command won't be necessary once Gradle can build the JMOD archives >> or the `jmod` tool itself has the required support. For more information on >> the environment variable, see the [`SOURCE_DATE_EPOCH`][1] page. For more >> information on the command to recreate the JMOD files, see the >> [`strip-nondeterminism`][2] repository. I'd like to propose that we allow >> for reproducible builds in JavaFX 17 and consider making them the default in >> JavaFX 18. >> >> Fixes >> >> There are at least four sources of non-determinism in the JavaFX builds: >> >> 1. Build timestamp >> >> The class `com.sun.javafx.runtime.VersionInfo` in the JavaFX Base module >> stores the time of the build. Furthermore, for builds that don't run on the >> Hudson continuous integration tool, the class adds the build time to the >> system property `javafx.runtime.version`. >> >> 2. Modification times >> >> The JAR, JMOD, and ZIP archives store the modification time of each file. >> >> 3. File ordering >> >> The JAR, JMOD, and ZIP archives store their files in the order returned >> by the file system. The native shared libraries also store their object >> files in the order returned by the file system. Most file systems, though, >> do not guarantee the order of a directory's file listing. >> >> 4. Build path >> >> The class `com.sun.javafx.css.parser.Css2Bin` in the JavaFX Graphics >> module stores the absolute path of its `.css` input file in the >> corresponding `.bss` output file, which is then included in the JavaFX >> Controls module. >> >> This pull request modifies the Gradle and Groovy build files to fix the >> first three sources of non-determinism. A later pull request can modify the >> Java files to fix the fourth. >> >> [1]: https://reproducible-builds.org/docs/source-date-epoch/ >> [2]: https://salsa.debian.org/reproducible-builds/strip-nondeterminism > > John Neffenger has updated the pull request with a new target base due to a > merge or a rebase. The pull request now contains ten commits: > > - Get build timestamp in UTC and set ZIP timestamps > >Create the build timestamp as a zoned date and time in UTC instead >of a local date and time. If SOURCE_DATE_EPOCH is defined, set the >last modification time of ZIP and JAR entries to the local date and >time in UTC of the instant defined by SOURCE_DATE_EPOCH. > >Add a comment as a reminder to make JMOD files deterministic when >the following enhancements are complete: > >* Enable deterministic file content ordering for Jar and Jmod > https://bugs.openjdk.java.net/browse/JDK-8276764 > https://github.com/openjdk/jdk/pull/6395 > >* Enable jar and jmod to produce deterministic timestamped content > https://bugs.openjdk.java.net/browse/JDK-8276766 > https://github.com/openjdk/jdk/pull/6481 > - Merge branch 'master' into allow-reproducible-builds > - Make build of SDK ZIP bundle reproducible > - Merge branch 'master' into allow-reproducible-builds > - Merge branch 'master' into allow-reproducible-builds > - Include WebKit shared library for Windows > >Enable reproducible builds of the native WebKit shared library for >Windows (jfxwebkit.dll) when SOURCE_DATE_EPOCH is defined. > - Include media shared libraries for Windows > >Enable reproducible builds of the native media shared libraries for >Windows when SOURCE_DATE_EPOCH is defined. The libraries are: > > fxplugins.dll > glib-lite.dll > gstreamer-lite.dll > jfxmedia.dll > - Enable reproducible builds with SOURCE_DATE_EPOCH > - 8238650: Allow to override buildDate with SOURCE_DATE_EPOCH Good news: starting today, the JDK can create reproducible builds of the JDK!
Re: JavaFX Launch Failure on Ubuntu from JNI
On 1/19/22 2:12 PM, Steve Hannah wrote: I have been resisting using modules for a long time because it just makes things more complicated, ... It also makes some things easier, though, and certainly smaller. It's easier to use an old-school Makefile with modules, and using 'jlink' can get a simple Hello World JavaFX application and Java runtime down to just 31 megabytes. Here's my minimal, no-magic example: https://github.com/jgneff/hello-javafx with a simple Makefile: https://github.com/jgneff/hello-javafx/blob/main/Makefile and a Maven POM for use online with Maven Central or offline with a local Debian- or Ubuntu-built Maven repository: https://github.com/jgneff/hello-javafx/blob/main/pom.xml John
Re: RFR: 8264449: Enable reproducible builds with SOURCE_DATE_EPOCH [v6]
On Mon, 22 Nov 2021 06:43:30 GMT, John Neffenger wrote: >> This pull request allows for reproducible builds of JavaFX on Linux, macOS, >> and Windows by defining the `SOURCE_DATE_EPOCH` environment variable. For >> example, the following commands create a reproducible build: >> >> >> $ export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) >> $ bash gradlew sdk jmods javadoc >> $ strip-nondeterminism -v -T $SOURCE_DATE_EPOCH build/jmods/*.jmod >> >> >> The three commands: >> >> 1. set the build timestamp to the date of the latest source code change, >> 2. build the JavaFX SDK libraries, JMOD archives, and API documentation, and >> 3. recreate the JMOD files with stable file modification times and ordering. >> >> The third command won't be necessary once Gradle can build the JMOD archives >> or the `jmod` tool itself has the required support. For more information on >> the environment variable, see the [`SOURCE_DATE_EPOCH`][1] page. For more >> information on the command to recreate the JMOD files, see the >> [`strip-nondeterminism`][2] repository. I'd like to propose that we allow >> for reproducible builds in JavaFX 17 and consider making them the default in >> JavaFX 18. >> >> Fixes >> >> There are at least four sources of non-determinism in the JavaFX builds: >> >> 1. Build timestamp >> >> The class `com.sun.javafx.runtime.VersionInfo` in the JavaFX Base module >> stores the time of the build. Furthermore, for builds that don't run on the >> Hudson continuous integration tool, the class adds the build time to the >> system property `javafx.runtime.version`. >> >> 2. Modification times >> >> The JAR, JMOD, and ZIP archives store the modification time of each file. >> >> 3. File ordering >> >> The JAR, JMOD, and ZIP archives store their files in the order returned >> by the file system. The native shared libraries also store their object >> files in the order returned by the file system. Most file systems, though, >> do not guarantee the order of a directory's file listing. >> >> 4. Build path >> >> The class `com.sun.javafx.css.parser.Css2Bin` in the JavaFX Graphics >> module stores the absolute path of its `.css` input file in the >> corresponding `.bss` output file, which is then included in the JavaFX >> Controls module. >> >> This pull request modifies the Gradle and Groovy build files to fix the >> first three sources of non-determinism. A later pull request can modify the >> Java files to fix the fourth. >> >> [1]: https://reproducible-builds.org/docs/source-date-epoch/ >> [2]: https://salsa.debian.org/reproducible-builds/strip-nondeterminism > > John Neffenger has updated the pull request with a new target base due to a > merge or a rebase. The pull request now contains ten commits: > > - Get build timestamp in UTC and set ZIP timestamps > >Create the build timestamp as a zoned date and time in UTC instead >of a local date and time. If SOURCE_DATE_EPOCH is defined, set the >last modification time of ZIP and JAR entries to the local date and >time in UTC of the instant defined by SOURCE_DATE_EPOCH. > >Add a comment as a reminder to make JMOD files deterministic when >the following enhancements are complete: > >* Enable deterministic file content ordering for Jar and Jmod > https://bugs.openjdk.java.net/browse/JDK-8276764 > https://github.com/openjdk/jdk/pull/6395 > >* Enable jar and jmod to produce deterministic timestamped content > https://bugs.openjdk.java.net/browse/JDK-8276766 > https://github.com/openjdk/jdk/pull/6481 > - Merge branch 'master' into allow-reproducible-builds > - Make build of SDK ZIP bundle reproducible > - Merge branch 'master' into allow-reproducible-builds > - Merge branch 'master' into allow-reproducible-builds > - Include WebKit shared library for Windows > >Enable reproducible builds of the native WebKit shared library for >Windows (jfxwebkit.dll) when SOURCE_DATE_EPOCH is defined. > - Include media shared libraries for Windows > >Enable reproducible builds of the native media shared libraries for >Windows when SOURCE_DATE_EPOCH is defined. The libraries are: > > fxplugins.dll > glib-lite.dll > gstreamer-lite.dll > jfxmedia.dll > - Enable reproducible builds with SOURCE_DATE_EPOCH > - 8238650: Allow to override buildDate with SOURCE_DATE_EPOCH The results of 33 builds are shown in the table below: | Builds | Linux | m
Re: RFR: 8264449: Enable reproducible builds with SOURCE_DATE_EPOCH [v5]
On Sat, 18 Sep 2021 15:15:10 GMT, Kevin Rushforth wrote: > I did a CI build yesterday and again today on all three platforms and > compared the sdk between the two builds for each platform. On all three > platforms the results are the same: All files were identical except the > native jfxwebkit library. So there is something in the WebKit build that is > affected by an external input (perhaps the system date or similar). I can recreate this difference, but only on macOS. **Lesson learned:** When testing reproducible builds on a permanent system, never use the Gradle Daemon! I always get a different `libjfxwebkit.dylib` file when I run the build with `bash gradlew --no-daemon`. Shown below are the versions of the compilers I'm using: john@linux:~$ gcc --version gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0 john@macos:~$ gcc --version Apple clang version 13.0.0 (clang-1300.0.29.3) john@win10:~$ gcc --version gcc (GCC) 11.2.0 When I dump each of the shared libraries with the following script, I get a text file with 10,413,716 lines, yet there are only 108 lines different between the two output files. #!/bin/bash # Dumps macOS shared libraries # Diffoscope fails due to missing '=' after '--arch-name' option: # llvm-objdump: error: unknown argument '--arch-name' llvm-objdump --arch-name=x86_64 --section="__TEXT,__text" --macho \ --demangle --no-leading-addr --no-show-raw-insn "$@" The differences are shown below: diff --git a/libjfxwebkit.5.txt b/libjfxwebkit.6.txt index 9d6932a..63f2b70 100644 --- a/libjfxwebkit.5.txt +++ b/libjfxwebkit.6.txt @@ -1,4 +1,4 @@ -libjfxwebkit.5.dylib: +libjfxwebkit.6.dylib: Contents of (__TEXT,__text) section __ZN6WebKit15StorageAreaImplD2Ev: pushq %rbp @@ -881412,30 +881412,22 @@ __ZN7WebCore30isCSSPropertyEnabledBySettingsENS_13CSSPropertyIDEPKNS_8SettingsE: movq%rsp, %rbp movb$1, %al testq %rsi, %rsi - je 0x319ee0 + je 0x319ee7 leal-258(%rdi), %ecx cmpw$37, %cx - ja 0x319eb7 + ja 0x319eb2 movzwl %cx, %ecx - leaq79(%rip), %rdx + leaq75(%rip), %rdx movslq (%rdx,%rcx,4), %rcx addq%rdx, %rcx jmpq*%rcx movb466(%rsi), %al - andb$4, %al - shrb$2, %al - popq%rbp - retq + jmp 0x319ee2 + cmpw$43, %di + je 0x319edc movzwl %di, %ecx cmpl$366, %ecx - je 0x319ed5 - cmpw$43, %di - jne 0x319ee0 - movb451(%rsi), %al - andb$4, %al - shrb$2, %al - popq%rbp - retq + jne 0x319ee7 movb454(%rsi), %al andb$32, %al shrb$5, %al @@ -881446,46 +881438,52 @@ __ZN7WebCore30isCSSPropertyEnabledBySettingsENS_13CSSPropertyIDEPKNS_8SettingsE: shrb%al popq%rbp retq + movb451(%rsi), %al + andb$4, %al + shrb$2, %al + popq%rbp + retq + nopl(%rax) + .long 4294967230@ KIND_JUMP_TABLE32 + .long 4294967230@ KIND_JUMP_TABLE32 + .long 4294967230@ KIND_JUMP_TABLE32 + .long 4294967291@ KIND_JUMP_TABLE32 + .long 4294967291@ KIND_JUMP_TABLE32 + .long 4294967291@ KIND_JUMP_TABLE32 + .long 4294967291@ KIND_JUMP_TABLE32 + .long 4294967291@ KIND_JUMP_TABLE32 + .long 4294967291@ KIND_JUMP_TABLE32 + .long 4294967291@ KIND_JUMP_TABLE32 + .long 4294967291@ KIND_JUMP_TABLE32 + .long 4294967291@ KIND_JUMP_TABLE32 + .long 4294967291@ KIND_JUMP_TABLE32 + .long 4294967291@ KIND_JUMP_TABLE32 + .long 4294967291@ KIND_JUMP_TABLE32 + .long 4294967291@ KIND_JUMP_TABLE32 + .long 4294967291@ KIND_JUMP_TABLE32 + .long 4294967291@ KIND_JUMP_TABLE32 + .long 4294967291@ KIND_JUMP_TABLE32 + .long 4294967291@ KIND_JUMP_TABLE32 + .long 4294967291@ KIND_JUMP_TABLE32 + .long 4294967291@ KIND_JUMP_TABLE32 + .long 4294967291@ KIND_JUMP_TABLE32 + .long 4294967291@ KIND_JUMP_TABLE32 + .long 4294967291@ KIND_JUMP_TABLE32 + .long 4294967291@ KIND_JUMP_TABLE32 + .long 4294967291@ KIND_JUMP_TABLE32 + .long 4294967291@ KIND_JUMP_TABLE32 + .long 4294967291@ KIND_JUMP_TABLE32 + .long 4294967291@ KIND_JUMP_TABLE32 + .long 4294967291@ KIND_JUMP_TABLE32 + .long 4294967291@ KIND_JUMP_TABLE32 + .long 4294967255@ KIND_JUMP_TABLE32 + .long 4294967291@ KIND_JUMP_TABLE32 + .long 4294967291@ KIND_JUMP_TABLE32 + .long 4294967291@ KIND_JUMP_TABLE32 + .long 4294967255
Re: RFR: 8264449: Enable reproducible builds with SOURCE_DATE_EPOCH [v5]
On Fri, 17 Sep 2021 22:10:41 GMT, Kevin Rushforth wrote:
>> John Neffenger has updated the pull request with a new target base due to a
>> merge or a rebase. The pull request now contains seven commits:
>>
>> - Make build of SDK ZIP bundle reproducible
>> - Merge branch 'master' into allow-reproducible-builds
>> - Merge branch 'master' into allow-reproducible-builds
>> - Include WebKit shared library for Windows
>>
>>Enable reproducible builds of the native WebKit shared library for
>>Windows (jfxwebkit.dll) when SOURCE_DATE_EPOCH is defined.
>> - Include media shared libraries for Windows
>>
>>Enable reproducible builds of the native media shared libraries for
>>Windows when SOURCE_DATE_EPOCH is defined. The libraries are:
>>
>> fxplugins.dll
>> glib-lite.dll
>> gstreamer-lite.dll
>> jfxmedia.dll
>> - Enable reproducible builds with SOURCE_DATE_EPOCH
>> - 8238650: Allow to override buildDate with SOURCE_DATE_EPOCH
>
> build.gradle line 559:
>
>> 557: buildDate = new java.util.Date(ms)
>> 558: }
>> 559: def buildTimestamp = new
>> java.text.SimpleDateFormat("-MM-dd-HHmmss").format(buildDate)
>
> I think it would be useful to format `buildDate` using UTC as the time zone
> when `SOURCE_DATE_EPOCH` is set.
Indeed. The build now uses the new `java.time.Instant` class to get the instant
on the time-line, whether or not `SOURCE_DATE_EPOCH` is set, and creates the
timestamp in UTC using the ISO 8601 date and time format.
> build.gradle line 3518:
>
>> 3516: def lFlags =
>> webkitProperties.linkFlags?.join(' ') ?: ''
>> 3517: cmakeArgs = "$cmakeArgs
>> -DCMAKE_C_FLAGS='${cFlags}' -DCMAKE_CXX_FLAGS='${cFlags}'"
>> 3518: cmakeArgs = "$cmakeArgs
>> -DCMAKE_SHARED_LINKER_FLAGS='${lFlags}'"
>
> I presume you've tested this both with and without `SOURCE_DATE_EPOCH`?
Well, now I have! Thanks.
> build.gradle line 3914:
>
>> 3912: tasks.withType(AbstractArchiveTask) {
>> 3913: if (sourceDateEpoch != null) {
>> 3914: preserveFileTimestamps = false
>
> This is a problem given how gradle generates a zip archive when this is set.
> How hard would it be to force all time stamps to be `SOURCE_DATE_EPOCH`?
It was harder than I had hoped, but I think I found a good solution. The only
changes in the ZIP and JAR archives are shown in the `diff` extract below (from
`zipinfo -v`):
29c29
< minimum software version required to extract: 1.0
---
> minimum software version required to extract: 2.0
33,34c33,34
< extended local header: no
< file last modified on (DOS date/time): 1980 Feb 1 00:00:00
---
> extended local header: yes
> file last modified on (DOS date/time): 2021 Nov 16 17:55:42
44c44
< MS-DOS file attributes (10 hex):dir
---
> MS-DOS file attributes (00 hex):none
where:
* The minimum software version required to extract is now set to the correct
value of 2.0, which is the minimum for file entries compressed with DEFLATE.
* The extended local header is defined but with length zero.
* The timestamp is set to the local date and time in UTC of the instant of the
build.
* The MS-DOS file attribute for a directory is not set. It does not appear to
be useful, as an entry in a ZIP file is a directory if and only if its name
ends with a slash ("/").
These changes are due to Gradle using the Apache Ant
`org.apache.tools.zip.ZipEntry` class, while the build now uses the
`java.util.zip.ZipEntry` class in OpenJDK to create the archives.
-
PR: https://git.openjdk.java.net/jfx/pull/446
Re: RFR: 8264449: Enable reproducible builds with SOURCE_DATE_EPOCH [v6]
> This pull request allows for reproducible builds of JavaFX on Linux, macOS, > and Windows by defining the `SOURCE_DATE_EPOCH` environment variable. For > example, the following commands create a reproducible build: > > > $ export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) > $ bash gradlew sdk jmods javadoc > $ strip-nondeterminism -v -T $SOURCE_DATE_EPOCH build/jmods/*.jmod > > > The three commands: > > 1. set the build timestamp to the date of the latest source code change, > 2. build the JavaFX SDK libraries, JMOD archives, and API documentation, and > 3. recreate the JMOD files with stable file modification times and ordering. > > The third command won't be necessary once Gradle can build the JMOD archives > or the `jmod` tool itself has the required support. For more information on > the environment variable, see the [`SOURCE_DATE_EPOCH`][1] page. For more > information on the command to recreate the JMOD files, see the > [`strip-nondeterminism`][2] repository. I'd like to propose that we allow for > reproducible builds in JavaFX 17 and consider making them the default in > JavaFX 18. > > Fixes > > There are at least four sources of non-determinism in the JavaFX builds: > > 1. Build timestamp > > The class `com.sun.javafx.runtime.VersionInfo` in the JavaFX Base module > stores the time of the build. Furthermore, for builds that don't run on the > Hudson continuous integration tool, the class adds the build time to the > system property `javafx.runtime.version`. > > 2. Modification times > > The JAR, JMOD, and ZIP archives store the modification time of each file. > > 3. File ordering > > The JAR, JMOD, and ZIP archives store their files in the order returned > by the file system. The native shared libraries also store their object files > in the order returned by the file system. Most file systems, though, do not > guarantee the order of a directory's file listing. > > 4. Build path > > The class `com.sun.javafx.css.parser.Css2Bin` in the JavaFX Graphics > module stores the absolute path of its `.css` input file in the corresponding > `.bss` output file, which is then included in the JavaFX Controls module. > > This pull request modifies the Gradle and Groovy build files to fix the first > three sources of non-determinism. A later pull request can modify the Java > files to fix the fourth. > > [1]: https://reproducible-builds.org/docs/source-date-epoch/ > [2]: https://salsa.debian.org/reproducible-builds/strip-nondeterminism John Neffenger has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains ten commits: - Get build timestamp in UTC and set ZIP timestamps Create the build timestamp as a zoned date and time in UTC instead of a local date and time. If SOURCE_DATE_EPOCH is defined, set the last modification time of ZIP and JAR entries to the local date and time in UTC of the instant defined by SOURCE_DATE_EPOCH. Add a comment as a reminder to make JMOD files deterministic when the following enhancements are complete: * Enable deterministic file content ordering for Jar and Jmod https://bugs.openjdk.java.net/browse/JDK-8276764 https://github.com/openjdk/jdk/pull/6395 * Enable jar and jmod to produce deterministic timestamped content https://bugs.openjdk.java.net/browse/JDK-8276766 https://github.com/openjdk/jdk/pull/6481 - Merge branch 'master' into allow-reproducible-builds - Make build of SDK ZIP bundle reproducible - Merge branch 'master' into allow-reproducible-builds - Merge branch 'master' into allow-reproducible-builds - Include WebKit shared library for Windows Enable reproducible builds of the native WebKit shared library for Windows (jfxwebkit.dll) when SOURCE_DATE_EPOCH is defined. - Include media shared libraries for Windows Enable reproducible builds of the native media shared libraries for Windows when SOURCE_DATE_EPOCH is defined. The libraries are: fxplugins.dll glib-lite.dll gstreamer-lite.dll jfxmedia.dll - Enable reproducible builds with SOURCE_DATE_EPOCH - 8238650: Allow to override buildDate with SOURCE_DATE_EPOCH - Changes: https://git.openjdk.java.net/jfx/pull/446/files Webrev: https://webrevs.openjdk.java.net/?repo=jfx=446=05 Stats: 149 lines in 7 files changed: 119 ins; 13 del; 17 mod Patch: https://git.openjdk.java.net/jfx/pull/446.diff Fetch: git fetch https://git.openjdk.java.net/jfx pull/446/head:pull/446 PR: https://git.openjdk.java.net/jfx/pull/446
Re: CFV: New OpenJFX Committer: Thiago Sayao
Vote: YES Just in time, John On 9/13/21 7:37 AM, Kevin Rushforth wrote: I hereby nominate Thiago Sayao [1] to OpenJFX Committer. Thiago is an OpenJFX community member, who has contributed 15 commits [2] to OpenJFX. Votes are due by September 27, 2021 at 15:00 UTC. Only current OpenJFX Committers [3] are eligible to vote on this nomination. Votes must be cast in the open by replying to this mailing list. For Lazy Consensus voting instructions, see [4]. Nomination to a project Committer is described in [6]. Thanks. -- Kevin [1] https://openjdk.java.net/census#tsayao [2] https://github.com/openjdk/jfx/search?q=author-name%3A%22Thiago+Sayao%22=author-date=commits [3] https://openjdk.java.net/census#openjfx [4] https://openjdk.java.net/bylaws#lazy-consensus [6] https://openjdk.java.net/projects#project-committer
Re: RFR: 8264449: Enable reproducible builds with SOURCE_DATE_EPOCH [v5]
On Fri, 17 Sep 2021 23:24:45 GMT, Kevin Rushforth wrote: > * The timestamps for all files in the zip archives are set to a hard-coded > "1980-02-01", rather than the date and time specified by `SOURCE_DATE_EPOCH`. That date was chosen by the Gradle project five years ago in the commit ["Add sortedFileOrder and preserveFileTimestamps for archive tasks,"][1] and not changed five months later in the follow-up commit ["Move constants to classes using them."][2] This year, they confirmed the choice by closing the issue ["allow to configure timestamp used for preserveFileTimestamps."][3]. Also see [my comment][4] on the closed issue for more background information. At this point, it might be safest to go with the five-year-old Gradle default when compared to the alternatives: 1. Invoke the `find` and `touch` command-line tools from Gradle as described on the Reproducible Builds website under the "File modification times" section of the [Archive metadata][5] page. 2. Figure out how to do the equivalent thing in Gradle, perhaps using the `eachFile(closure)` or `eachFile(action)` method of the [`Zip` task][6]. 3. Add a post-processing step outside of the Gradle build that runs [`strip-nondeterminism`][7] to normalize the JAR and ZIP archives and then repackages the SDK, JMOD, and Javadoc bundles. Gradle is our build system. In for a penny, in for a pound? [1]: https://github.com/gradle/gradle/commit/0209b5dd [2]: https://github.com/gradle/gradle/commit/8b8daf34 [3]: https://github.com/gradle/gradle/issues/14819 [4]: https://github.com/gradle/gradle/issues/14819#issuecomment-922181921 [5]: https://reproducible-builds.org/docs/archives/ [6]: https://docs.gradle.org/current/dsl/org.gradle.api.tasks.bundling.Zip.html [7]: https://salsa.debian.org/reproducible-builds/strip-nondeterminism - PR: https://git.openjdk.java.net/jfx/pull/446
Re: RFR: 8264449: Enable reproducible builds with SOURCE_DATE_EPOCH [v5]
On Mon, 14 Jun 2021 20:53:50 GMT, John Neffenger wrote: >> This pull request allows for reproducible builds of JavaFX on Linux, macOS, >> and Windows by defining the `SOURCE_DATE_EPOCH` environment variable. For >> example, the following commands create a reproducible build: >> >> >> $ export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) >> $ bash gradlew sdk jmods javadoc >> $ strip-nondeterminism -v -T $SOURCE_DATE_EPOCH build/jmods/*.jmod >> >> >> The three commands: >> >> 1. set the build timestamp to the date of the latest source code change, >> 2. build the JavaFX SDK libraries, JMOD archives, and API documentation, and >> 3. recreate the JMOD files with stable file modification times and ordering. >> >> The third command won't be necessary once Gradle can build the JMOD archives >> or the `jmod` tool itself has the required support. For more information on >> the environment variable, see the [`SOURCE_DATE_EPOCH`][1] page. For more >> information on the command to recreate the JMOD files, see the >> [`strip-nondeterminism`][2] repository. I'd like to propose that we allow >> for reproducible builds in JavaFX 17 and consider making them the default in >> JavaFX 18. >> >> Fixes >> >> There are at least four sources of non-determinism in the JavaFX builds: >> >> 1. Build timestamp >> >> The class `com.sun.javafx.runtime.VersionInfo` in the JavaFX Base module >> stores the time of the build. Furthermore, for builds that don't run on the >> Hudson continuous integration tool, the class adds the build time to the >> system property `javafx.runtime.version`. >> >> 2. Modification times >> >> The JAR, JMOD, and ZIP archives store the modification time of each file. >> >> 3. File ordering >> >> The JAR, JMOD, and ZIP archives store their files in the order returned >> by the file system. The native shared libraries also store their object >> files in the order returned by the file system. Most file systems, though, >> do not guarantee the order of a directory's file listing. >> >> 4. Build path >> >> The class `com.sun.javafx.css.parser.Css2Bin` in the JavaFX Graphics >> module stores the absolute path of its `.css` input file in the >> corresponding `.bss` output file, which is then included in the JavaFX >> Controls module. >> >> This pull request modifies the Gradle and Groovy build files to fix the >> first three sources of non-determinism. A later pull request can modify the >> Java files to fix the fourth. >> >> [1]: https://reproducible-builds.org/docs/source-date-epoch/ >> [2]: https://salsa.debian.org/reproducible-builds/strip-nondeterminism > > John Neffenger has updated the pull request with a new target base due to a > merge or a rebase. The pull request now contains seven commits: > > - Make build of SDK ZIP bundle reproducible > - Merge branch 'master' into allow-reproducible-builds > - Merge branch 'master' into allow-reproducible-builds > - Include WebKit shared library for Windows > >Enable reproducible builds of the native WebKit shared library for >Windows (jfxwebkit.dll) when SOURCE_DATE_EPOCH is defined. > - Include media shared libraries for Windows > >Enable reproducible builds of the native media shared libraries for >Windows when SOURCE_DATE_EPOCH is defined. The libraries are: > > fxplugins.dll > glib-lite.dll > gstreamer-lite.dll > jfxmedia.dll > - Enable reproducible builds with SOURCE_DATE_EPOCH > - 8238650: Allow to override buildDate with SOURCE_DATE_EPOCH > 1. On all three platforms the results are the same: All files were identical > except the native jfxwebkit library. The [diffoscope][1] tool can show you the difference between the two files. You don't even need to install it. If the files aren't too big, you can upload them to the [online version][2]. > 2. On Mac, at least, there are several differences in the dylib files between > a build on my local system and on our CI system. I would like to enable reproducible builds on ephemeral systems that create a clean and isolated build environment, like those created by GitHub Actions or the Launchpad build farm. To compare across developer systems, we would need a full system software bill of materials (SBOM) beyond what's listed in the Gradle dependency verification file. The SBOM is the next step in allowing for reproducible builds in any environment, but it's not a part of this pull request. [1]: https://diffoscope.org/ [2]: https://try.diffoscope.org/ - PR: https://git.openjdk.java.net/jfx/pull/446
Re: RFR: 8264449: Enable reproducible builds with SOURCE_DATE_EPOCH [v5]
On Mon, 14 Jun 2021 20:53:50 GMT, John Neffenger wrote: >> This pull request allows for reproducible builds of JavaFX on Linux, macOS, >> and Windows by defining the `SOURCE_DATE_EPOCH` environment variable. For >> example, the following commands create a reproducible build: >> >> >> $ export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) >> $ bash gradlew sdk jmods javadoc >> $ strip-nondeterminism -v -T $SOURCE_DATE_EPOCH build/jmods/*.jmod >> >> >> The three commands: >> >> 1. set the build timestamp to the date of the latest source code change, >> 2. build the JavaFX SDK libraries, JMOD archives, and API documentation, and >> 3. recreate the JMOD files with stable file modification times and ordering. >> >> The third command won't be necessary once Gradle can build the JMOD archives >> or the `jmod` tool itself has the required support. For more information on >> the environment variable, see the [`SOURCE_DATE_EPOCH`][1] page. For more >> information on the command to recreate the JMOD files, see the >> [`strip-nondeterminism`][2] repository. I'd like to propose that we allow >> for reproducible builds in JavaFX 17 and consider making them the default in >> JavaFX 18. >> >> Fixes >> >> There are at least four sources of non-determinism in the JavaFX builds: >> >> 1. Build timestamp >> >> The class `com.sun.javafx.runtime.VersionInfo` in the JavaFX Base module >> stores the time of the build. Furthermore, for builds that don't run on the >> Hudson continuous integration tool, the class adds the build time to the >> system property `javafx.runtime.version`. >> >> 2. Modification times >> >> The JAR, JMOD, and ZIP archives store the modification time of each file. >> >> 3. File ordering >> >> The JAR, JMOD, and ZIP archives store their files in the order returned >> by the file system. The native shared libraries also store their object >> files in the order returned by the file system. Most file systems, though, >> do not guarantee the order of a directory's file listing. >> >> 4. Build path >> >> The class `com.sun.javafx.css.parser.Css2Bin` in the JavaFX Graphics >> module stores the absolute path of its `.css` input file in the >> corresponding `.bss` output file, which is then included in the JavaFX >> Controls module. >> >> This pull request modifies the Gradle and Groovy build files to fix the >> first three sources of non-determinism. A later pull request can modify the >> Java files to fix the fourth. >> >> [1]: https://reproducible-builds.org/docs/source-date-epoch/ >> [2]: https://salsa.debian.org/reproducible-builds/strip-nondeterminism > > John Neffenger has updated the pull request with a new target base due to a > merge or a rebase. The pull request now contains seven commits: > > - Make build of SDK ZIP bundle reproducible > - Merge branch 'master' into allow-reproducible-builds > - Merge branch 'master' into allow-reproducible-builds > - Include WebKit shared library for Windows > >Enable reproducible builds of the native WebKit shared library for >Windows (jfxwebkit.dll) when SOURCE_DATE_EPOCH is defined. > - Include media shared libraries for Windows > >Enable reproducible builds of the native media shared libraries for >Windows when SOURCE_DATE_EPOCH is defined. The libraries are: > > fxplugins.dll > glib-lite.dll > gstreamer-lite.dll > jfxmedia.dll > - Enable reproducible builds with SOURCE_DATE_EPOCH > - 8238650: Allow to override buildDate with SOURCE_DATE_EPOCH Thanks for the great comments, Kevin. I'm looking into the issues you raised. Would it be possible to coordinate the reviews of this pull request? My previous [set of tests in June][1] required 28 builds on three platforms and took me over four days to run and test. I would prefer to do that just one more time. So my plan is to address each of the review comments from @kevinrushforth, @johanvos, and @tiainen, but merge the master branch and do a final round of testing only after all three reviews are done and I have addressed each of the reviewers' comments. Would that work? Should I also wait for @arun-Joseph and @sashamatveev to re-review the parts they looked at before? [1]: https://github.com/openjdk/jfx/pull/446#issuecomment-861000786 - PR: https://git.openjdk.java.net/jfx/pull/446
Re: Enhancements for JavaFX 18
On 8/19/21 1:27 PM, Kevin Rushforth wrote: Those enhancements that have reached general consensus, are far enough along in the process that we have a good idea how much they will "cost", and have a willing contributor and sponsor (for contributors who are not Committers), are more likely to make JavaFX 18. I'd like to make a pitch for integrating the reproducible build support: 8264449: Enable reproducible builds with SOURCE_DATE_EPOCH https://github.com/openjdk/jfx/pull/446 It's not as glamorous as the user interface changes that have been suggested. I think the change is inevitable, though, so it would be great to have it integrated while I can still remember what I did. I've tested it in two separate rounds of tests on all three platforms, and it has been approved by two reviewers. The JDK is also doing this work, driven mostly by Magnus Ihse Bursie at Oracle in the OpenJDK Build Infrastructure Group. He calls it "a low-effort, background-style project to make the build of OpenJDK itself building reproducible." For the original motivation, see the following closed pull request that is now included in the one linked above: 8238650: Allow to override buildDate with SOURCE_DATE_EPOCH https://github.com/openjdk/jfx/pull/422 Thanks, John
Re: Moving src.zip out of the lib directory of the JavaFX SDK
On 6/23/21 6:50 AM, Kevin Rushforth wrote: Are there any IDE users who are currently having problems as a result of this? If not, I'll retarget this for a future release. I haven't seen problems with the current location of the 'src.zip' file in NetBeans. For Apache Ant projects, though, I do need to extract the archive into a different location and attach the individual source directories to their corresponding modules one at a time. For anyone interested, the following issue tracks the sub-tasks for each of the problems I encountered using the JavaFX Javadocs and Sources with NetBeans: NETBEANS-3296: Attaching JavaFX Javadoc and Sources https://issues.apache.org/jira/browse/NETBEANS-3296 For the Apache Ant work-around, see the section with the title "Add to JAR files." John
Re: RFR: 8268915: WebKit build fails with Xcode 12.5
On Thu, 17 Jun 2021 14:52:53 GMT, Kevin Rushforth wrote: > The JavaFX WebKit build fails with Xcode 12.5 + MacOS 11.3 SDK. This is > related to the added C++20 support where some of the system include files now > do `#include `. Because the macOS file system is case insensitive, > it matches a file named `VERSION` in the sqlite source directory. That file > is just a plain text file (not used by our build at all) so the fix is to > rename it to `VERSION.txt`. I've done a CI build on all three platforms. > @jgneff has verified that renaming this file fixes the build for XCode 12.5. I just tested this pull request on macOS using the Command Line Tools for Xcode 12.5.1 update released on June 20, 2021: $ pkgutil --pkg-info=com.apple.pkg.CLTools_Executables package-id: com.apple.pkg.CLTools_Executables version: 12.5.1.0.1.1623191612 volume: / location: / install-time: 1624422801 groups: com.apple.FindSystemFiles.pkg-group The build and unit tests were successful: BUILD SUCCESSFUL in 1h 5m 9s 214 actionable tasks: 214 executed - PR: https://git.openjdk.java.net/jfx/pull/535
Re: Command Line Tools for Xcode 12.4 → 12.5
On 6/16/21 10:21 AM, John Neffenger wrote: 4. Finally look up what the GitHub Actions are using (12.4). Oops, make that "Finally look up what the project uses (12.4)." For the record, our GitHub Actions use Xcode_11.3.1. [1] John [1]: https://github.com/openjdk/jfx/blob/master/.github/workflows/submit.yml#L202
Re: Command Line Tools for Xcode 12.4 → 12.5
On 6/16/21 9:56 AM, Philip Race wrote: 1) Don't let macOS upgrade my xcode tools automatically Lesson learned! But then there's that annoying red badge on your System Preferences forever. :-) Actually, it was worse, and went more like this (just in case any else gets led off-course by the error message below): 1. Naively download the latest CLTools 12.5. 2. See the following error message and think I need the full Xcode: xcode-select: error: tool 'xcodebuild' requires Xcode, but active developer directory '/Library/Developer/CommandLineTools' is a command line tools instance That is a red herring. It's just the build testing whether the full Xcode is installed by running the command 'xcodebuild -version -showsdks'. When that command fails, the build runs 'xcrun --show-sdk-path' and continues without problems. 3. Download and install the full Xcode 12.5, and it still fails! 4. Finally look up what the GitHub Actions are using (12.4). 5. Download and install CLTools 12.4 -- works! 6. Uninstall the 11-GB constantly-updating Xcode app. John
Command Line Tools for Xcode 12.4 → 12.5
Just a heads-up about using the latest Xcode 12.5 ... I use the Command Line Tools for Xcode 12.4 (at 431 MB) to build JavaFX on macOS as an alternative to the full Xcode package (at 11.86 GB). Thank you, Arunprasad Rajkumar! [1] Then Apple Software Update installed the latest Command Line Tools for Xcode 12.5, and my builds of JavaFX with WebKit started failing. You can find out what version you're using with the command: $ pkgutil --pkg-info=com.apple.pkg.CLTools_Executables package-id: com.apple.pkg.CLTools_Executables version: 12.5.0.0.1.1617976050 volume: / location: / install-time: 1623800794 groups: com.apple.FindSystemFiles.pkg-group You can download and revert to the 12.4 release below: More Downloads https://developer.apple.com/download/all/?q=12.4 Note that the OpenJFX project uses version 12.4 (build.properties): jfx.build.macosx.xcode.version=Xcode12.4+1.0 The errors using version 12.5 start with: .../modules/javafx.web/src/main/native/Source/ThirdParty/ sqlite/./version:1:1: error: expected unqualified-id /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/ usr/include/c++/v1/cstddef:49:9: error: no member named 'ptrdiff_t' in the global namespace and end with: PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT: Preprocessed source(s) and associated run script(s) are located at: clang: note: diagnostic msg: /var/folders/... clang: note: diagnostic msg: /var/folders/... clang: note: diagnostic msg: Crash backtrace is located in clang: note: diagnostic msg: /Users/john/Library/Logs/ DiagnosticReports/clang__.crash clang: note: diagnostic msg: (choose the .crash file that corresponds to your crash) clang: note: diagnostic msg: John [1]: https://github.com/openjdk/jfx/pull/13
Re: RFR: 8264449: Enable reproducible builds with SOURCE_DATE_EPOCH [v5]
On Mon, 14 Jun 2021 20:53:50 GMT, John Neffenger wrote: >> This pull request allows for reproducible builds of JavaFX on Linux, macOS, >> and Windows by defining the `SOURCE_DATE_EPOCH` environment variable. For >> example, the following commands create a reproducible build: >> >> >> $ export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) >> $ bash gradlew sdk jmods javadoc >> $ strip-nondeterminism -v -T $SOURCE_DATE_EPOCH build/jmods/*.jmod >> >> >> The three commands: >> >> 1. set the build timestamp to the date of the latest source code change, >> 2. build the JavaFX SDK libraries, JMOD archives, and API documentation, and >> 3. recreate the JMOD files with stable file modification times and ordering. >> >> The third command won't be necessary once Gradle can build the JMOD archives >> or the `jmod` tool itself has the required support. For more information on >> the environment variable, see the [`SOURCE_DATE_EPOCH`][1] page. For more >> information on the command to recreate the JMOD files, see the >> [`strip-nondeterminism`][2] repository. I'd like to propose that we allow >> for reproducible builds in JavaFX 17 and consider making them the default in >> JavaFX 18. >> >> Fixes >> >> There are at least four sources of non-determinism in the JavaFX builds: >> >> 1. Build timestamp >> >> The class `com.sun.javafx.runtime.VersionInfo` in the JavaFX Base module >> stores the time of the build. Furthermore, for builds that don't run on the >> Hudson continuous integration tool, the class adds the build time to the >> system property `javafx.runtime.version`. >> >> 2. Modification times >> >> The JAR, JMOD, and ZIP archives store the modification time of each file. >> >> 3. File ordering >> >> The JAR, JMOD, and ZIP archives store their files in the order returned >> by the file system. The native shared libraries also store their object >> files in the order returned by the file system. Most file systems, though, >> do not guarantee the order of a directory's file listing. >> >> 4. Build path >> >> The class `com.sun.javafx.css.parser.Css2Bin` in the JavaFX Graphics >> module stores the absolute path of its `.css` input file in the >> corresponding `.bss` output file, which is then included in the JavaFX >> Controls module. >> >> This pull request modifies the Gradle and Groovy build files to fix the >> first three sources of non-determinism. A later pull request can modify the >> Java files to fix the fourth. >> >> [1]: https://reproducible-builds.org/docs/source-date-epoch/ >> [2]: https://salsa.debian.org/reproducible-builds/strip-nondeterminism > > John Neffenger has updated the pull request with a new target base due to a > merge or a rebase. The pull request now contains seven commits: > > - Make build of SDK ZIP bundle reproducible > - Merge branch 'master' into allow-reproducible-builds > - Merge branch 'master' into allow-reproducible-builds > - Include WebKit shared library for Windows > >Enable reproducible builds of the native WebKit shared library for >Windows (jfxwebkit.dll) when SOURCE_DATE_EPOCH is defined. > - Include media shared libraries for Windows > >Enable reproducible builds of the native media shared libraries for >Windows when SOURCE_DATE_EPOCH is defined. The libraries are: > > fxplugins.dll > glib-lite.dll > gstreamer-lite.dll > jfxmedia.dll > - Enable reproducible builds with SOURCE_DATE_EPOCH > - 8238650: Allow to override buildDate with SOURCE_DATE_EPOCH I re-tested the following build commands on Linux, macOS, and Windows. 1. Developer - the tasks I use for my JavaFX developer builds: `gradle sdk jmods javadoc` 2. GitHub Actions - the task used by the repository to test pull requests: `gradle all` 3. Production - the properties and tasks for the JavaFX production builds: `gradle -PCONF=Release -PPROMOTED_BUILD_NUMBER=12 -PHUDSON_BUILD_NUMBER=101 -PHUDSON_JOB_NAME=jfx -PCOMPILE_WEBKIT=true -PCOMPILE_MEDIA=true -PBUILD_LIBAV_STUBS=true sdk jmods javadoc` I also ran the unit test tasks on each platform: `test -x :web:test` for the first two and `test` for the third. In each case, the files under the `build` directory created by two consecutive builds from the same location are identical except for the JMOD files, as explained in the description at the top of this pull request. For now, the JMOD files and their bundled artifact `javafx-jmods-17.zip` can be normalized using the `strip-nondeterminism` command. The table below shows the number of files created under the `build` directory for each type of build on the three platforms: | Build Type | Linux | macOS | Win10 | | -- | --:| --:| --:| | Developer | 17,109 | 17,121 | 17,338 | | GitHub Actions | 12,600 | 12,608 | 12,563 | | Production | 10,676 | 10,680 | 10,633 | - PR: https://git.openjdk.java.net/jfx/pull/446
Re: RFR: 8264449: Enable reproducible builds with SOURCE_DATE_EPOCH [v5]
> This pull request allows for reproducible builds of JavaFX on Linux, macOS, > and Windows by defining the `SOURCE_DATE_EPOCH` environment variable. For > example, the following commands create a reproducible build: > > > $ export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) > $ bash gradlew sdk jmods javadoc > $ strip-nondeterminism -v -T $SOURCE_DATE_EPOCH build/jmods/*.jmod > > > The three commands: > > 1. set the build timestamp to the date of the latest source code change, > 2. build the JavaFX SDK libraries, JMOD archives, and API documentation, and > 3. recreate the JMOD files with stable file modification times and ordering. > > The third command won't be necessary once Gradle can build the JMOD archives > or the `jmod` tool itself has the required support. For more information on > the environment variable, see the [`SOURCE_DATE_EPOCH`][1] page. For more > information on the command to recreate the JMOD files, see the > [`strip-nondeterminism`][2] repository. I'd like to propose that we allow for > reproducible builds in JavaFX 17 and consider making them the default in > JavaFX 18. > > Fixes > > There are at least four sources of non-determinism in the JavaFX builds: > > 1. Build timestamp > > The class `com.sun.javafx.runtime.VersionInfo` in the JavaFX Base module > stores the time of the build. Furthermore, for builds that don't run on the > Hudson continuous integration tool, the class adds the build time to the > system property `javafx.runtime.version`. > > 2. Modification times > > The JAR, JMOD, and ZIP archives store the modification time of each file. > > 3. File ordering > > The JAR, JMOD, and ZIP archives store their files in the order returned > by the file system. The native shared libraries also store their object files > in the order returned by the file system. Most file systems, though, do not > guarantee the order of a directory's file listing. > > 4. Build path > > The class `com.sun.javafx.css.parser.Css2Bin` in the JavaFX Graphics > module stores the absolute path of its `.css` input file in the corresponding > `.bss` output file, which is then included in the JavaFX Controls module. > > This pull request modifies the Gradle and Groovy build files to fix the first > three sources of non-determinism. A later pull request can modify the Java > files to fix the fourth. > > [1]: https://reproducible-builds.org/docs/source-date-epoch/ > [2]: https://salsa.debian.org/reproducible-builds/strip-nondeterminism John Neffenger has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains seven commits: - Make build of SDK ZIP bundle reproducible - Merge branch 'master' into allow-reproducible-builds - Merge branch 'master' into allow-reproducible-builds - Include WebKit shared library for Windows Enable reproducible builds of the native WebKit shared library for Windows (jfxwebkit.dll) when SOURCE_DATE_EPOCH is defined. - Include media shared libraries for Windows Enable reproducible builds of the native media shared libraries for Windows when SOURCE_DATE_EPOCH is defined. The libraries are: fxplugins.dll glib-lite.dll gstreamer-lite.dll jfxmedia.dll - Enable reproducible builds with SOURCE_DATE_EPOCH - 8238650: Allow to override buildDate with SOURCE_DATE_EPOCH - Changes: https://git.openjdk.java.net/jfx/pull/446/files Webrev: https://webrevs.openjdk.java.net/?repo=jfx=446=04 Stats: 83 lines in 7 files changed: 55 ins; 13 del; 15 mod Patch: https://git.openjdk.java.net/jfx/pull/446.diff Fetch: git fetch https://git.openjdk.java.net/jfx pull/446/head:pull/446 PR: https://git.openjdk.java.net/jfx/pull/446
Withdrawn: 8238650: Allow to override buildDate with SOURCE_DATE_EPOCH
On Tue, 9 Mar 2021 22:20:10 GMT, John Neffenger wrote: > This is a continuation of the [pull request][1] started by @bmwiedemann in > January 2020. After this change is integrated, I can follow up immediately > with additional pull requests that get us much closer to providing fully > reproducible builds. > > Motivation > > The only conclusive way to verify a software package is to reproduce it. > That's the main point of the Linux Foundation article [Preventing Supply > Chain Attacks like SolarWinds][2] by David Wheeler, Director of Open Source > Supply Chain Security. "In the longer term," he writes, "I know of only one > strong countermeasure for this kind of attack: verified reproducible builds." > > It's not enough anymore to trust the person, organization, or company that > publishes a software package. David Wheeler explains, "Assuming a system can > 'never be broken into' is a failing strategy." As I see it, any project that > doesn't yet allow for reproducible builds is on a list of possible attack > vectors. I'd like to get OpenJFX off that list. > > This is a huge undertaking involving the entire open-source community. Just > [Debian, for example][3], has over 30,000 source packages to build in a > reproducible manner. Remarkably, it's almost 96 percent complete. The OpenJFX > package is one of three percent still failing. Our first step towards helping > in this goal is to support the [`SOURCE_DATE_EPOCH` specification][4]. > > Implementation > > When you want to build 30,000 packages in a reproducible manner, > [command-line flags][5] unique to each package aren't so helpful. The > environment variable needs to be set, anyway, for the tools invoked by Gradle > to pick it up. We could allow for a Gradle property in addition to the > environment variable. The Gradle property would override the default current > date and export the environment variable, and the environment variable would > override the command-line property. I think it makes more sense in the > OpenJFX build to support the environment variable directly. > > With these considerations, I added the support just as recommended by the > example for "Java / gradle" on the Reproducible Builds > [`SOURCE_DATE_EPOCH`][6] page. For comparison, the [OpenJDK build][7] does > the reverse, using the *configure* script option `--with-source-date` to > export the `SOURCE_DATE_EPOCH` environment variable. > > [1]: https://github.com/openjdk/jfx/pull/99 > [2]: > https://www.linuxfoundation.org/en/blog/preventing-supply-chain-attacks-like-solarwinds > [3]: https://tests.reproducible-builds.org/debian/reproducible.html > [4]: https://reproducible-builds.org/specs/source-date-epoch/ > [5]: > https://wiki.debian.org/ReproducibleBuilds/StandardEnvironmentVariables#More_detailed_discussion > [6]: https://reproducible-builds.org/docs/source-date-epoch/ > [7]: https://github.com/openjdk/jdk/commit/1a16a4b6 This pull request has been closed without being integrated. - PR: https://git.openjdk.java.net/jfx/pull/422
Re: RFR: 8238650: Allow to override buildDate with SOURCE_DATE_EPOCH [v2]
On Fri, 30 Apr 2021 00:10:30 GMT, John Neffenger wrote: >> This is a continuation of the [pull request][1] started by @bmwiedemann in >> January 2020. After this change is integrated, I can follow up immediately >> with additional pull requests that get us much closer to providing fully >> reproducible builds. >> >> Motivation >> >> The only conclusive way to verify a software package is to reproduce it. >> That's the main point of the Linux Foundation article [Preventing Supply >> Chain Attacks like SolarWinds][2] by David Wheeler, Director of Open Source >> Supply Chain Security. "In the longer term," he writes, "I know of only one >> strong countermeasure for this kind of attack: verified reproducible builds." >> >> It's not enough anymore to trust the person, organization, or company that >> publishes a software package. David Wheeler explains, "Assuming a system can >> 'never be broken into' is a failing strategy." As I see it, any project that >> doesn't yet allow for reproducible builds is on a list of possible attack >> vectors. I'd like to get OpenJFX off that list. >> >> This is a huge undertaking involving the entire open-source community. Just >> [Debian, for example][3], has over 30,000 source packages to build in a >> reproducible manner. Remarkably, it's almost 96 percent complete. The >> OpenJFX package is one of three percent still failing. Our first step >> towards helping in this goal is to support the [`SOURCE_DATE_EPOCH` >> specification][4]. >> >> Implementation >> >> When you want to build 30,000 packages in a reproducible manner, >> [command-line flags][5] unique to each package aren't so helpful. The >> environment variable needs to be set, anyway, for the tools invoked by >> Gradle to pick it up. We could allow for a Gradle property in addition to >> the environment variable. The Gradle property would override the default >> current date and export the environment variable, and the environment >> variable would override the command-line property. I think it makes more >> sense in the OpenJFX build to support the environment variable directly. >> >> With these considerations, I added the support just as recommended by the >> example for "Java / gradle" on the Reproducible Builds >> [`SOURCE_DATE_EPOCH`][6] page. For comparison, the [OpenJDK build][7] does >> the reverse, using the *configure* script option `--with-source-date` to >> export the `SOURCE_DATE_EPOCH` environment variable. >> >> [1]: https://github.com/openjdk/jfx/pull/99 >> [2]: >> https://www.linuxfoundation.org/en/blog/preventing-supply-chain-attacks-like-solarwinds >> [3]: https://tests.reproducible-builds.org/debian/reproducible.html >> [4]: https://reproducible-builds.org/specs/source-date-epoch/ >> [5]: >> https://wiki.debian.org/ReproducibleBuilds/StandardEnvironmentVariables#More_detailed_discussion >> [6]: https://reproducible-builds.org/docs/source-date-epoch/ >> [7]: https://github.com/openjdk/jdk/commit/1a16a4b6 > > John Neffenger has updated the pull request with a new target base due to a > merge or a rebase. The incremental webrev excludes the unrelated changes > brought in by the merge/rebase. The pull request contains two additional > commits since the last revision: > > - Merge branch 'master' into source-date-epoch > - 8238650: Allow to override buildDate with SOURCE_DATE_EPOCH Closing this pull request in favor of #446. The [changes in this pull request][1] are included as the [first commit][2] of pull request #446. [1]: https://github.com/openjdk/jfx/pull/422/commits/3ada5a30c490abec884d878b4fd86219f2f8c60c [2]: https://github.com/openjdk/jfx/pull/446/commits/7ef5031c25fe52c3967c482d6960544aa0e1c9f5 - PR: https://git.openjdk.java.net/jfx/pull/422
Re: RFR: 8264449: Enable reproducible builds with SOURCE_DATE_EPOCH [v3]
On Fri, 16 Apr 2021 19:05:34 GMT, Kevin Rushforth wrote: >> John Neffenger has updated the pull request incrementally with one >> additional commit since the last revision: >> >> Include WebKit shared library for Windows >> >> Enable reproducible builds of the native WebKit shared library for >> Windows (jfxwebkit.dll) when SOURCE_DATE_EPOCH is defined. > > Note that this PR is dependent on PR #422 . > > @jgneff As an alternative, you could close that PR (since all of the changes > from that one are currently included here), and list both issues in this PR > with Skara's `/issue add ...` command. It's up to you. @kevinrushforth I'll take the opportunity provided by this *merge-conflict* to simplify as you suggested. I'll close pull request #422, list both issues in this pull request, add Bernhard as a contributor, and add a commit that makes the `all` task reproducible. That will consolidate what would have been three pull requests into one and will leave only the build path problem for a follow-up request (fix number 4 in my first comment). - PR: https://git.openjdk.java.net/jfx/pull/446
Re: RFR: 8266396: Save VSCMD_DEBUG output in Windows build [v3]
On Tue, 11 May 2021 01:17:32 GMT, John Neffenger wrote: >> The Windows build calls a series of batch files to get the Visual Studio >> paths and environment variables. The batch files are a black box: any >> messages they print are discarded. If anything goes wrong, the only signs >> are a vague Gradle exception and a corrupted properties file. >> >> This has been causing problems for years. There are at least 49 messages on >> the mailing since 2017 that discuss the exception and ways to work around it. >> >> This pull request lets you enable the batch file messages, shedding light on >> their internal workings and allowing you to catch any errors at their >> source. Specifically, it adds the variable `VSCMD_DEBUG` to the environment >> of `genVSproperties.bat` and documents its use. >> >> ### Before >> >> For example, if your `PATH` environment variable is missing >> `C:/Windows/System32`, like mine was, you'll see the following errors: >> >> >> $ gradle sdk >> Dependency verification is an incubating feature. >> >> FAILURE: Build failed with an exception. >> >> * Where: >> Script 'C:\cygwin64\home\john\src\jfx\buildSrc\win.gradle' line: 108 >> >> * What went wrong: >> A problem occurred evaluating script. >>> FAIL: WINSDK_DIR not defined >> >> ︙ >> >> BUILD FAILED in 4s >> >> >> *Me:* What's a `WINSDK_DIR`? The Wiki says, "This means that you will have >> to define these paths manually." I guess this is normal. ️ >> >> ### After >> >> With this change, you can set the debug level to "3" in the `win.gradle` >> file and get a better picture of the problem: >> >> >> $ gradle sdk >> Dependency verification is an incubating feature. >> >>> Configure project : >> 'reg' is not recognized as an internal or external command, >> operable program or batch file. >> 'reg' is not recognized as an internal or external command, >> operable program or batch file. >> 'reg' is not recognized as an internal or external command, >> operable program or batch file. >> >> ︙ >> >> 'powershell.exe' is not recognized as an internal or external command, >> operable program or batch file. >> >> FAILURE: Build failed with an exception. >> >> * Where: >> Script 'C:\cygwin64\home\john\src\jfx\buildSrc\win.gradle' line: 108 >> >> * What went wrong: >> A problem occurred evaluating script. >>> FAIL: WINSDK_DIR not defined >> >> ︙ >> >> BUILD FAILED in 3s >> >> >> *Me:* Oh, it's just a "command not found" error. I'll check my `PATH`. 邏 > > John Neffenger has updated the pull request incrementally with one additional > commit since the last revision: > > Skip sending telemetry to fix "file in use" error Thank you for the review, Marius. > But turns out the whole build folder should be deleted first as otherwise you > will get the error again immediately. Yes, that's a good point! If you don't delete the `build` directory after the error, you'll get stuck and even `gradle clean` and `gradle cleanAll` won't let you recover. That's not obvious the first time it happens. > I think it will be more clear for everyone including new developers. :) I hope so, especially considering that the Windows platform is probably our largest source of potential new contributors. I fell into this trap after building JavaFX on Windows for years, though, so it might even help old developers, too. - PR: https://git.openjdk.java.net/jfx/pull/488
Re: Minimum JDK policy for OpenJFX
On 5/19/21 1:17 PM, Ty Young wrote: Biggest things for JavaFX that I can think of is jextract, a tool for generating Java headers from a C header, and having all binding code written in Java. JavaFX has been doing its own manual form of Project Panama since 2014. Look for the string "extends C.Structure" in the following two files: https://github.com/openjdk/jfx/blob/master/modules/javafx.graphics/src/main/java/com/sun/glass/ui/monocle/MX6Cursor.java https://github.com/openjdk/jfx/blob/master/modules/javafx.graphics/src/main/java/com/sun/glass/ui/monocle/EPDSystem.java In both cases, the technique was used to let JavaFX bypass the header file 'mxcfb.h' from NXP (formerly Freescale). I'm looking forward to replacing those hard-coded offsets with the tools from Project Panama. In fact, I'm hopeful that Project Panama will let me remove all of the native C code from the Monocle EPD platform. John
Re: Unknown command v - for a list of valid commands use /help.
On 5/25/21 2:26 PM, John Neffenger wrote: Unknown command v - for a list of valid commands use /help. Just to follow up, there's a Skara bug report for this glitch: SKARA-755: Anything starting with a "/" is treated as a command https://bugs.openjdk.java.net/browse/SKARA-755 John
Unknown command v - for a list of valid commands use /help.
$ grep -A2 'reg query' build/vcvarsall.log
C:\cygwin64\home\john\src\jfx>for /F "tokens=1,2*" %i in
('reg query "HKLM\SOFTWARE\Wow6432Node\Microsoft\Microsoft SDKs\Windows\v10.0"
I did it again -- started a line with a Windows command option inside a
Shell Session code block, resulting in a truncated message on the
mailing list and the feedback:
Unknown command v - for a list of valid commands use /help.
For my full comment, please see:
https://github.com/openjdk/jfx/pull/488#issuecomment-848264495
Sorry about that.
John
Re: RFR: 8266396: Save VSCMD_DEBUG output in Windows build [v3]
On Thu, 13 May 2021 19:11:26 GMT, John Neffenger wrote:
>> John Neffenger has updated the pull request incrementally with one
>> additional commit since the last revision:
>>
>> Skip sending telemetry to fix "file in use" error
>
> I wrote a Bash shell script, included below, that can help in reviewing this
> pull request. It isolates the call to `vcvarsall.bat`, making it easier to
> experiment with different choices, such as:
>
> - leaving `VSCMD_SKIP_SENDTELEMETRY` undefined,
> - adding the `/q` option to `cmd`,
> - removing `2>&1` to use the default standard error, or
> - discarding standard output with `> /dev/null`.
>
>
> #!/bin/bash
> # Tests the Visual Studio 'vcvarsall.bat' batch file
> trap exit INT TERM
> set -o errexit
>
> # Path to 'vcvarsall.bat' batch file
> vsroot="C:\\Program Files (x86)\\Microsoft Visual Studio"
> vcfile="2019\\Community\\VC\\Auxiliary\\Build\\vcvarsall.bat"
> vcpath="$vsroot\$vcfile"
>
> # Windows command interpreter and options
> # /C Carries out the command specified by string and then terminates
> # /Q Turns echo off
> command="cmd /c"
>
> # Skips calling PowerShell script to send telemetry
> export VSCMD_SKIP_SENDTELEMETRY="1"
> printf "VSCMD_SKIP_SENDTELEMETRY="%s"\n" "$VSCMD_SKIP_SENDTELEMETRY"
>
> # VSCMD_DEBUG undefined
> unset VSCMD_DEBUG
> printf "Testing VSCMD_DEBUG="%s"\n" "$VSCMD_DEBUG"
> $command "$vcpath" x86 > 0-debug.log 2>&1
> $command "$vcpath" x64 >> 0-debug.log 2>&1
>
> # VSCMD_DEBUG = 1 (basic), 2 (detailed), and 3 (trace)
> for n in 1 2 3; do
> export VSCMD_DEBUG="$n"
> printf "Testing VSCMD_DEBUG="%s"\n" "$VSCMD_DEBUG"
> $command "$vcpath" x86 > $n-debug.log 2>&1
> $command "$vcpath" x64 >> $n-debug.log 2>&1
> done
>
>
> I run the script under Cygwin on Windows and evaluate its output as follows:
>
>
> $ ./vcvarstest.sh
> VSCMD_SKIP_SENDTELEMETRY="1"
> Testing VSCMD_DEBUG=""
> Testing VSCMD_DEBUG="1"
> Testing VSCMD_DEBUG="2"
> Testing VSCMD_DEBUG="3"
> $ wc -l *.log
> 10 0-debug.log
> 92 1-debug.log
>508 2-debug.log
> 5690 3-debug.log
> 6300 total
> @jgneff, do you have an idea of what kind of errors would be better detected
> with these changes?
Thanks for trying it, Joeri. The intent is to make all such errors better
detected, but that can happen only when you run the build with `VSCMD_DEBUG=3`
and when you're able to find the errors in the log file.
Without `VSCMD_DEBUG=3`, the batch files discard the output and errors of each
`reg query` command, as shown below in the file `winsdk.bat`:
:GetWin10SdkDir
if "%VSCMD_DEBUG%" GEQ "3" goto :GetWin10SdkDirVerbose
call :GetWin10SdkDirHelper HKLM\SOFTWARE\Wow6432Node > nul 2>&1
In addition, it helps to have an idea what to look for in the log file because
not all errors are prefixed with something helpful like `ERROR`. In the test
that you ran, the following steps could lead you to the source of the problem:
$ export VSCMD_DEBUG=3
$ gradle sdk
︙
> FAIL: WINSDK_DIR not defined
︙
$ grep -A2 'reg query' build/vcvarsall.log
C:\cygwin64\home\john\src\jfx>for /F "tokens=1,2*" %i in
('reg query "HKLM\SOFTWARE\Wow6432Node\Microsoft\Microsoft SDKs\Windows\v10.0"
-
PR: https://git.openjdk.java.net/jfx/pull/488
Re: RFR: JDK-8266396: Allow VSCMD_DEBUG to work as intended [v3]
On Tue, 11 May 2021 01:17:32 GMT, John Neffenger wrote: >> The Windows build calls a series of batch files to get the Visual Studio >> paths and environment variables. The batch files are a black box: any >> messages they print are discarded. If anything goes wrong, the only signs >> are a vague Gradle exception and a corrupted properties file. >> >> This has been causing problems for years. There are at least 49 messages on >> the mailing since 2017 that discuss the exception and ways to work around it. >> >> This pull request lets you enable the batch file messages, shedding light on >> their internal workings and allowing you to catch any errors at their >> source. Specifically, it adds the variable `VSCMD_DEBUG` to the environment >> of `genVSproperties.bat` and documents its use. >> >> ### Before >> >> For example, if your `PATH` environment variable is missing >> `C:/Windows/System32`, like mine was, you'll see the following errors: >> >> >> $ gradle sdk >> Dependency verification is an incubating feature. >> >> FAILURE: Build failed with an exception. >> >> * Where: >> Script 'C:\cygwin64\home\john\src\jfx\buildSrc\win.gradle' line: 108 >> >> * What went wrong: >> A problem occurred evaluating script. >>> FAIL: WINSDK_DIR not defined >> >> ︙ >> >> BUILD FAILED in 4s >> >> >> *Me:* What's a `WINSDK_DIR`? The Wiki says, "This means that you will have >> to define these paths manually." I guess this is normal. ️ >> >> ### After >> >> With this change, you can set the debug level to "3" in the `win.gradle` >> file and get a better picture of the problem: >> >> >> $ gradle sdk >> Dependency verification is an incubating feature. >> >>> Configure project : >> 'reg' is not recognized as an internal or external command, >> operable program or batch file. >> 'reg' is not recognized as an internal or external command, >> operable program or batch file. >> 'reg' is not recognized as an internal or external command, >> operable program or batch file. >> >> ︙ >> >> 'powershell.exe' is not recognized as an internal or external command, >> operable program or batch file. >> >> FAILURE: Build failed with an exception. >> >> * Where: >> Script 'C:\cygwin64\home\john\src\jfx\buildSrc\win.gradle' line: 108 >> >> * What went wrong: >> A problem occurred evaluating script. >>> FAIL: WINSDK_DIR not defined >> >> ︙ >> >> BUILD FAILED in 3s >> >> >> *Me:* Oh, it's just a "command not found" error. I'll check my `PATH`. 邏 > > John Neffenger has updated the pull request incrementally with one additional > commit since the last revision: > > Skip sending telemetry to fix "file in use" error I wrote a Bash shell script, included below, that can help in reviewing this pull request. It isolates the call to `vcvarsall.bat`, making it easier to experiment with different choices, such as: - leaving `VSCMD_SKIP_SENDTELEMETRY` undefined, - adding the `/q` option to `cmd`, - removing `2>&1` to use the default standard error, or - discarding standard output with `> /dev/null`. #!/bin/bash # Tests the Visual Studio 'vcvarsall.bat' batch file trap exit INT TERM set -o errexit # Path to 'vcvarsall.bat' batch file vsroot="C:\\Program Files (x86)\\Microsoft Visual Studio" vcfile="2019\\Community\\VC\\Auxiliary\\Build\\vcvarsall.bat" vcpath="$vsroot\$vcfile" # Windows command interpreter and options # /C Carries out the command specified by string and then terminates # /Q Turns echo off command="cmd /c" # Skips calling PowerShell script to send telemetry export VSCMD_SKIP_SENDTELEMETRY="1" printf "VSCMD_SKIP_SENDTELEMETRY="%s"\n" "$VSCMD_SKIP_SENDTELEMETRY" # VSCMD_DEBUG undefined unset VSCMD_DEBUG printf "Testing VSCMD_DEBUG="%s"\n" "$VSCMD_DEBUG" $command "$vcpath" x86 > 0-debug.log 2>&1 $command "$vcpath" x64 >> 0-debug.log 2>&1 # VSCMD_DEBUG = 1 (basic), 2 (detailed), and 3 (trace) for n in 1 2 3; do export VSCMD_DEBUG="$n" printf "Testing VSCMD_DEBUG="%s"\n" "$VSCMD_DEBUG" $command "$vcpath" x86 > $n-debug.log 2>&1 $command "$vcpath" x64 >> $n-debug.log 2>&1 done I run the script under Cygwin on Windows and evaluate its output as follows: $ ./vcvarstest.sh VSCMD_SKIP_SENDTELEMETRY="1" Testing VSCMD_DEBUG="" Testing VSCMD_DEBUG="1" Testing VSCMD_DEBUG="2" Testing VSCMD_DEBUG="3" $ wc -l *.log 10 0-debug.log 92 1-debug.log 508 2-debug.log 5690 3-debug.log 6300 total - PR: https://git.openjdk.java.net/jfx/pull/488
Re: Build error with gradle (command line)
On 5/11/21 10:27 AM, Johan Vos wrote: I'm still +1 to keep Gradle for OpenJFX. The OpenJFX build is shockingly complicated. It's not just a Gradle build. It's a Gradle, Groovy Custom Task, Apache Maven, Apache Ant, GNU Make, CMake, Ninja, ANTLR, JUnit, GNU GCC, XCode, Visual Studio, MSBuild, and Windows Batch File build. I think the only missing is Bazel. John
Re: Build error with gradle (command line)
On 5/11/21 5:24 AM, Jeanette Winzenburg wrote: deleting the caches did work, at last ;) That's also what I had to do after similar errors. I thought there might be some bumps in the road when I proposed adding the Gradle dependency verification, but I hope we can retain enough of it to make the builds safer than before. If we notice that the the POM files are changing (without updating their versions), Kevin's idea of removing the POM entries should help. Even the Gradle documentation anticipates some problems, saying "It means that you will be tempted to switch it off." [1] The more I learn Gradle, the less likely I am to choose it for my own projects, but it is far ahead of Maven, for example, in protecting against supply-chain attacks. For Maven, this feature is still just a couple of old bug reports: Extend the Project Object Model (POM) with trust information (OpenPGP, hash values) https://issues.apache.org/jira/browse/MNG-6026 Switch the default checksum policy from "warn" to "fail" https://issues.apache.org/jira/browse/MNG-5728 John [1] https://docs.gradle.org/current/userguide/dependency_verification.html
Re: RFR: JDK-8266396: Allow VSCMD_DEBUG to work as intended [v3]
On Tue, 11 May 2021 01:17:32 GMT, John Neffenger wrote: >> The Windows build calls a series of batch files to get the Visual Studio >> paths and environment variables. The batch files are a black box: any >> messages they print are discarded. If anything goes wrong, the only signs >> are a vague Gradle exception and a corrupted properties file. >> >> This has been causing problems for years. There are at least 49 messages on >> the mailing since 2017 that discuss the exception and ways to work around it. >> >> This pull request lets you enable the batch file messages, shedding light on >> their internal workings and allowing you to catch any errors at their >> source. Specifically, it adds the variable `VSCMD_DEBUG` to the environment >> of `genVSproperties.bat` and documents its use. >> >> ### Before >> >> For example, if your `PATH` environment variable is missing >> `C:/Windows/System32`, like mine was, you'll see the following errors: >> >> >> $ gradle sdk >> Dependency verification is an incubating feature. >> >> FAILURE: Build failed with an exception. >> >> * Where: >> Script 'C:\cygwin64\home\john\src\jfx\buildSrc\win.gradle' line: 108 >> >> * What went wrong: >> A problem occurred evaluating script. >>> FAIL: WINSDK_DIR not defined >> >> ︙ >> >> BUILD FAILED in 4s >> >> >> *Me:* What's a `WINSDK_DIR`? The Wiki says, "This means that you will have >> to define these paths manually." I guess this is normal. ️ >> >> ### After >> >> With this change, you can set the debug level to "3" in the `win.gradle` >> file and get a better picture of the problem: >> >> >> $ gradle sdk >> Dependency verification is an incubating feature. >> >>> Configure project : >> 'reg' is not recognized as an internal or external command, >> operable program or batch file. >> 'reg' is not recognized as an internal or external command, >> operable program or batch file. >> 'reg' is not recognized as an internal or external command, >> operable program or batch file. >> >> ︙ >> >> 'powershell.exe' is not recognized as an internal or external command, >> operable program or batch file. >> >> FAILURE: Build failed with an exception. >> >> * Where: >> Script 'C:\cygwin64\home\john\src\jfx\buildSrc\win.gradle' line: 108 >> >> * What went wrong: >> A problem occurred evaluating script. >>> FAIL: WINSDK_DIR not defined >> >> ︙ >> >> BUILD FAILED in 3s >> >> >> *Me:* Oh, it's just a "command not found" error. I'll check my `PATH`. 邏 > > John Neffenger has updated the pull request incrementally with one additional > commit since the last revision: > > Skip sending telemetry to fix "file in use" error With today's fix, the table in my prior comment is the same. In the "Before" column, the messages are printed with the normal Gradle output. In the "After" column, they're found in the new `vcvarsall.log` file. The table is included again below, showing the number of lines logged: | VSCMD_DEBUG | Level| Before | After | |:---:|:-|---:|--:| | undefined | none | 0 |10 | | 1 | basic| 0 |93 | | 2 | detailed | 0 | 615 | | 3 | trace| 2 | 5,832 | No more "black box" Windows batch files. - PR: https://git.openjdk.java.net/jfx/pull/488
Re: RFR: JDK-8266396: Allow VSCMD_DEBUG to work as intended [v3]
> The Windows build calls a series of batch files to get the Visual Studio > paths and environment variables. The batch files are a black box: any > messages they print are discarded. If anything goes wrong, the only signs are > a vague Gradle exception and a corrupted properties file. > > This has been causing problems for years. There are at least 49 messages on > the mailing since 2017 that discuss the exception and ways to work around it. > > This pull request lets you enable the batch file messages, shedding light on > their internal workings and allowing you to catch any errors at their source. > Specifically, it adds the variable `VSCMD_DEBUG` to the environment of > `genVSproperties.bat` and documents its use. > > ### Before > > For example, if your `PATH` environment variable is missing > `C:/Windows/System32`, like mine was, you'll see the following errors: > > > $ gradle sdk > Dependency verification is an incubating feature. > > FAILURE: Build failed with an exception. > > * Where: > Script 'C:\cygwin64\home\john\src\jfx\buildSrc\win.gradle' line: 108 > > * What went wrong: > A problem occurred evaluating script. >> FAIL: WINSDK_DIR not defined > > ︙ > > BUILD FAILED in 4s > > > *Me:* What's a `WINSDK_DIR`? The Wiki says, "This means that you will have to > define these paths manually." I guess this is normal. ️ > > ### After > > With this change, you can set the debug level to "3" in the `win.gradle` file > and get a better picture of the problem: > > > $ gradle sdk > Dependency verification is an incubating feature. > >> Configure project : > 'reg' is not recognized as an internal or external command, > operable program or batch file. > 'reg' is not recognized as an internal or external command, > operable program or batch file. > 'reg' is not recognized as an internal or external command, > operable program or batch file. > > ︙ > > 'powershell.exe' is not recognized as an internal or external command, > operable program or batch file. > > FAILURE: Build failed with an exception. > > * Where: > Script 'C:\cygwin64\home\john\src\jfx\buildSrc\win.gradle' line: 108 > > * What went wrong: > A problem occurred evaluating script. >> FAIL: WINSDK_DIR not defined > > ︙ > > BUILD FAILED in 3s > > > *Me:* Oh, it's just a "command not found" error. I'll check my `PATH`. 邏 John Neffenger has updated the pull request incrementally with one additional commit since the last revision: Skip sending telemetry to fix "file in use" error - Changes: - all: https://git.openjdk.java.net/jfx/pull/488/files - new: https://git.openjdk.java.net/jfx/pull/488/files/6f6f4932..c58bbdd9 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jfx=488=02 - incr: https://webrevs.openjdk.java.net/?repo=jfx=488=01-02 Stats: 8 lines in 2 files changed: 3 ins; 2 del; 3 mod Patch: https://git.openjdk.java.net/jfx/pull/488.diff Fetch: git fetch https://git.openjdk.java.net/jfx pull/488/head:pull/488 PR: https://git.openjdk.java.net/jfx/pull/488
Re: RFR: JDK-8266396: Allow VSCMD_DEBUG to work as intended [v2]
On Mon, 10 May 2021 05:54:51 GMT, John Neffenger wrote: >> The Windows build calls a series of batch files to get the Visual Studio >> paths and environment variables. The batch files are a black box: any >> messages they print are discarded. If anything goes wrong, the only signs >> are a vague Gradle exception and a corrupted properties file. >> >> This has been causing problems for years. There are at least 49 messages on >> the mailing since 2017 that discuss the exception and ways to work around it. >> >> This pull request lets you enable the batch file messages, shedding light on >> their internal workings and allowing you to catch any errors at their >> source. Specifically, it adds the variable `VSCMD_DEBUG` to the environment >> of `genVSproperties.bat` and documents its use. >> >> ### Before >> >> For example, if your `PATH` environment variable is missing >> `C:/Windows/System32`, like mine was, you'll see the following errors: >> >> >> $ gradle sdk >> Dependency verification is an incubating feature. >> >> FAILURE: Build failed with an exception. >> >> * Where: >> Script 'C:\cygwin64\home\john\src\jfx\buildSrc\win.gradle' line: 108 >> >> * What went wrong: >> A problem occurred evaluating script. >>> FAIL: WINSDK_DIR not defined >> >> ︙ >> >> BUILD FAILED in 4s >> >> >> *Me:* What's a `WINSDK_DIR`? The Wiki says, "This means that you will have >> to define these paths manually." I guess this is normal. ️ >> >> ### After >> >> With this change, you can set the debug level to "3" in the `win.gradle` >> file and get a better picture of the problem: >> >> >> $ gradle sdk >> Dependency verification is an incubating feature. >> >>> Configure project : >> 'reg' is not recognized as an internal or external command, >> operable program or batch file. >> 'reg' is not recognized as an internal or external command, >> operable program or batch file. >> 'reg' is not recognized as an internal or external command, >> operable program or batch file. >> >> ︙ >> >> 'powershell.exe' is not recognized as an internal or external command, >> operable program or batch file. >> >> FAILURE: Build failed with an exception. >> >> * Where: >> Script 'C:\cygwin64\home\john\src\jfx\buildSrc\win.gradle' line: 108 >> >> * What went wrong: >> A problem occurred evaluating script. >>> FAIL: WINSDK_DIR not defined >> >> ︙ >> >> BUILD FAILED in 3s >> >> >> *Me:* Oh, it's just a "command not found" error. I'll check my `PATH`. 邏 > > John Neffenger has updated the pull request with a new target base due to a > merge or a rebase. The incremental webrev excludes the unrelated changes > brought in by the merge/rebase. The pull request contains three additional > commits since the last revision: > > - JDK-8266396: Allow VSCMD_DEBUG to work as intended > - Merge branch 'master' into vscmd-debug > - JDK-8266396: Add VSCMD_DEBUG to genVSproperties.bat environment To understand the build error that occurred with my last commit, you need to know three things: 1. by default, the JavaFX build on Windows always sends telemetry to Microsoft each time it runs the `vcvarsall.bat` batch file, 2. by default, the process that sends the telemetry is asynchronous, and 3. the Windows build runs that batch file twice, once for `x86` (via `vcvars32.bat`) and again for `x64`. The actual build error is way back at [Line 182][1] with the message "The process cannot access the file because it is being used by another process." It's a timing problem that occurs only when both `VSCMD_SKIP_SENDTELEMETRY` and `VSCMD_DEBUG` are undefined (or set to the empty string). In that case, the Visual Studio batch files start an asynchronous PowerShell script to send telemetry to Microsoft. Because the asynchronous process will continue to use the same standard error file descriptor as the batch file that started it, we hit the error when we try to run the batch file a second time before the PowerShell script has completed. If the PowerShell script completes before we run the batch file again, the error does not occur. I didn't hit the error because `powershell.exe` was not found on my `PATH`, essentially disabling the telemetry. The fix is simple: skip sending the telemetry. We should probably be doing that, anyway. An alternative, and perhaps safer, fix is not to redirect standard error when calling the `vcvarsall.bat` batch file. In that case, the trace messages would go to the log file but the error messages would go to the console, making it difficult to associate any errors with the commands that caused them. [1]: https://github.com/jgneff/jfx/runs/2542639912#step:8:182 - PR: https://git.openjdk.java.net/jfx/pull/488
Re: RFR: JDK-8266396: Add VSCMD_DEBUG for solving WINSDK_DIR build error [v2]
On Mon, 10 May 2021 05:54:51 GMT, John Neffenger wrote: >> The Windows build calls a series of batch files to get the Visual Studio >> paths and environment variables. The batch files are a black box: any >> messages they print are discarded. If anything goes wrong, the only signs >> are a vague Gradle exception and a corrupted properties file. >> >> This has been causing problems for years. There are at least 49 messages on >> the mailing since 2017 that discuss the exception and ways to work around it. >> >> This pull request lets you enable the batch file messages, shedding light on >> their internal workings and allowing you to catch any errors at their >> source. Specifically, it adds the variable `VSCMD_DEBUG` to the environment >> of `genVSproperties.bat` and documents its use. >> >> ### Before >> >> For example, if your `PATH` environment variable is missing >> `C:/Windows/System32`, like mine was, you'll see the following errors: >> >> >> $ gradle sdk >> Dependency verification is an incubating feature. >> >> FAILURE: Build failed with an exception. >> >> * Where: >> Script 'C:\cygwin64\home\john\src\jfx\buildSrc\win.gradle' line: 108 >> >> * What went wrong: >> A problem occurred evaluating script. >>> FAIL: WINSDK_DIR not defined >> >> ︙ >> >> BUILD FAILED in 4s >> >> >> *Me:* What's a `WINSDK_DIR`? The Wiki says, "This means that you will have >> to define these paths manually." I guess this is normal. ️ >> >> ### After >> >> With this change, you can set the debug level to "3" in the `win.gradle` >> file and get a better picture of the problem: >> >> >> $ gradle sdk >> Dependency verification is an incubating feature. >> >>> Configure project : >> 'reg' is not recognized as an internal or external command, >> operable program or batch file. >> 'reg' is not recognized as an internal or external command, >> operable program or batch file. >> 'reg' is not recognized as an internal or external command, >> operable program or batch file. >> >> ︙ >> >> 'powershell.exe' is not recognized as an internal or external command, >> operable program or batch file. >> >> FAILURE: Build failed with an exception. >> >> * Where: >> Script 'C:\cygwin64\home\john\src\jfx\buildSrc\win.gradle' line: 108 >> >> * What went wrong: >> A problem occurred evaluating script. >>> FAIL: WINSDK_DIR not defined >> >> ︙ >> >> BUILD FAILED in 3s >> >> >> *Me:* Oh, it's just a "command not found" error. I'll check my `PATH`. 邏 > > John Neffenger has updated the pull request with a new target base due to a > merge or a rebase. The incremental webrev excludes the unrelated changes > brought in by the merge/rebase. The pull request contains three additional > commits since the last revision: > > - JDK-8266396: Allow VSCMD_DEBUG to work as intended > - Merge branch 'master' into vscmd-debug > - JDK-8266396: Add VSCMD_DEBUG to genVSproperties.bat environment I committed a better proposal, thanks to Kevin's suggestion, and after seeing what it takes to create the `trace.bat` file in my previous comment. Unfortunately, it's not enough to define `VSCMD_DEBUG` externally. The `cmd /q` option (in `win.gradle`) and the redirection of output to `NUL` (in `genVSproperties.bat`) conspire to suppress the requested information, regardless of the variable's value. So this commit adds the minimum changes for `VSCMD_DEBUG` to work. Instead of redirecting output to `NUL`, it redirects output to a new log file called `vcvarsall.log`. Because the trace-level debugging of `VSCMD_DEBUG=3` depends on `echo` being enabled, it adds `echo off` and `echo on` commands selectively in the OpenJFX batch file instead of using the global `cmd /q` option in the Gradle build file. The table below shows the number of debugging lines that are logged for each value of `VSCMD_DEBUG`, both before this pull request (on the `master` branch) and after the latest commit (on the pull request branch): | VSCMD_DEBUG | Level| Before | After | |:---:|:-|---:|--:| | undefined | none | 0 |10 | | 1 | basic| 0 |93 | | 2 | detailed | 0 | 615 | | 3 | trace| 2 | 5,832 | With this change, `VSCMD_DEBUG` actually works as intended. When the build is successful, there's just an extra 10-line log file in the build directory. If something goes wrong, though, developers can set the environment variable and run `gradle help`. The log file will then contain all the information needed to fix the error. If they're having trouble figuring it out, they can post the log file and ask for help on the mailing list. - PR: https://git.openjdk.java.net/jfx/pull/488
Re: RFR: JDK-8266396: Add VSCMD_DEBUG for solving WINSDK_DIR build error [v2]
> The Windows build calls a series of batch files to get the Visual Studio > paths and environment variables. The batch files are a black box: any > messages they print are discarded. If anything goes wrong, the only signs are > a vague Gradle exception and a corrupted properties file. > > This has been causing problems for years. There are at least 49 messages on > the mailing since 2017 that discuss the exception and ways to work around it. > > This pull request lets you enable the batch file messages, shedding light on > their internal workings and allowing you to catch any errors at their source. > Specifically, it adds the variable `VSCMD_DEBUG` to the environment of > `genVSproperties.bat` and documents its use. > > ### Before > > For example, if your `PATH` environment variable is missing > `C:/Windows/System32`, like mine was, you'll see the following errors: > > > $ gradle sdk > Dependency verification is an incubating feature. > > FAILURE: Build failed with an exception. > > * Where: > Script 'C:\cygwin64\home\john\src\jfx\buildSrc\win.gradle' line: 108 > > * What went wrong: > A problem occurred evaluating script. >> FAIL: WINSDK_DIR not defined > > ︙ > > BUILD FAILED in 4s > > > *Me:* What's a `WINSDK_DIR`? The Wiki says, "This means that you will have to > define these paths manually." I guess this is normal. ️ > > ### After > > With this change, you can set the debug level to "3" in the `win.gradle` file > and get a better picture of the problem: > > > $ gradle sdk > Dependency verification is an incubating feature. > >> Configure project : > 'reg' is not recognized as an internal or external command, > operable program or batch file. > 'reg' is not recognized as an internal or external command, > operable program or batch file. > 'reg' is not recognized as an internal or external command, > operable program or batch file. > > ︙ > > 'powershell.exe' is not recognized as an internal or external command, > operable program or batch file. > > FAILURE: Build failed with an exception. > > * Where: > Script 'C:\cygwin64\home\john\src\jfx\buildSrc\win.gradle' line: 108 > > * What went wrong: > A problem occurred evaluating script. >> FAIL: WINSDK_DIR not defined > > ︙ > > BUILD FAILED in 3s > > > *Me:* Oh, it's just a "command not found" error. I'll check my `PATH`. 邏 John Neffenger has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains three additional commits since the last revision: - JDK-8266396: Allow VSCMD_DEBUG to work as intended - Merge branch 'master' into vscmd-debug - JDK-8266396: Add VSCMD_DEBUG to genVSproperties.bat environment - Changes: - all: https://git.openjdk.java.net/jfx/pull/488/files - new: https://git.openjdk.java.net/jfx/pull/488/files/248d0f72..6f6f4932 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jfx=488=01 - incr: https://webrevs.openjdk.java.net/?repo=jfx=488=00-01 Stats: 227 lines in 11 files changed: 179 ins; 21 del; 27 mod Patch: https://git.openjdk.java.net/jfx/pull/488.diff Fetch: git fetch https://git.openjdk.java.net/jfx pull/488/head:pull/488 PR: https://git.openjdk.java.net/jfx/pull/488
Re: RFR: JDK-8266396: Add VSCMD_DEBUG for solving WINSDK_DIR build error
On Fri, 7 May 2021 13:42:49 GMT, Kevin Rushforth wrote:
> It would be more convenient to ask the developer set `VSCMD_DEBUG` via an env
> variable, rather than asking them to edit `win.gradle`.
Thanks, Kevin. That is the most direct approach. I didn't document it that way
for the following reasons:
1. Using the `export` command under Cygwin sets the variable for the *entire*
build. In the current commit, the variable, when defined, is set only for the
duration of the `genVSproperties.bat` batch file and the batch files it calls.
2. I would like to see this documented in the code, even if just as comments,
as opposed to the Wiki. When I got the error, the first thing I did was search
through the code base for `WINSDK_DIR`. I would have found the lines added by
this pull request right away.
Don't let the name confused you. The variable is not a debugging variable; it
doesn't enable debug trace logging. What it actually does is prevent the batch
files from sending both standard output and standard error to "NUL", which they
do by default. Defining it with `VSCMD_DEBUG=3` really means, "Please don't
silently discard really important error messages!" You don't expect all error
messages to be hidden from you when you invoke 3,985 Windows batch file
statements, but that's exactly what's happening now:
$ cloc trace.bat
1 text file.
1 unique file.
0 files ignored.
github.com/AlDanial/cloc v 1.82 T=0.01 s (73.6 files/s, 438775.4 lines/s)
---
Language files blankcomment code
---
DOS Batch1 1938 40 3985
---
I'll up the ante and suggest that we set the value to "3" by default in the
build file (and disable telemetry). Then the batch files will behave as one
would expect. I tested all values of the variable (undefined, valid, and
invalid), and the generated file `windows_tools.properties` was identical in
all cases.
Otherwise, where do you suggest we document this? If you disfavor the current
options ("" and "3"), what do you think of making it three comment lines in the
`win.gradle` file?
-
PR: https://git.openjdk.java.net/jfx/pull/488
Re: RFR: 8242508: Upgrade to Visual Studio 2019 version 16.5.3
On Wed, 6 May 2020 20:37:10 GMT, Kevin Rushforth wrote: > This is a toolchain upgrade on Windows from the current Visual Studio 2017 > (version 15.9.16) to Visual Studio 2019 (version 16.5.3). This will match a > recent upgrade done for JDK 15 -- see > [JDK-8244214](https://bugs.openjdk.java.net/browse/JDK-8244214). > > I have run a full build and test using this new compiler. > > NOTE: although this isn't strictly dependent on > [JDK-8244487](https://bugs.openjdk.java.net/browse/JDK-8244487), which is out > for review as PR #211 , I plan to wait until that one is approved. I will > then integrate PR #211 followed by this PR. I also think that we should remove it. If you read its history on the mailing list, it was added to the repository in part as a solution to the `WINSDK_DIR not defined` error. See the message ["Problems building openjfx"][1] on the mailing list. That discussion starts with `WINSDK_DIR`. My goal in #488 is to give developers a better chance at getting the official build under Cygwin working after hitting the error. [1]: https://mail.openjdk.java.net/pipermail/openjfx-dev/2018-November/022858.html - PR: https://git.openjdk.java.net/jfx/pull/212
Re: RFR: 8242508: Upgrade to Visual Studio 2019 version 16.5.3
On Wed, 3 Mar 2021 22:07:34 GMT, Alessadro Parisi wrote: >> This is a toolchain upgrade on Windows from the current Visual Studio 2017 >> (version 15.9.16) to Visual Studio 2019 (version 16.5.3). This will match a >> recent upgrade done for JDK 15 -- see >> [JDK-8244214](https://bugs.openjdk.java.net/browse/JDK-8244214). >> >> I have run a full build and test using this new compiler. >> >> NOTE: although this isn't strictly dependent on >> [JDK-8244487](https://bugs.openjdk.java.net/browse/JDK-8244487), which is >> out for review as PR #211 , I plan to wait until that one is approved. I >> will then integrate PR #211 followed by this PR. > > Can you also update the build script located in tools/scripts? > Build fails on my system with `FAIL: WINSDK_DIR not defined` > Also, it's not possible to install Windows SDK 7.1 on Windows 10, so that > should be updated too @palexdev Please ignore my previous comment and see #488 if you're still getting the `WINSDK_DIR` error. - PR: https://git.openjdk.java.net/jfx/pull/212
Re: RFR: JDK-8266396: Add VSCMD_DEBUG for solving WINSDK_DIR build error
On 5/6/21 1:46 PM, John Neffenger wrote: $ reg query "HKLM\SOFTWARE\Microsoft\VisualStudio\VSPerf" \ The last part of my comment was truncated by Skara when it tripped over the '/v' Windows command-line option. It's repeated below for the mailing list ... We could instead set the default value to "3". In that case, I get two error messages during the Windows build due to following registry query failing on my system: $ reg query "HKLM\SOFTWARE\Microsoft\VisualStudio\VSPerf" /v "CollectionToolsDir2019" ERROR: The system was unable to find the specified registry key or value. ### Wiki I suggest that we remove the "Missing paths issue" section from the Wiki, added on October 31, 2019. It only led me astray. If you get `WINSDK_DIR not defined`, it's an error, and you should find its source and fix it. This pull request can help.
Re: RFR: JDK-8266396: Add VSCMD_DEBUG for solving WINSDK_DIR build error
On Thu, 6 May 2021 20:39:11 GMT, John Neffenger wrote:
> The Windows build calls a series of batch files to get the Visual Studio
> paths and environment variables. The batch files are a black box: any
> messages they print are discarded. If anything goes wrong, the only signs are
> a vague Gradle exception and a corrupted properties file.
>
> This has been causing problems for years. There are at least 49 messages on
> the mailing since 2017 that discuss the exception and ways to work around it.
>
> This pull request lets you enable the batch file messages, shedding light on
> their internal workings and allowing you to catch any errors at their source.
> Specifically, it adds the variable `VSCMD_DEBUG` to the environment of
> `genVSproperties.bat` and documents its use.
>
> ### Before
>
> For example, if your `PATH` environment variable is missing
> `C:/Windows/System32`, like mine was, you'll see the following errors:
>
>
> $ gradle sdk
> Dependency verification is an incubating feature.
>
> FAILURE: Build failed with an exception.
>
> * Where:
> Script 'C:\cygwin64\home\john\src\jfx\buildSrc\win.gradle' line: 108
>
> * What went wrong:
> A problem occurred evaluating script.
>> FAIL: WINSDK_DIR not defined
>
> ︙
>
> BUILD FAILED in 4s
>
>
> *Me:* What's a `WINSDK_DIR`? The Wiki says, "This means that you will have to
> define these paths manually." I guess this is normal. ️
>
> ### After
>
> With this change, you can set the debug level to "3" in the `win.gradle` file
> and get a better picture of the problem:
>
>
> $ gradle sdk
> Dependency verification is an incubating feature.
>
>> Configure project :
> 'reg' is not recognized as an internal or external command,
> operable program or batch file.
> 'reg' is not recognized as an internal or external command,
> operable program or batch file.
> 'reg' is not recognized as an internal or external command,
> operable program or batch file.
>
> ︙
>
> 'powershell.exe' is not recognized as an internal or external command,
> operable program or batch file.
>
> FAILURE: Build failed with an exception.
>
> * Where:
> Script 'C:\cygwin64\home\john\src\jfx\buildSrc\win.gradle' line: 108
>
> * What went wrong:
> A problem occurred evaluating script.
>> FAIL: WINSDK_DIR not defined
>
> ︙
>
> BUILD FAILED in 3s
>
>
> *Me:* Oh, it's just a "command not found" error. I'll check my `PATH`. 邏
### Notes
The sequence of nested calls to the batch files is shown below, where
`C:/Program Files (x86)/Microsoft Visual Studio` is the root directory of the
Visual Studio files:
buildSrc/genVSproperties.bat
↳ "2019/Community/VC/Auxiliary/Build/vcvarsall.bat"
↳ "2019/Community/Common7/Tools/VsDevCmd.bat"
↳ "2019/Community/Common7/Tools/vsdevcmd/core/winsdk.bat"
The file `VsDevCmd.bat` calls `powershell.exe` to send telemetry to Microsoft
when `VSCMD_DEBUG` is defined. To avoid sending telemetry, define the
environment variable `VSCMD_SKIP_SENDTELEMETRY`. For example, in Cygwin you can
define:
export VSCMD_SKIP_SENDTELEMETRY=1
This pull request sets `VSCMD_DEBUG` to the empty string (""). In batch files,
setting a variable to the empty string is equivalent to leaving the variable
undefined. Furthermore, setting the value in Gradle to the string "3" is the
same as setting the value to the number 3. To verify, I replaced
`genVSproperties.bat` with this batch file:
@echo off
if defined VSCMD_DEBUG (
echo VSCMD_DEBUG is defined 1>&2
) else (
echo VSCMD_DEBUG is not defined 1>&2
)
if "%VSCMD_DEBUG%" NEQ "" (
echo VSCMD_DEBUG = "%VSCMD_DEBUG%" 1>&2
) else (
echo VSCMD_DEBUG is the empty string "%VSCMD_DEBUG%" 1>&2
)
We could instead set the default value to "3". In that case, I get two error
messages during the Windows build due to following registry query failing on my
system:
$ reg query "HKLM\SOFTWARE\Microsoft\VisualStudio\VSPerf" \
-
PR: https://git.openjdk.java.net/jfx/pull/488
RFR: JDK-8266396: Add VSCMD_DEBUG for solving WINSDK_DIR build error
The Windows build calls a series of batch files to get the Visual Studio paths and environment variables. The batch files are a black box: any messages they print are discarded. If anything goes wrong, the only signs are a vague Gradle exception and a corrupted properties file. This has been causing problems for years. There are at least 49 messages on the mailing since 2017 that discuss the exception and ways to work around it. This pull request lets you enable the batch file messages, shedding light on their internal workings and allowing you to catch any errors at their source. Specifically, it adds the variable `VSCMD_DEBUG` to the environment of `genVSproperties.bat` and documents its use. ### Before For example, if your `PATH` environment variable is missing `C:/Windows/System32`, like mine was, you'll see the following errors: $ gradle sdk Dependency verification is an incubating feature. FAILURE: Build failed with an exception. * Where: Script 'C:\cygwin64\home\john\src\jfx\buildSrc\win.gradle' line: 108 * What went wrong: A problem occurred evaluating script. > FAIL: WINSDK_DIR not defined ︙ BUILD FAILED in 4s *Me:* What's a `WINSDK_DIR`? The Wiki says, "This means that you will have to define these paths manually." I guess this is normal. ️ ### After With this change, you can set the debug level to "3" in the `win.gradle` file and get a better picture of the problem: $ gradle sdk Dependency verification is an incubating feature. > Configure project : 'reg' is not recognized as an internal or external command, operable program or batch file. 'reg' is not recognized as an internal or external command, operable program or batch file. 'reg' is not recognized as an internal or external command, operable program or batch file. ︙ 'powershell.exe' is not recognized as an internal or external command, operable program or batch file. FAILURE: Build failed with an exception. * Where: Script 'C:\cygwin64\home\john\src\jfx\buildSrc\win.gradle' line: 108 * What went wrong: A problem occurred evaluating script. > FAIL: WINSDK_DIR not defined ︙ BUILD FAILED in 3s *Me:* Oh, it's just a "command not found" error. I'll check my `PATH`. 邏 - Commit messages: - JDK-8266396: Add VSCMD_DEBUG to genVSproperties.bat environment Changes: https://git.openjdk.java.net/jfx/pull/488/files Webrev: https://webrevs.openjdk.java.net/?repo=jfx=488=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8266396 Stats: 3 lines in 1 file changed: 3 ins; 0 del; 0 mod Patch: https://git.openjdk.java.net/jfx/pull/488.diff Fetch: git fetch https://git.openjdk.java.net/jfx pull/488/head:pull/488 PR: https://git.openjdk.java.net/jfx/pull/488
Re: Add method to save a JavaFX image to file without the use of SwingFXUtils?
On 5/4/21 5:08 AM, Frank Delporte wrote: I researched this topic for a few hours, but didn't manage to find a good approach to save a JavaFX Image. For what it's worth, I ran 224 benchmarks to find the fastest method to go in the other direction: read in an image using AWT and convert it to a JavaFX image. The results are posted here: Benchmarks https://jgneff.github.io/tofximage/ I found that the methods in SwingFXUtils are general enough that you can almost always write them to be much, much faster for your specific case. The code of the benchmarks is here: AWT to JavaFX Image Conversion Benchmarks https://github.com/jgneff/tofximage If you don't care about the alpha channel, this method came out tops: public void setArgbAsArgbPre(SourceAwtImage, ...) https://github.com/jgneff/tofximage/blob/b40a1e01238f1431295716ca09e4c40b82ac95d9/src/main/java/org/status6/tofximage/Benchmarks.java#L475 That method is shown below in a real application: public void handle(long now) https://github.com/jgneff/epd-javafx/blob/master/src/org/status6/epd/javafx/ImageAnimation.java#L172 John
Integrated: 8264010: Add Gradle dependency verification
On Tue, 23 Mar 2021 05:32:04 GMT, John Neffenger wrote: > This pull request adds dependency verification to the Gradle builds of JavaFX > on Linux, macOS, and Windows. It is the third of three changes that close the > gaps in the JavaFX build security: > > * [JDK-8262236][1]: Configure Gradle checksum verification > * [JDK-8263204][2]: Add Gradle Wrapper Validation Action > * [JDK-8264010][3]: Add Gradle dependency verification > > "Without dependency verification it's easy for an attacker to compromise your > supply chain," warns the [Gradle User Guide][4]. All three changes come from > conference talks by members of the Gradle team, available as [PDF slides][5] > or on YouTube in the following two videos: > > * [Cédric Champeau at Devoxx][6] in November 2019 > * [Louis Jacomet at Jfokus][7] in February 2020 > > "We all run in a crazy-unsafe environment, in a way," says Louis Jacomet at > the end of his talk. These three changes make it just a little less > crazy-unsafe for all of us building JavaFX, regardless of our system, > network, or country. > > [1]: https://bugs.openjdk.java.net/browse/JDK-8262236 > [2]: https://bugs.openjdk.java.net/browse/JDK-8263204 > [3]: https://bugs.openjdk.java.net/browse/JDK-8264010 > > [4]: https://docs.gradle.org/current/userguide/dependency_verification.html > [5]: > https://www.jfokus.se/jfokus20-preso/Protecting-your-organization-against-attacks-via-the-build-system.pdf > [6]: https://youtu.be/GWGNp3a3hpk > [7]: https://youtu.be/bwiafNatsf0 This pull request has now been integrated. Changeset: a9f6035c Author:John Neffenger URL: https://git.openjdk.java.net/jfx/commit/a9f6035c9c1d4dc60aa960498d8dbb5e52827017 Stats: 325 lines in 3 files changed: 313 ins; 4 del; 8 mod 8264010: Add Gradle dependency verification Co-authored-by: Kevin Rushforth Reviewed-by: kcr, jvos - PR: https://git.openjdk.java.net/jfx/pull/437
Re: RFR: 8264449: Enable reproducible builds with SOURCE_DATE_EPOCH [v4]
> This pull request allows for reproducible builds of JavaFX on Linux, macOS, > and Windows by defining the `SOURCE_DATE_EPOCH` environment variable. For > example, the following commands create a reproducible build: > > > $ export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) > $ bash gradlew sdk jmods javadoc > $ strip-nondeterminism -v -T $SOURCE_DATE_EPOCH build/jmods/*.jmod > > > The three commands: > > 1. set the build timestamp to the date of the latest source code change, > 2. build the JavaFX SDK libraries, JMOD archives, and API documentation, and > 3. recreate the JMOD files with stable file modification times and ordering. > > The third command won't be necessary once Gradle can build the JMOD archives > or the `jmod` tool itself has the required support. For more information on > the environment variable, see the [`SOURCE_DATE_EPOCH`][1] page. For more > information on the command to recreate the JMOD files, see the > [`strip-nondeterminism`][2] repository. I'd like to propose that we allow for > reproducible builds in JavaFX 17 and consider making them the default in > JavaFX 18. > > Fixes > > There are at least four sources of non-determinism in the JavaFX builds: > > 1. Build timestamp > > The class `com.sun.javafx.runtime.VersionInfo` in the JavaFX Base module > stores the time of the build. Furthermore, for builds that don't run on the > Hudson continuous integration tool, the class adds the build time to the > system property `javafx.runtime.version`. > > 2. Modification times > > The JAR, JMOD, and ZIP archives store the modification time of each file. > > 3. File ordering > > The JAR, JMOD, and ZIP archives store their files in the order returned > by the file system. The native shared libraries also store their object files > in the order returned by the file system. Most file systems, though, do not > guarantee the order of a directory's file listing. > > 4. Build path > > The class `com.sun.javafx.css.parser.Css2Bin` in the JavaFX Graphics > module stores the absolute path of its `.css` input file in the corresponding > `.bss` output file, which is then included in the JavaFX Controls module. > > This pull request modifies the Gradle and Groovy build files to fix the first > three sources of non-determinism. A later pull request can modify the Java > files to fix the fourth. > > [1]: https://reproducible-builds.org/docs/source-date-epoch/ > [2]: https://salsa.debian.org/reproducible-builds/strip-nondeterminism John Neffenger has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains five additional commits since the last revision: - Merge branch 'master' into allow-reproducible-builds - Include WebKit shared library for Windows Enable reproducible builds of the native WebKit shared library for Windows (jfxwebkit.dll) when SOURCE_DATE_EPOCH is defined. - Include media shared libraries for Windows Enable reproducible builds of the native media shared libraries for Windows when SOURCE_DATE_EPOCH is defined. The libraries are: fxplugins.dll glib-lite.dll gstreamer-lite.dll jfxmedia.dll - Enable reproducible builds with SOURCE_DATE_EPOCH - 8238650: Allow to override buildDate with SOURCE_DATE_EPOCH - Changes: - all: https://git.openjdk.java.net/jfx/pull/446/files - new: https://git.openjdk.java.net/jfx/pull/446/files/cb8510c6..c17eb491 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jfx=446=03 - incr: https://webrevs.openjdk.java.net/?repo=jfx=446=02-03 Stats: 90189 lines in 1212 files changed: 52172 ins; 13953 del; 24064 mod Patch: https://git.openjdk.java.net/jfx/pull/446.diff Fetch: git fetch https://git.openjdk.java.net/jfx pull/446/head:pull/446 PR: https://git.openjdk.java.net/jfx/pull/446
Re: RFR: 8238650: Allow to override buildDate with SOURCE_DATE_EPOCH [v2]
> This is a continuation of the [pull request][1] started by @bmwiedemann in > January 2020. After this change is integrated, I can follow up immediately > with additional pull requests that get us much closer to providing fully > reproducible builds. > > Motivation > > The only conclusive way to verify a software package is to reproduce it. > That's the main point of the Linux Foundation article [Preventing Supply > Chain Attacks like SolarWinds][2] by David Wheeler, Director of Open Source > Supply Chain Security. "In the longer term," he writes, "I know of only one > strong countermeasure for this kind of attack: verified reproducible builds." > > It's not enough anymore to trust the person, organization, or company that > publishes a software package. David Wheeler explains, "Assuming a system can > 'never be broken into' is a failing strategy." As I see it, any project that > doesn't yet allow for reproducible builds is on a list of possible attack > vectors. I'd like to get OpenJFX off that list. > > This is a huge undertaking involving the entire open-source community. Just > [Debian, for example][3], has over 30,000 source packages to build in a > reproducible manner. Remarkably, it's almost 96 percent complete. The OpenJFX > package is one of three percent still failing. Our first step towards helping > in this goal is to support the [`SOURCE_DATE_EPOCH` specification][4]. > > Implementation > > When you want to build 30,000 packages in a reproducible manner, > [command-line flags][5] unique to each package aren't so helpful. The > environment variable needs to be set, anyway, for the tools invoked by Gradle > to pick it up. We could allow for a Gradle property in addition to the > environment variable. The Gradle property would override the default current > date and export the environment variable, and the environment variable would > override the command-line property. I think it makes more sense in the > OpenJFX build to support the environment variable directly. > > With these considerations, I added the support just as recommended by the > example for "Java / gradle" on the Reproducible Builds > [`SOURCE_DATE_EPOCH`][6] page. For comparison, the [OpenJDK build][7] does > the reverse, using the *configure* script option `--with-source-date` to > export the `SOURCE_DATE_EPOCH` environment variable. > > [1]: https://github.com/openjdk/jfx/pull/99 > [2]: > https://www.linuxfoundation.org/en/blog/preventing-supply-chain-attacks-like-solarwinds > [3]: https://tests.reproducible-builds.org/debian/reproducible.html > [4]: https://reproducible-builds.org/specs/source-date-epoch/ > [5]: > https://wiki.debian.org/ReproducibleBuilds/StandardEnvironmentVariables#More_detailed_discussion > [6]: https://reproducible-builds.org/docs/source-date-epoch/ > [7]: https://github.com/openjdk/jdk/commit/1a16a4b6 John Neffenger has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains two additional commits since the last revision: - Merge branch 'master' into source-date-epoch - 8238650: Allow to override buildDate with SOURCE_DATE_EPOCH - Changes: - all: https://git.openjdk.java.net/jfx/pull/422/files - new: https://git.openjdk.java.net/jfx/pull/422/files/3ada5a30..b24a86cb Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jfx=422=01 - incr: https://webrevs.openjdk.java.net/?repo=jfx=422=00-01 Stats: 95067 lines in 1313 files changed: 56763 ins; 14002 del; 24302 mod Patch: https://git.openjdk.java.net/jfx/pull/422.diff Fetch: git fetch https://git.openjdk.java.net/jfx pull/422/head:pull/422 PR: https://git.openjdk.java.net/jfx/pull/422
Re: RFR: 8264010: Add Gradle dependency verification [v5]
> This pull request adds dependency verification to the Gradle builds of JavaFX > on Linux, macOS, and Windows. It is the third of three changes that close the > gaps in the JavaFX build security: > > * [JDK-8262236][1]: Configure Gradle checksum verification > * [JDK-8263204][2]: Add Gradle Wrapper Validation Action > * [JDK-8264010][3]: Add Gradle dependency verification > > "Without dependency verification it's easy for an attacker to compromise your > supply chain," warns the [Gradle User Guide][4]. All three changes come from > conference talks by members of the Gradle team, available as [PDF slides][5] > or on YouTube in the following two videos: > > * [Cédric Champeau at Devoxx][6] in November 2019 > * [Louis Jacomet at Jfokus][7] in February 2020 > > "We all run in a crazy-unsafe environment, in a way," says Louis Jacomet at > the end of his talk. These three changes make it just a little less > crazy-unsafe for all of us building JavaFX, regardless of our system, > network, or country. > > [1]: https://bugs.openjdk.java.net/browse/JDK-8262236 > [2]: https://bugs.openjdk.java.net/browse/JDK-8263204 > [3]: https://bugs.openjdk.java.net/browse/JDK-8264010 > > [4]: https://docs.gradle.org/current/userguide/dependency_verification.html > [5]: > https://www.jfokus.se/jfokus20-preso/Protecting-your-organization-against-attacks-via-the-build-system.pdf > [6]: https://youtu.be/GWGNp3a3hpk > [7]: https://youtu.be/bwiafNatsf0 John Neffenger has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains ten additional commits since the last revision: - Add more details to the instructions in the README Add more details to the file 'gradle/README.txt' on how to create and update the dependency verification file for Linux, macOS, Windows, and the internal Oracle builds. - Remove older unused Oracle internal dependencies - Add two more Oracle internal dependencies - Merge branch 'master' into dependency-verification - Add dependencies for internal builds at Oracle - Add dependencies for media and WebKit libraries - Merge branch 'master' into dependency-verification - Add a README file and update 'UPDATING-lucene.txt' - 8264010: Add Gradle dependency verification - Changes: - all: https://git.openjdk.java.net/jfx/pull/437/files - new: https://git.openjdk.java.net/jfx/pull/437/files/b0435b29..75fa032e Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jfx=437=04 - incr: https://webrevs.openjdk.java.net/?repo=jfx=437=03-04 Stats: 3879 lines in 71 files changed: 3202 ins; 428 del; 249 mod Patch: https://git.openjdk.java.net/jfx/pull/437.diff Fetch: git fetch https://git.openjdk.java.net/jfx pull/437/head:pull/437 PR: https://git.openjdk.java.net/jfx/pull/437
Re: RFR: 8264010: Add Gradle dependency verification [v4]
On Tue, 27 Apr 2021 21:19:26 GMT, Kevin Rushforth wrote: > Here are the two additional internal downloads for the new versions of the > gcc-10.3 and vs2019-16.9.3 download Thanks, Kevin. I'll assume those replace the older versions shown in the excerpt below, and I'll remove them. Let me know otherwise. - PR: https://git.openjdk.java.net/jfx/pull/437
Re: Monocle, headless isolated from embedded
On 4/21/21 12:06 PM, Johan Vos wrote: However, I don't think Monocle is ready at this moment to be bundled with the desktop releases. Serious question: Why not? If we do move the Headless Platform out of Monocle, we should consider bringing its VNC Platform subclass along as well. John
Re: RFR: 8264010: Add Gradle dependency verification [v4]
On Sat, 17 Apr 2021 23:17:04 GMT, John Neffenger wrote: >> This pull request adds dependency verification to the Gradle builds of >> JavaFX on Linux, macOS, and Windows. It is the third of three changes that >> close the gaps in the JavaFX build security: >> >> * [JDK-8262236][1]: Configure Gradle checksum verification >> * [JDK-8263204][2]: Add Gradle Wrapper Validation Action >> * [JDK-8264010][3]: Add Gradle dependency verification >> >> "Without dependency verification it's easy for an attacker to compromise >> your supply chain," warns the [Gradle User Guide][4]. All three changes come >> from conference talks by members of the Gradle team, available as [PDF >> slides][5] or on YouTube in the following two videos: >> >> * [Cédric Champeau at Devoxx][6] in November 2019 >> * [Louis Jacomet at Jfokus][7] in February 2020 >> >> "We all run in a crazy-unsafe environment, in a way," says Louis Jacomet at >> the end of his talk. These three changes make it just a little less >> crazy-unsafe for all of us building JavaFX, regardless of our system, >> network, or country. >> >> [1]: https://bugs.openjdk.java.net/browse/JDK-8262236 >> [2]: https://bugs.openjdk.java.net/browse/JDK-8263204 >> [3]: https://bugs.openjdk.java.net/browse/JDK-8264010 >> >> [4]: https://docs.gradle.org/current/userguide/dependency_verification.html >> [5]: >> https://www.jfokus.se/jfokus20-preso/Protecting-your-organization-against-attacks-via-the-build-system.pdf >> [6]: https://youtu.be/GWGNp3a3hpk >> [7]: https://youtu.be/bwiafNatsf0 > > John Neffenger has updated the pull request incrementally with one additional > commit since the last revision: > > Add dependencies for internal builds at Oracle Thanks, Kevin. I added your list to the file and ran just the Linux build followed by tests, all successful: $ gradle -PCONF=Release -PPROMOTED_BUILD_NUMBER=7 \ -PHUDSON_BUILD_NUMBER=101 -PHUDSON_JOB_NAME=jfx \ -PCOMPILE_WEBKIT=true -PCOMPILE_MEDIA=true -PBUILD_LIBAV_STUBS=true \ sdk jmods javadoc test ... BUILD SUCCESSFUL in 4m 26s 224 actionable tasks: 93 executed, 131 up-to-date - PR: https://git.openjdk.java.net/jfx/pull/437
Re: RFR: 8264010: Add Gradle dependency verification [v4]
> This pull request adds dependency verification to the Gradle builds of JavaFX > on Linux, macOS, and Windows. It is the third of three changes that close the > gaps in the JavaFX build security: > > * [JDK-8262236][1]: Configure Gradle checksum verification > * [JDK-8263204][2]: Add Gradle Wrapper Validation Action > * [JDK-8264010][3]: Add Gradle dependency verification > > "Without dependency verification it's easy for an attacker to compromise your > supply chain," warns the [Gradle User Guide][4]. All three changes come from > conference talks by members of the Gradle team, available as [PDF slides][5] > or on YouTube in the following two videos: > > * [Cédric Champeau at Devoxx][6] in November 2019 > * [Louis Jacomet at Jfokus][7] in February 2020 > > "We all run in a crazy-unsafe environment, in a way," says Louis Jacomet at > the end of his talk. These three changes make it just a little less > crazy-unsafe for all of us building JavaFX, regardless of our system, > network, or country. > > [1]: https://bugs.openjdk.java.net/browse/JDK-8262236 > [2]: https://bugs.openjdk.java.net/browse/JDK-8263204 > [3]: https://bugs.openjdk.java.net/browse/JDK-8264010 > > [4]: https://docs.gradle.org/current/userguide/dependency_verification.html > [5]: > https://www.jfokus.se/jfokus20-preso/Protecting-your-organization-against-attacks-via-the-build-system.pdf > [6]: https://youtu.be/GWGNp3a3hpk > [7]: https://youtu.be/bwiafNatsf0 John Neffenger has updated the pull request incrementally with one additional commit since the last revision: Add dependencies for internal builds at Oracle - Changes: - all: https://git.openjdk.java.net/jfx/pull/437/files - new: https://git.openjdk.java.net/jfx/pull/437/files/7609914a..b0435b29 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jfx=437=03 - incr: https://webrevs.openjdk.java.net/?repo=jfx=437=02-03 Stats: 45 lines in 1 file changed: 45 ins; 0 del; 0 mod Patch: https://git.openjdk.java.net/jfx/pull/437.diff Fetch: git fetch https://git.openjdk.java.net/jfx pull/437/head:pull/437 PR: https://git.openjdk.java.net/jfx/pull/437
Re: RFR: 8264449: Enable reproducible builds with SOURCE_DATE_EPOCH [v3]
On Sun, 4 Apr 2021 16:36:27 GMT, John Neffenger wrote: >> This pull request allows for reproducible builds of JavaFX on Linux, macOS, >> and Windows by defining the `SOURCE_DATE_EPOCH` environment variable. For >> example, the following commands create a reproducible build: >> >> >> $ export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) >> $ bash gradlew sdk jmods javadoc >> $ strip-nondeterminism -v -T $SOURCE_DATE_EPOCH build/jmods/*.jmod >> >> >> The three commands: >> >> 1. set the build timestamp to the date of the latest source code change, >> 2. build the JavaFX SDK libraries, JMOD archives, and API documentation, and >> 3. recreate the JMOD files with stable file modification times and ordering. >> >> The third command won't be necessary once Gradle can build the JMOD archives >> or the `jmod` tool itself has the required support. For more information on >> the environment variable, see the [`SOURCE_DATE_EPOCH`][1] page. For more >> information on the command to recreate the JMOD files, see the >> [`strip-nondeterminism`][2] repository. I'd like to propose that we allow >> for reproducible builds in JavaFX 17 and consider making them the default in >> JavaFX 18. >> >> Fixes >> >> There are at least four sources of non-determinism in the JavaFX builds: >> >> 1. Build timestamp >> >> The class `com.sun.javafx.runtime.VersionInfo` in the JavaFX Base module >> stores the time of the build. Furthermore, for builds that don't run on the >> Hudson continuous integration tool, the class adds the build time to the >> system property `javafx.runtime.version`. >> >> 2. Modification times >> >> The JAR, JMOD, and ZIP archives store the modification time of each file. >> >> 3. File ordering >> >> The JAR, JMOD, and ZIP archives store their files in the order returned >> by the file system. The native shared libraries also store their object >> files in the order returned by the file system. Most file systems, though, >> do not guarantee the order of a directory's file listing. >> >> 4. Build path >> >> The class `com.sun.javafx.css.parser.Css2Bin` in the JavaFX Graphics >> module stores the absolute path of its `.css` input file in the >> corresponding `.bss` output file, which is then included in the JavaFX >> Controls module. >> >> This pull request modifies the Gradle and Groovy build files to fix the >> first three sources of non-determinism. A later pull request can modify the >> Java files to fix the fourth. >> >> [1]: https://reproducible-builds.org/docs/source-date-epoch/ >> [2]: https://salsa.debian.org/reproducible-builds/strip-nondeterminism > > John Neffenger has updated the pull request incrementally with one additional > commit since the last revision: > > Include WebKit shared library for Windows > > Enable reproducible builds of the native WebKit shared library for > Windows (jfxwebkit.dll) when SOURCE_DATE_EPOCH is defined. > As an alternative, you could close that PR ... and list both issues in this > PR ... I prefer to keep them separate because they have distinct attributions and descriptions. Few developers will read #422 if we merge them. They also pose distinct questions for reviewers. The implicit question in #422 is, should we define an external environment variable or just remove the build timestamp entirely? The implicit question in this pull request is, given the environment variable, should we also use it as a flag for reproducible builds? Furthermore, which sources of non-determinism should be removed unconditionally, and which should be conditional on the flag? I chose conservative answers to those questions in my changes, but I'm open to other suggestions. - PR: https://git.openjdk.java.net/jfx/pull/446
Re: RFR: 8264449: Enable reproducible builds with SOURCE_DATE_EPOCH [v3]
On Sun, 4 Apr 2021 16:36:27 GMT, John Neffenger wrote: >> This pull request allows for reproducible builds of JavaFX on Linux, macOS, >> and Windows by defining the `SOURCE_DATE_EPOCH` environment variable. For >> example, the following commands create a reproducible build: >> >> >> $ export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) >> $ bash gradlew sdk jmods javadoc >> $ strip-nondeterminism -v -T $SOURCE_DATE_EPOCH build/jmods/*.jmod >> >> >> The three commands: >> >> 1. set the build timestamp to the date of the latest source code change, >> 2. build the JavaFX SDK libraries, JMOD archives, and API documentation, and >> 3. recreate the JMOD files with stable file modification times and ordering. >> >> The third command won't be necessary once Gradle can build the JMOD archives >> or the `jmod` tool itself has the required support. For more information on >> the environment variable, see the [`SOURCE_DATE_EPOCH`][1] page. For more >> information on the command to recreate the JMOD files, see the >> [`strip-nondeterminism`][2] repository. I'd like to propose that we allow >> for reproducible builds in JavaFX 17 and consider making them the default in >> JavaFX 18. >> >> Fixes >> >> There are at least four sources of non-determinism in the JavaFX builds: >> >> 1. Build timestamp >> >> The class `com.sun.javafx.runtime.VersionInfo` in the JavaFX Base module >> stores the time of the build. Furthermore, for builds that don't run on the >> Hudson continuous integration tool, the class adds the build time to the >> system property `javafx.runtime.version`. >> >> 2. Modification times >> >> The JAR, JMOD, and ZIP archives store the modification time of each file. >> >> 3. File ordering >> >> The JAR, JMOD, and ZIP archives store their files in the order returned >> by the file system. The native shared libraries also store their object >> files in the order returned by the file system. Most file systems, though, >> do not guarantee the order of a directory's file listing. >> >> 4. Build path >> >> The class `com.sun.javafx.css.parser.Css2Bin` in the JavaFX Graphics >> module stores the absolute path of its `.css` input file in the >> corresponding `.bss` output file, which is then included in the JavaFX >> Controls module. >> >> This pull request modifies the Gradle and Groovy build files to fix the >> first three sources of non-determinism. A later pull request can modify the >> Java files to fix the fourth. >> >> [1]: https://reproducible-builds.org/docs/source-date-epoch/ >> [2]: https://salsa.debian.org/reproducible-builds/strip-nondeterminism > > John Neffenger has updated the pull request incrementally with one additional > commit since the last revision: > > Include WebKit shared library for Windows > > Enable reproducible builds of the native WebKit shared library for > Windows (jfxwebkit.dll) when SOURCE_DATE_EPOCH is defined. *IEEE Software* just published a good article that describes the problems solved in part by this pull request. The article is called "Reproducible Builds: Increasing the Integrity of Software Supply Chains," by Chris Lamb and Stefano Zacchiroli. It's an easy read of 10 pages, available at the links below: * [IEEE Xplore](https://ieeexplore.ieee.org/document/9403390) - Registration required * [arXiv.org](https://arxiv.org/abs/2104.06020) - Free PDF download - PR: https://git.openjdk.java.net/jfx/pull/446
Re: RFR: 8238650: Allow to override buildDate with SOURCE_DATE_EPOCH
On Sat, 27 Mar 2021 18:49:32 GMT, Bernhard M. Wiedemann wrote: > Sorted output still complies with that API, though. Well, that's true. I hadn't thought of it that way. I have tried twice and failed to [add you as a contributor](https://github.com/openjdk/jfx/pull/422#issuecomment-806317783) to my follow-up pull request. The only option left is the syntax "contributor add J. Duke ", but I'm not sure whether you want the e-mail address in your GitHub profile made public. Let me know whether I should pursue the problem. If you're added, it puts your name and address in the commit log with `Co-authored-by` like this example: commit 8adbc673d095607e8a6109fbb951fa17b9d6caad (tag: 17+3) Author: Alexander Matveev Date: Mon Mar 15 20:57:19 2021 + 8257895: Allow building of JavaFX media libs for Apple Silicon Co-authored-by: Johan Vos Reviewed-by: jvos, kcr It might be useful to have a previous commit if you plan to contribute to the OpenJFX project in the future. - PR: https://git.openjdk.java.net/jfx/pull/99
Re: RFR: 8238650: Allow to override buildDate with SOURCE_DATE_EPOCH
On Wed, 29 Jan 2020 08:58:47 GMT, Bernhard M. Wiedemann wrote: > Allow to override buildDate with `SOURCE_DATE_EPOCH` > in order to make builds reproducible. > See https://reproducible-builds.org/ for why this is good > and https://reproducible-builds.org/specs/source-date-epoch/ > for the definition of this variable. > > This PR was done while working on reproducible builds for openSUSE. By the way, we have reproducible builds on Linux, macOS, and Windows now! (Well, still using `strip-nondeterminism` on the JMOD files, and as long as you build from the same file system path, thanks to `com.sun.javafx.css.parser.Css2Bin`. Sigh. This rabbit hole is deep.) - PR: https://git.openjdk.java.net/jfx/pull/99
Re: RFR: 8238650: Allow to override buildDate with SOURCE_DATE_EPOCH
On Sat, 27 Mar 2021 03:34:11 GMT, Bernhard M. Wiedemann wrote: > Now the interesting question is, if it is possible to change `listFiles` to > always return deterministic output ... Yes, that would be interesting, if `listFiles()` were defined by the Gradle build tool. I think it originates, though, in the Java [java.io.File.listFiles][1] API and is made available to Gradle through the Groovy programming language. The API states, "There is no guarantee that the name strings in the resulting array will appear in any specific order; they are not, in particular, guaranteed to appear in alphabetical order." That API was defined in December 1998 with Java version 1.2, so there's no changing it now. [1]: https://docs.oracle.com/en/java/javase/16/docs/api/java.base/java/io/File.html#listFiles() - PR: https://git.openjdk.java.net/jfx/pull/99
Re: RFR: 8238650: Allow to override buildDate with SOURCE_DATE_EPOCH
On Tue, 9 Mar 2021 03:11:48 GMT, Bernhard M. Wiedemann
wrote:
> When linking shared libraries, ensure that .o files are listed in
> deterministic order - i.e. `sort(glob(*.c))` and check if individual .c or .o
> files vary between builds.
Wow, that tip was priceless! Thank you, Bernhard. It took forever to figure out
*where* to add it, but simply adding `.sort()` to the list of object files for
the linker did the trick:
index 8f1e04a833..9181ccc7fb 100644
--- a/buildSrc/src/main/groovy/com/sun/javafx/gradle/LinkTask.groovy
+++ b/buildSrc/src/main/groovy/com/sun/javafx/gradle/LinkTask.groovy
@@ -49,7 +49,7 @@ class LinkTask extends DefaultTask {
args("$lib");
}
// Exclude parfait files (.bc)
-args(objectDir.listFiles().findAll{
!it.getAbsolutePath().endsWith(".bc") });
+args(objectDir.listFiles().sort().findAll{
!it.getAbsolutePath().endsWith(".bc") });
if (project.IS_WINDOWS) {
args("/out:$lib");
} else {
We have reproducible builds! (Well, we have them on Linux, at least, and only
after running `strip-nondeterminism` on the JMOD files, but that's still major
progress.) I'll follow up with a pull request after checking whether I can get
reproducible builds on macOS and Windows, too.
-
PR: https://git.openjdk.java.net/jfx/pull/99
Re: RFR: 8238650: Allow to override buildDate with SOURCE_DATE_EPOCH
On Wed, 15 Apr 2020 02:35:44 GMT, Bernhard M. Wiedemann wrote: > I am currently low on time and my Java-foo is not that great, so I was hoping > that someone would pick it up. @bmwiedemann Thank you for getting this important change started. If you can finish this pull request, I'll follow up with some additional changes to support reproducible builds. Otherwise, if you're still short on time, I can pick up where you left off. So far I managed to create reproducible Java artifacts — all but the native shared libraries (and the `javafx.graphics.jmod` file that includes them). - PR: https://git.openjdk.java.net/jfx/pull/99
Re: RFR: 8264010: Add Gradle dependency verification [v3]
On Wed, 14 Apr 2021 04:32:29 GMT, John Neffenger wrote:
>> This pull request adds dependency verification to the Gradle builds of
>> JavaFX on Linux, macOS, and Windows. It is the third of three changes that
>> close the gaps in the JavaFX build security:
>>
>> * [JDK-8262236][1]: Configure Gradle checksum verification
>> * [JDK-8263204][2]: Add Gradle Wrapper Validation Action
>> * [JDK-8264010][3]: Add Gradle dependency verification
>>
>> "Without dependency verification it's easy for an attacker to compromise
>> your supply chain," warns the [Gradle User Guide][4]. All three changes come
>> from conference talks by members of the Gradle team, available as [PDF
>> slides][5] or on YouTube in the following two videos:
>>
>> * [Cédric Champeau at Devoxx][6] in November 2019
>> * [Louis Jacomet at Jfokus][7] in February 2020
>>
>> "We all run in a crazy-unsafe environment, in a way," says Louis Jacomet at
>> the end of his talk. These three changes make it just a little less
>> crazy-unsafe for all of us building JavaFX, regardless of our system,
>> network, or country.
>>
>> [1]: https://bugs.openjdk.java.net/browse/JDK-8262236
>> [2]: https://bugs.openjdk.java.net/browse/JDK-8263204
>> [3]: https://bugs.openjdk.java.net/browse/JDK-8264010
>>
>> [4]: https://docs.gradle.org/current/userguide/dependency_verification.html
>> [5]:
>> https://www.jfokus.se/jfokus20-preso/Protecting-your-organization-against-attacks-via-the-build-system.pdf
>> [6]: https://youtu.be/GWGNp3a3hpk
>> [7]: https://youtu.be/bwiafNatsf0
>
> John Neffenger has updated the pull request with a new target base due to a
> merge or a rebase. The incremental webrev excludes the unrelated changes
> brought in by the merge/rebase. The pull request contains four additional
> commits since the last revision:
>
> - Add dependencies for media and WebKit libraries
> - Merge branch 'master' into dependency-verification
> - Add a README file and update 'UPDATING-lucene.txt'
> - 8264010: Add Gradle dependency verification
There's an odd thing happening with some of the artifact names in the
dependency file. For example:
Gradle creates the artifact name from the component name and version, separated
by a hyphen. Because the version attribute is empty, we get an artifact name
ending in `-.tar.gz`. It should be `ffmpeg-3.3.3.tar.gz`, but Gradle sees
`-3.3.3` as part of the name. This particular name originates from:
dependencies {
if (IS_BUILD_LIBAV_STUBS) {
media name: "libav-9.14", ext: "tar.gz"
media name: "libav-11.4", ext: "tar.gz"
media name: "libav-12.1", ext: "tar.gz"
media name: "ffmpeg-3.3.3", ext: "tar.gz"
media name: "ffmpeg-4.0.2", ext: "tar.gz"
}
compile project(":base")
compile project(":graphics")
}
Other places in the build file seem to depend on having the version as part of
the name string, and my initial attempt to separate the two failed. The build
works with the names as they are, and these are in fact the names of the files
in the Gradle cache. Some day, though, we might want to have better component
elements in the dependency file, seeing as it functions also like a software
bill of materials.
-
PR: https://git.openjdk.java.net/jfx/pull/437
Re: RFR: 8264010: Add Gradle dependency verification [v3]
> This pull request adds dependency verification to the Gradle builds of JavaFX > on Linux, macOS, and Windows. It is the third of three changes that close the > gaps in the JavaFX build security: > > * [JDK-8262236][1]: Configure Gradle checksum verification > * [JDK-8263204][2]: Add Gradle Wrapper Validation Action > * [JDK-8264010][3]: Add Gradle dependency verification > > "Without dependency verification it's easy for an attacker to compromise your > supply chain," warns the [Gradle User Guide][4]. All three changes come from > conference talks by members of the Gradle team, available as [PDF slides][5] > or on YouTube in the following two videos: > > * [Cédric Champeau at Devoxx][6] in November 2019 > * [Louis Jacomet at Jfokus][7] in February 2020 > > "We all run in a crazy-unsafe environment, in a way," says Louis Jacomet at > the end of his talk. These three changes make it just a little less > crazy-unsafe for all of us building JavaFX, regardless of our system, > network, or country. > > [1]: https://bugs.openjdk.java.net/browse/JDK-8262236 > [2]: https://bugs.openjdk.java.net/browse/JDK-8263204 > [3]: https://bugs.openjdk.java.net/browse/JDK-8264010 > > [4]: https://docs.gradle.org/current/userguide/dependency_verification.html > [5]: > https://www.jfokus.se/jfokus20-preso/Protecting-your-organization-against-attacks-via-the-build-system.pdf > [6]: https://youtu.be/GWGNp3a3hpk > [7]: https://youtu.be/bwiafNatsf0 John Neffenger has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains four additional commits since the last revision: - Add dependencies for media and WebKit libraries - Merge branch 'master' into dependency-verification - Add a README file and update 'UPDATING-lucene.txt' - 8264010: Add Gradle dependency verification - Changes: - all: https://git.openjdk.java.net/jfx/pull/437/files - new: https://git.openjdk.java.net/jfx/pull/437/files/c7ac7f62..7609914a Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jfx=437=02 - incr: https://webrevs.openjdk.java.net/?repo=jfx=437=01-02 Stats: 87251 lines in 1199 files changed: 49762 ins; 13569 del; 23920 mod Patch: https://git.openjdk.java.net/jfx/pull/437.diff Fetch: git fetch https://git.openjdk.java.net/jfx pull/437/head:pull/437 PR: https://git.openjdk.java.net/jfx/pull/437
Re: RFR: 8264010: Add Gradle dependency verification [v2]
On Wed, 24 Mar 2021 19:45:55 GMT, John Neffenger wrote: >> This pull request adds dependency verification to the Gradle builds of >> JavaFX on Linux, macOS, and Windows. It is the third of three changes that >> close the gaps in the JavaFX build security: >> >> * [JDK-8262236][1]: Configure Gradle checksum verification >> * [JDK-8263204][2]: Add Gradle Wrapper Validation Action >> * [JDK-8264010][3]: Add Gradle dependency verification >> >> "Without dependency verification it's easy for an attacker to compromise >> your supply chain," warns the [Gradle User Guide][4]. All three changes come >> from conference talks by members of the Gradle team, available as [PDF >> slides][5] or on YouTube in the following two videos: >> >> * [Cédric Champeau at Devoxx][6] in November 2019 >> * [Louis Jacomet at Jfokus][7] in February 2020 >> >> "We all run in a crazy-unsafe environment, in a way," says Louis Jacomet at >> the end of his talk. These three changes make it just a little less >> crazy-unsafe for all of us building JavaFX, regardless of our system, >> network, or country. >> >> [1]: https://bugs.openjdk.java.net/browse/JDK-8262236 >> [2]: https://bugs.openjdk.java.net/browse/JDK-8263204 >> [3]: https://bugs.openjdk.java.net/browse/JDK-8264010 >> >> [4]: https://docs.gradle.org/current/userguide/dependency_verification.html >> [5]: >> https://www.jfokus.se/jfokus20-preso/Protecting-your-organization-against-attacks-via-the-build-system.pdf >> [6]: https://youtu.be/GWGNp3a3hpk >> [7]: https://youtu.be/bwiafNatsf0 > > John Neffenger has updated the pull request incrementally with one additional > commit since the last revision: > > Add a README file and update 'UPDATING-lucene.txt' Thanks for the list, Kevin. Pull request #450 was closed in favor of #456, which was integrated. I'll build with the media and WebKit libraries so Gradle can add their dependencies, and I'll update this pull request with the new verification file. - PR: https://git.openjdk.java.net/jfx/pull/437
Re: RFR: 8264449: Enable reproducible builds with SOURCE_DATE_EPOCH [v3]
On Sun, 4 Apr 2021 16:36:27 GMT, John Neffenger wrote: >> This pull request allows for reproducible builds of JavaFX on Linux, macOS, >> and Windows by defining the `SOURCE_DATE_EPOCH` environment variable. For >> example, the following commands create a reproducible build: >> >> >> $ export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) >> $ bash gradlew sdk jmods javadoc >> $ strip-nondeterminism -v -T $SOURCE_DATE_EPOCH build/jmods/*.jmod >> >> >> The three commands: >> >> 1. set the build timestamp to the date of the latest source code change, >> 2. build the JavaFX SDK libraries, JMOD archives, and API documentation, and >> 3. recreate the JMOD files with stable file modification times and ordering. >> >> The third command won't be necessary once Gradle can build the JMOD archives >> or the `jmod` tool itself has the required support. For more information on >> the environment variable, see the [`SOURCE_DATE_EPOCH`][1] page. For more >> information on the command to recreate the JMOD files, see the >> [`strip-nondeterminism`][2] repository. I'd like to propose that we allow >> for reproducible builds in JavaFX 17 and consider making them the default in >> JavaFX 18. >> >> Fixes >> >> There are at least four sources of non-determinism in the JavaFX builds: >> >> 1. Build timestamp >> >> The class `com.sun.javafx.runtime.VersionInfo` in the JavaFX Base module >> stores the time of the build. Furthermore, for builds that don't run on the >> Hudson continuous integration tool, the class adds the build time to the >> system property `javafx.runtime.version`. >> >> 2. Modification times >> >> The JAR, JMOD, and ZIP archives store the modification time of each file. >> >> 3. File ordering >> >> The JAR, JMOD, and ZIP archives store their files in the order returned >> by the file system. The native shared libraries also store their object >> files in the order returned by the file system. Most file systems, though, >> do not guarantee the order of a directory's file listing. >> >> 4. Build path >> >> The class `com.sun.javafx.css.parser.Css2Bin` in the JavaFX Graphics >> module stores the absolute path of its `.css` input file in the >> corresponding `.bss` output file, which is then included in the JavaFX >> Controls module. >> >> This pull request modifies the Gradle and Groovy build files to fix the >> first three sources of non-determinism. A later pull request can modify the >> Java files to fix the fourth. >> >> [1]: https://reproducible-builds.org/docs/source-date-epoch/ >> [2]: https://salsa.debian.org/reproducible-builds/strip-nondeterminism > > John Neffenger has updated the pull request incrementally with one additional > commit since the last revision: > > Include WebKit shared library for Windows > > Enable reproducible builds of the native WebKit shared library for > Windows (jfxwebkit.dll) when SOURCE_DATE_EPOCH is defined. Thank you, Alexander and Arun, for looking over the Makefiles and verifying the builds. > I'd like to take a week to get feedback and mull over the code changes ... After realizing how little I knew about Gradle, I did some research last week and took four courses on Groovy, Gradle, and the Gradle Java Plugin. Although I now see a couple things I could change, I'm still happy with the commits as they are. I welcome any comments or questions. Once #422 and this pull request are integrated, I can follow up with changes to make the `all` task reproducible (fixing `javafx-sdk-17.zip`) and to remove the build path from the `.bss` files (fix number 4 in the first comment). - PR: https://git.openjdk.java.net/jfx/pull/446
Re: RFR: 8264449: Enable reproducible builds with SOURCE_DATE_EPOCH [v2]
On Sat, 3 Apr 2021 01:49:47 GMT, John Neffenger wrote: >>> Silly question, is the difference with Windows just the nature of the >>> native support on each platform (Unix based vs Windows) and libraries used >>> as part of that? >> >> That's a good question. I'm hoping the answer is no. So far, the difference >> with Windows is the linker. The other systems use the GNU linker where it's >> enough to sort the input object files. The Microsoft linker, though, creates >> reproducible builds with the flag `/experimental:deterministic` (or >> `-experimental:deterministic`). So on Windows, we just need to figure out >> where to put the flag. I'm finding that to be a challenge for the WebKit >> library. > >> /contributor add ... > > @bmwiedemann I added you as co-author of pull request #422, which is my > continuation of your pull request #99 from last year. This pull request #446 > temporarily includes our co-authored commit, but those changes will disappear > when #422 is integrated and I merge the master branch into this one. Sorry > for the confusion. > It would be better to add the flag in `libxml/CMakeLists.txt` or in > `build.gradle` via `cmakeArgs`. Thank you, Arun, for that helpful suggestion. I updated this pull request for the WebKit library on Windows and ran my tests again, as described below. I saved the output of the first build in the `build1.log` file and the `build1` directory: $ export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) $ bash gradlew -PCONF=Release -PPROMOTED_BUILD_NUMBER=5 \ -PHUDSON_BUILD_NUMBER=101 -PHUDSON_JOB_NAME=jfx -PCOMPILE_WEBKIT=true \ -PCOMPILE_MEDIA=true -PBUILD_LIBAV_STUBS=true sdk jmods javadoc \ | tee build1.log $ mv build build1 Then I ran another clean build, this time saving the output in the `build2.log` file and the `build2` directory: $ bash gradlew cleanAll $ bash gradlew -PCONF=Release -PPROMOTED_BUILD_NUMBER=5 \ -PHUDSON_BUILD_NUMBER=101 -PHUDSON_JOB_NAME=jfx -PCOMPILE_WEBKIT=true \ -PCOMPILE_MEDIA=true -PBUILD_LIBAV_STUBS=true sdk jmods javadoc \ | tee build2.log $ mv build build2 I copied the builds to my Linux workstation and ran the following command on the JMOD files for each pair of builds: $ strip-nondeterminism -v -T $SOURCE_DATE_EPOCH build?/jmods/*.jmod Normalizing build1/jmods/javafx.base.jmod Normalizing build1/jmods/javafx.controls.jmod Normalizing build1/jmods/javafx.fxml.jmod Normalizing build1/jmods/javafx.graphics.jmod Normalizing build1/jmods/javafx.media.jmod Normalizing build1/jmods/javafx.swing.jmod Normalizing build1/jmods/javafx.web.jmod Normalizing build2/jmods/javafx.base.jmod Normalizing build2/jmods/javafx.controls.jmod Normalizing build2/jmods/javafx.fxml.jmod Normalizing build2/jmods/javafx.graphics.jmod Normalizing build2/jmods/javafx.media.jmod Normalizing build2/jmods/javafx.swing.jmod Normalizing build2/jmods/javafx.web.jmod After normalizing the JMOD files, the output of the two builds on each of the three operating systems is identical: $ diff -qr linux/build1 linux/build2 $ diff -qr macos/build1 macos/build2 $ diff -qr win10/build1 win10/build2 $ The following commands show the count of identical output files for each system: $ diff -sr linux/build1 linux/build2 | wc -l 10676 $ diff -sr macos/build1 macos/build2 | wc -l 10680 $ diff -sr win10/build1 win10/build2 | wc -l 10633 Considering the time it takes to run all these builds, I'd like to take a week to get feedback and mull over the code changes before anyone spends a lot of time testing. I wish the changes for Windows were not so scattered about, for example. - PR: https://git.openjdk.java.net/jfx/pull/446
Re: RFR: 8264449: Enable reproducible builds with SOURCE_DATE_EPOCH [v3]
> This pull request allows for reproducible builds of JavaFX on Linux, macOS, > and Windows by defining the `SOURCE_DATE_EPOCH` environment variable. For > example, the following commands create a reproducible build: > > $ export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) > $ bash gradlew sdk jmods javadoc > $ strip-nondeterminism -v -T $SOURCE_DATE_EPOCH build/jmods/*.jmod > > The three commands: > > 1. set the build timestamp to the date of the latest source code change, > 2. build the JavaFX SDK libraries, JMOD archives, and API documentation, and > 3. recreate the JMOD files with stable file modification times and ordering. > > The third command won't be necessary once Gradle can build the JMOD archives > or the `jmod` tool itself has the required support. For more information on > the environment variable, see the [`SOURCE_DATE_EPOCH`][1] page. For more > information on the command to recreate the JMOD files, see the > [`strip-nondeterminism`][2] repository. I'd like to propose that we allow for > reproducible builds in JavaFX 17 and consider making them the default in > JavaFX 18. > > Fixes > > There are at least four sources of non-determinism in the JavaFX builds: > > 1. Build timestamp > > The class `com.sun.javafx.runtime.VersionInfo` in the JavaFX Base module > stores the time of the build. Furthermore, for builds that don't run on the > Hudson continuous integration tool, the class adds the build time to the > system property `javafx.runtime.version`. > > 2. Modification times > > The JAR, JMOD, and ZIP archives store the modification time of each file. > > 3. File ordering > > The JAR, JMOD, and ZIP archives store their files in the order returned > by the file system. The native shared libraries also store their object files > in the order returned by the file system. Most file systems, though, do not > guarantee the order of a directory's file listing. > > 4. Build path > > The class `com.sun.javafx.css.parser.Css2Bin` in the JavaFX Graphics > module stores the absolute path of its `.css` input file in the corresponding > `.bss` output file, which is then included in the JavaFX Controls module. > > This pull request modifies the Gradle and Groovy build files to fix the first > three sources of non-determinism. A later pull request can modify the Java > files to fix the fourth. > > [1]: https://reproducible-builds.org/docs/source-date-epoch/ > [2]: https://salsa.debian.org/reproducible-builds/strip-nondeterminism John Neffenger has updated the pull request incrementally with one additional commit since the last revision: Include WebKit shared library for Windows Enable reproducible builds of the native WebKit shared library for Windows (jfxwebkit.dll) when SOURCE_DATE_EPOCH is defined. - Changes: - all: https://git.openjdk.java.net/jfx/pull/446/files - new: https://git.openjdk.java.net/jfx/pull/446/files/afeb3fe0..cb8510c6 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jfx=446=02 - incr: https://webrevs.openjdk.java.net/?repo=jfx=446=01-02 Stats: 9 lines in 2 files changed: 9 ins; 0 del; 0 mod Patch: https://git.openjdk.java.net/jfx/pull/446.diff Fetch: git fetch https://git.openjdk.java.net/jfx pull/446/head:pull/446 PR: https://git.openjdk.java.net/jfx/pull/446
Re: RFR: 8264449: Enable reproducible builds with SOURCE_DATE_EPOCH [v2]
On Fri, 2 Apr 2021 17:45:38 GMT, John Neffenger wrote: >> Media makefiles look fine. > >> Silly question, is the difference with Windows just the nature of the native >> support on each platform (Unix based vs Windows) and libraries used as part >> of that? > > That's a good question. I'm hoping the answer is no. So far, the difference > with Windows is the linker. The other systems use the GNU linker where it's > enough to sort the input object files. The Microsoft linker, though, creates > reproducible builds with the flag `/experimental:deterministic` (or > `-experimental:deterministic`). So on Windows, we just need to figure out > where to put the flag. I'm finding that to be a challenge for the WebKit > library. > /contributor add ... @bmwiedemann I added you as co-author of pull request #422, which is my continuation of your pull request #99 from last year. This pull request #446 temporarily includes our co-authored commit, but those changes will disappear when #422 is integrated and I merge the master branch into this one. Sorry for the confusion. - PR: https://git.openjdk.java.net/jfx/pull/446
Re: RFR: 8264449: Enable reproducible builds with SOURCE_DATE_EPOCH [v2]
On Thu, 1 Apr 2021 23:21:55 GMT, Alexander Matveev wrote: >> John Neffenger has updated the pull request incrementally with one >> additional commit since the last revision: >> >> Include media shared libraries for Windows >> >> Enable reproducible builds of the native media shared libraries for >> Windows when SOURCE_DATE_EPOCH is defined. The libraries are: >> >> fxplugins.dll >> glib-lite.dll >> gstreamer-lite.dll >> jfxmedia.dll > > Media makefiles look fine. > Silly question, is the difference with Windows just the nature of the native > support on each platform (Unix based vs Windows) and libraries used as part > of that? That's a good question. I'm hoping the answer is no. So far, the difference with Windows is the linker. The other systems use the GNU linker where it's enough to sort the input object files. The Microsoft linker, though, creates reproducible builds with the flag `/experimental:deterministic` (or `-experimental:deterministic`). So on Windows, we just need to figure out where to put the flag. I'm finding that to be a challenge for the WebKit library. - PR: https://git.openjdk.java.net/jfx/pull/446
Re: RFR: 8264449: Enable reproducible builds with SOURCE_DATE_EPOCH
On Thu, 1 Apr 2021 19:38:18 GMT, Bernhard M. Wiedemann wrote: > IMHO, it would make sense to merge any clear improvement of the status-quo > and debug+fix more in a later PR. I agree that we will have to draw the line somewhere, but right now I'm down to just one file on one operating system out of 10,633 files in the build output! (It's just `jfxwebkit.dll` on Windows.) It would be a shame to give up now. I would, though, like to postpone tests of static builds and builds for Android and iOS. My priorities for this pull request are: 1. Linux distributions building JavaFX 2. Developers building JavaFX from the repository source with the build defaults 3. Oracle, Gluon, and BellSoft building JavaFX for their customers and for downloading I would be happy to satisfy just the first group, but we might be close to getting all three. - PR: https://git.openjdk.java.net/jfx/pull/446
Re: RFR: 8264449: Enable reproducible builds with SOURCE_DATE_EPOCH [v2]
> This pull request allows for reproducible builds of JavaFX on Linux, macOS, > and Windows by defining the `SOURCE_DATE_EPOCH` environment variable. For > example, the following commands create a reproducible build: > > $ export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) > $ bash gradlew sdk jmods javadoc > $ strip-nondeterminism -v -T $SOURCE_DATE_EPOCH build/jmods/*.jmod > > The three commands: > > 1. set the build timestamp to the date of the latest source code change, > 2. build the JavaFX SDK libraries, JMOD archives, and API documentation, and > 3. recreate the JMOD files with stable file modification times and ordering. > > The third command won't be necessary once Gradle can build the JMOD archives > or the `jmod` tool itself has the required support. For more information on > the environment variable, see the [`SOURCE_DATE_EPOCH`][1] page. For more > information on the command to recreate the JMOD files, see the > [`strip-nondeterminism`][2] repository. I'd like to propose that we allow for > reproducible builds in JavaFX 17 and consider making them the default in > JavaFX 18. > > Fixes > > There are at least four sources of non-determinism in the JavaFX builds: > > 1. Build timestamp > > The class `com.sun.javafx.runtime.VersionInfo` in the JavaFX Base module > stores the time of the build. Furthermore, for builds that don't run on the > Hudson continuous integration tool, the class adds the build time to the > system property `javafx.runtime.version`. > > 2. Modification times > > The JAR, JMOD, and ZIP archives store the modification time of each file. > > 3. File ordering > > The JAR, JMOD, and ZIP archives store their files in the order returned > by the file system. The native shared libraries also store their object files > in the order returned by the file system. Most file systems, though, do not > guarantee the order of a directory's file listing. > > 4. Build path > > The class `com.sun.javafx.css.parser.Css2Bin` in the JavaFX Graphics > module stores the absolute path of its `.css` input file in the corresponding > `.bss` output file, which is then included in the JavaFX Controls module. > > This pull request modifies the Gradle and Groovy build files to fix the first > three sources of non-determinism. A later pull request can modify the Java > files to fix the fourth. > > [1]: https://reproducible-builds.org/docs/source-date-epoch/ > [2]: https://salsa.debian.org/reproducible-builds/strip-nondeterminism John Neffenger has updated the pull request incrementally with one additional commit since the last revision: Include media shared libraries for Windows Enable reproducible builds of the native media shared libraries for Windows when SOURCE_DATE_EPOCH is defined. The libraries are: fxplugins.dll glib-lite.dll gstreamer-lite.dll jfxmedia.dll - Changes: - all: https://git.openjdk.java.net/jfx/pull/446/files - new: https://git.openjdk.java.net/jfx/pull/446/files/ad46dc38..afeb3fe0 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jfx=446=01 - incr: https://webrevs.openjdk.java.net/?repo=jfx=446=00-01 Stats: 25 lines in 4 files changed: 22 ins; 0 del; 3 mod Patch: https://git.openjdk.java.net/jfx/pull/446.diff Fetch: git fetch https://git.openjdk.java.net/jfx pull/446/head:pull/446 PR: https://git.openjdk.java.net/jfx/pull/446
Re: RFR: 8264449: Enable reproducible builds with SOURCE_DATE_EPOCH
On Wed, 31 Mar 2021 13:59:12 GMT, Kevin Rushforth wrote: >>> I think there might be a Skara bug. >> >> No, no bug. Sorry about that. I just don't know how GitHub works. >> :frowning_face: The pre-submit tests ran two days ago when I pushed the >> branch to GitHub, so by the time I submitted the pull request, they had >> finished long ago. > > I recommend trying this with the following gradle flags, to match the > settings for production builds: > > -PCONF=Release -PPROMOTED_BUILD_NUMBER=NNN -PHUDSON_BUILD_NUMBER=MMM > -PHUDSON_JOB_NAME=jfx -PCOMPILE_WEBKIT=true -PCOMPILE_MEDIA=true > -PBUILD_LIBAV_STUBS=true > > where `NNN` is the promoted build number that is being built (usually taken > from the repo tag) and `MMM` is the CI build number. You can just pick any > two positive numbers for your test builds. > > Note that this will build the native media libraries and the native webkit > library. @kevinrushforth I found the Makefiles building the media libraries for Windows. I can fix those, but there's also `libxml` and `libxslt` that invoke the Windows linker here: modules/javafx.web/src/main/native/Source/ThirdParty/libxml/src/win32/Makefile.msvc modules/javafx.web/src/main/native/Source/ThirdParty/libxslt/src/win32/Makefile.msvc They both state at the top, "There should never be a need to modify anything below this line." Are those files directly from their upstream sources? If so, what's our policy on patching them while waiting for fixes? - PR: https://git.openjdk.java.net/jfx/pull/446
Re: RFR: 8264449: Enable reproducible builds with SOURCE_DATE_EPOCH
On Wed, 31 Mar 2021 23:08:38 GMT, John Neffenger wrote: >> I recommend trying this with the following gradle flags, to match the >> settings for production builds: >> >> -PCONF=Release -PPROMOTED_BUILD_NUMBER=NNN -PHUDSON_BUILD_NUMBER=MMM >> -PHUDSON_JOB_NAME=jfx -PCOMPILE_WEBKIT=true -PCOMPILE_MEDIA=true >> -PBUILD_LIBAV_STUBS=true >> >> where `NNN` is the promoted build number that is being built (usually taken >> from the repo tag) and `MMM` is the CI build number. You can just pick any >> two positive numbers for your test builds. >> >> Note that this will build the native media libraries and the native webkit >> library. > >> I recommend trying this with the following gradle flags, to match the >> settings for production builds: > > Thanks, Kevin. Good news so far. I'm posting the Linux results while I run > the macOS and Windows builds. > > Linux > > I ran the following commands twice, moving the `build` directory to `build1` > and then `build2` to save their output: > > $ bash gradlew clean > $ bash gradlew -PCONF=Release -PPROMOTED_BUILD_NUMBER=5 \ > -PHUDSON_BUILD_NUMBER=101 -PHUDSON_JOB_NAME=jfx \ > -PCOMPILE_WEBKIT=true -PCOMPILE_MEDIA=true \ > -PBUILD_LIBAV_STUBS=true sdk jmods javadoc > > Then I changed the Hudson job number with `-PHUDSON_BUILD_NUMBER=102`, ran > the build a third time, and moved `build` to `build3`. I also ran > `strip-nondeterminism` as shown in the first comment of this pull request. > > The first result is as hoped, and the second result is as expected: > > $ diff -qr build1 build2 > $ diff -qr build2 build3 > Files build2/jmods/javafx.base.jmod >and build3/jmods/javafx.base.jmod >differ > Files > build2/modular-sdk/modules/javafx.base/com/sun/javafx/runtime/VersionInfo.class > and > build3/modular-sdk/modules/javafx.base/com/sun/javafx/runtime/VersionInfo.class > differ > Files > build2/modular-sdk/modules_src/javafx.base/com/sun/javafx/runtime/VersionInfo.java > and > build3/modular-sdk/modules_src/javafx.base/com/sun/javafx/runtime/VersionInfo.java > differ > Files build2/publications/javafx.base-linux.jar > and build3/publications/javafx.base-linux.jar > differ > Files build2/sdk/lib/javafx.base.jar > and build3/sdk/lib/javafx.base.jar > differ > Files build2/sdk/lib/src.zip > and build3/sdk/lib/src.zip > differ > > You have to appreciate the irony of adding all this information to the build > — the time, the path, even the job number — so that we can uniquely identify > a build by its output. Meanwhile, if we didn't add this information, our > builds could be uniquely identified by a single Git tag. There's good news and bad news. Good news first. macOS The two comparisons of the three builds on macOS were similar to those on Linux: $ diff -qr build1 build2 $ diff -qr build2 build3 Files build2/jmods/javafx.base.jmod and build3/jmods/javafx.base.jmod differ Files build2/modular-sdk/modules/javafx.base/com/sun/javafx/runtime/VersionInfo.class and build3/modular-sdk/modules/javafx.base/com/sun/javafx/runtime/VersionInfo.class differ Files build2/modular-sdk/modules_src/javafx.base/com/sun/javafx/runtime/VersionInfo.java and build3/modular-sdk/modules_src/javafx.base/com/sun/javafx/runtime/VersionInfo.java differ Files build2/publications/javafx.base-mac.jar and build3/publications/javafx.base-mac.jar differ Files build2/sdk/lib/javafx.base.jar and build3/sdk/lib/javafx.base.jar differ Files build2/sdk/lib/src.zip and build3/sdk/lib/src.zip differ Windows The Windows build, on the other hand, failed to produce identical copies of the media and WebKit shared libraries: $ diff -qr build1 build2 Files build1/jmods/javafx.media.jmod and build2/jmods/javafx.media.jmod differ Files build1/jmods/javafx.web.jmod and build2/jmods/javafx.web.jmod differ Files build1/modular-sdk/modules_libs/javafx.media/fxplugins.dll and build2/modular-sdk/modules_libs/javafx.media/fxplugins.dll differ Files build1/modular-sdk/modules_libs/javafx.media/glib-lite.dll and build2/modular-sdk/modules_libs/javafx.media/glib-lite.dll differ Files build1/modular-sdk/modules_libs/javafx.media/gstreamer-lite.dll and build2/modular-sdk/modules_libs/javafx.media/gstreamer-lite.dll differ Files build1/modular-sdk/modules_libs/javafx.media/jfxmedia.dll and build2/modular-sdk/modules_libs/javafx.media/jfxmedia.dll differ Files build1/modular-sdk/modules_libs/javafx.web/jfxwebkit.dll and build2/modular-sdk/modules_libs/javafx.web/jfxwebkit.dll differ Files build1/publications/javafx.media-win.jar and build2/publications/javafx.media-win.jar differ Files build1/publications/javafx.web-win.jar and build2/publications/jav
Re: RFR: 8264449: Enable reproducible builds with SOURCE_DATE_EPOCH
On Wed, 31 Mar 2021 13:59:12 GMT, Kevin Rushforth wrote: > I recommend trying this with the following gradle flags, to match the > settings for production builds: Thanks, Kevin. Good news so far. I'm posting the Linux results while I run the macOS and Windows builds. Linux I ran the following commands twice, moving the `build` directory to `build1` and then `build2` to save their output: $ bash gradlew clean $ bash gradlew -PCONF=Release -PPROMOTED_BUILD_NUMBER=5 \ -PHUDSON_BUILD_NUMBER=101 -PHUDSON_JOB_NAME=jfx \ -PCOMPILE_WEBKIT=true -PCOMPILE_MEDIA=true \ -PBUILD_LIBAV_STUBS=true sdk jmods javadoc Then I changed the Hudson job number with `-PHUDSON_BUILD_NUMBER=102`, ran the build a third time, and moved `build` to `build3`. I also ran `strip-nondeterminism` as shown in the first comment of this pull request. The first result is as hoped, and the second result is as expected: $ diff -qr build1 build2 $ diff -qr build2 build3 Files build2/jmods/javafx.base.jmod and build3/jmods/javafx.base.jmod differ Files build2/modular-sdk/modules/javafx.base/com/sun/javafx/runtime/VersionInfo.class and build3/modular-sdk/modules/javafx.base/com/sun/javafx/runtime/VersionInfo.class differ Files build2/modular-sdk/modules_src/javafx.base/com/sun/javafx/runtime/VersionInfo.java and build3/modular-sdk/modules_src/javafx.base/com/sun/javafx/runtime/VersionInfo.java differ Files build2/publications/javafx.base-linux.jar and build3/publications/javafx.base-linux.jar differ Files build2/sdk/lib/javafx.base.jar and build3/sdk/lib/javafx.base.jar differ Files build2/sdk/lib/src.zip and build3/sdk/lib/src.zip differ You have to appreciate the irony of adding all this information to the build — the time, the path, even the job number — so that we can uniquely identify a build by its output. Meanwhile, if we didn't add this information, our builds could be uniquely identified by a single Git tag. - PR: https://git.openjdk.java.net/jfx/pull/446
Re: RFR: 8242508: Upgrade to Visual Studio 2019 version 16.5.3
On Thu, 25 Mar 2021 22:58:57 GMT, Kevin Rushforth wrote: > Build fails on my system with `FAIL: WINSDK_DIR not defined` @palexdev I hit that error, too. You may have already found this, but you can work around the problem by using the sample [`windows_tools.properties`][1] file on the Building OpenJFX page (click "Expand source"). You just have to adjust the `WINDOWS_VS_PATH` and remember to copy the file to `build/windows_tools.properties` after running `gradle clean` and before running a build. [1]: https://wiki.openjdk.java.net/display/OpenJFX/Building+OpenJFX#BuildingOpenJFX-Missingpathsissue - PR: https://git.openjdk.java.net/jfx/pull/212
Re: RFR: 8264449: Enable reproducible builds with SOURCE_DATE_EPOCH
On Tue, 30 Mar 2021 16:30:25 GMT, John Neffenger wrote: > I think there might be a Skara bug. No, no bug. Sorry about that. I just don't know how GitHub works. :frowning_face: The pre-submit tests ran two days ago when I pushed the branch to GitHub, so by the time I submitted the pull request, they had finished long ago. - PR: https://git.openjdk.java.net/jfx/pull/446
Re: E-mail addresses at openjdk.org
On 3/25/21 4:12 AM, Kevin Rushforth wrote: Until then, adding the email address as an unverified additional email address in your GitHub profile is the way to go. And the first GitHub user to claim the address wins! There are a large number of unclaimed commits in our repository that are up for grabs, as far as GitHub is concerned. Worse yet, the GitHub website masks the real name and e-mail address in the commit with the name of the GitHub user who claimed it. It's not a security issue for the project or any of its contributors, but I suppose it could be used as a form of identify theft. It takes just a few seconds to add your openjdk.org address, if you have one, to your GitHub profile. John
Re: issues on javafxports/openjdk-jfx
On 3/28/21 5:24 AM, Johan Vos wrote: I'm still in favor of closing it though. It would be nice if there were a way in GitHub to close the repository to new issues but keep the old ones public and editable. For example, I'm still getting good information on the old closed issues, even as recently as last week: mgroth0 commented 8 days ago https://github.com/javafxports/openjdk-jfx/issues/229#issuecomment-804546080 That doesn't seem possible, though. I can find only a checkbox to enable or disable issues. John
Re: RFR: 8238650: Allow to override buildDate with SOURCE_DATE_EPOCH
On Thu, 25 Mar 2021 02:35:24 GMT, John Neffenger wrote: >>> @jgneff Could not parse `bmwiedemann` as a valid contributor. >> >> You might retry the "contributor" command with the `@` before their GitHub >> username (I'm not 100% sure that will work, though). > >> You might retry the "contributor" command with the `@` before their GitHub >> username. > > Thanks. I'll try that now. I checked and Bernhard is on the list of "OCAs > submitted prior to 2021," but not on the OpenJDK Census list, in case that > matters. > > By the way, should I be adding my own name, too? It looks as if sometimes > people do that and sometimes not. That would put me as "Author" and also on a > line with "Co-authored-by" in the commit message. > This is a simple enough change by itself, but since it is intended as the > start of a larger effort, I'd like @johanvos (or someone else he designates > from Gluon) to be a second reviewer. I jumped the gun and created the [follow-up pull request][1]. I thought it might help when reviewing this pull request to have a full picture of that larger effort. Although these two changes are part of a huge undertaking in the open-source community, it turns out that the changes required on our part are quite small. [1]: https://github.com/openjdk/jfx/pull/446 - PR: https://git.openjdk.java.net/jfx/pull/422
Re: RFR: 8264449: Enable reproducible builds with SOURCE_DATE_EPOCH
On Tue, 30 Mar 2021 16:20:21 GMT, John Neffenger wrote: > This pull request allows for reproducible builds of JavaFX on Linux, macOS, > and Windows by defining the `SOURCE_DATE_EPOCH` environment variable. For > example, the following commands create a reproducible build: > > $ export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) > $ bash gradlew sdk jmods javadoc > $ strip-nondeterminism -v -T $SOURCE_DATE_EPOCH build/jmods/*.jmod > > The three commands: > > 1. set the build timestamp to the date of the latest source code change, > 2. build the JavaFX SDK libraries, JMOD archives, and API documentation, and > 3. recreate the JMOD files with stable file modification times and ordering. > > The third command won't be necessary once Gradle can build the JMOD archives > or the `jmod` tool itself has the required support. For more information on > the environment variable, see the [`SOURCE_DATE_EPOCH`][1] page. For more > information on the command to recreate the JMOD files, see the > [`strip-nondeterminism`][2] repository. I'd like to propose that we allow for > reproducible builds in JavaFX 17 and consider making them the default in > JavaFX 18. > > Fixes > > There are at least four sources of non-determinism in the JavaFX builds: > > 1. Build timestamp > > The class `com.sun.javafx.runtime.VersionInfo` in the JavaFX Base module > stores the time of the build. Furthermore, for builds that don't run on the > Hudson continuous integration tool, the class adds the build time to the > system property `javafx.runtime.version`. > > 2. Modification times > > The JAR, JMOD, and ZIP archives store the modification time of each file. > > 3. File ordering > > The JAR, JMOD, and ZIP archives store their files in the order returned > by the file system. The native shared libraries also store their object files > in the order returned by the file system. Most file systems, though, do not > guarantee the order of a directory's file listing. > > 4. Build path > > The class `com.sun.javafx.css.parser.Css2Bin` in the JavaFX Graphics > module stores the absolute path of its `.css` input file in the corresponding > `.bss` output file, which is then included in the JavaFX Controls module. > > This pull request modifies the Gradle and Groovy build files to fix the first > three sources of non-determinism. A later pull request can modify the Java > files to fix the fourth. > > [1]: https://reproducible-builds.org/docs/source-date-epoch/ > [2]: https://salsa.debian.org/reproducible-builds/strip-nondeterminism I think there might be a Skara bug. The pre-submit builds on Linux, macOS, and Windows completed immediately. I think that's because the first of the two commits in this pull request includes the Java Bug ID from [another pending pull request][1], because this pull request is a continuation of that one. I can squash the two commits and force-push the changes, if that would help. [1]: https://github.com/openjdk/jfx/pull/422 - PR: https://git.openjdk.java.net/jfx/pull/446
RFR: 8264449: Enable reproducible builds with SOURCE_DATE_EPOCH
This pull request allows for reproducible builds of JavaFX on Linux, macOS, and Windows by defining the `SOURCE_DATE_EPOCH` environment variable. For example, the following commands create a reproducible build: $ export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) $ bash gradlew sdk jmods javadoc $ strip-nondeterminism -v -T $SOURCE_DATE_EPOCH build/jmods/*.jmod The three commands: 1. set the build timestamp to the date of the latest source code change, 2. build the JavaFX SDK libraries, JMOD archives, and API documentation, and 3. recreate the JMOD files with stable file modification times and ordering. The third command won't be necessary once Gradle can build the JMOD archives or the `jmod` tool itself has the required support. For more information on the environment variable, see the [`SOURCE_DATE_EPOCH`][1] page. For more information on the command to recreate the JMOD files, see the [`strip-nondeterminism`][2] repository. I'd like to propose that we allow for reproducible builds in JavaFX 17 and consider making them the default in JavaFX 18. Fixes There are at least four sources of non-determinism in the JavaFX builds: 1. Build timestamp The class `com.sun.javafx.runtime.VersionInfo` in the JavaFX Base module stores the time of the build. Furthermore, for builds that don't run on the Hudson continuous integration tool, the class adds the build time to the system property `javafx.runtime.version`. 2. Modification times The JAR, JMOD, and ZIP archives store the modification time of each file. 3. File ordering The JAR, JMOD, and ZIP archives store their files in the order returned by the file system. The native shared libraries also store their object files in the order returned by the file system. Most file systems, though, do not guarantee the order of a directory's file listing. 4. Build path The class `com.sun.javafx.css.parser.Css2Bin` in the JavaFX Graphics module stores the absolute path of its `.css` input file in the corresponding `.bss` output file, which is then included in the JavaFX Controls module. This pull request modifies the Gradle and Groovy build files to fix the first three sources of non-determinism. A later pull request can modify the Java files to fix the fourth. [1]: https://reproducible-builds.org/docs/source-date-epoch/ [2]: https://salsa.debian.org/reproducible-builds/strip-nondeterminism - Commit messages: - Enable reproducible builds with SOURCE_DATE_EPOCH - 8238650: Allow to override buildDate with SOURCE_DATE_EPOCH Changes: https://git.openjdk.java.net/jfx/pull/446/files Webrev: https://webrevs.openjdk.java.net/?repo=jfx=446=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8264449 Stats: 36 lines in 3 files changed: 22 ins; 10 del; 4 mod Patch: https://git.openjdk.java.net/jfx/pull/446.diff Fetch: git fetch https://git.openjdk.java.net/jfx pull/446/head:pull/446 PR: https://git.openjdk.java.net/jfx/pull/446
Re: RFR: 8264010: Add Gradle dependency verification
On Wed, 24 Mar 2021 19:50:11 GMT, John Neffenger wrote:
> Are all of them actually needed?
Just to follow up on that question, all of them are in fact downloaded during
the build, at least. I removed the Gradle directory `$HOME/.gradle` and ran the
build as follows. Then I compared the list of downloaded artifacts with the
ones listed in the dependency verification file.
$ rm -r $HOME/.gradle
$ bash gradlew sdk jmods javadoc apps test -x :web:test
...
$ find ~/.gradle/caches/modules-2 ( -name "*.jar" -o -name "*.pom" ) \
-exec basename {} ; | sort > downloaded.log
$ grep '/\1/' \
| sort > verified.log
$ diff downloaded.log verified.log
31a32
> org.eclipse.swt.cocoa.macosx.x86_64_3.105.3.v20170228-0512-.jar
32a34
> org.eclipse.swt.win32.win32.x86_64_3.105.3.v20170228-0512-.jar
A total of 36 artifacts (14 JAR files and 22 POM files) are downloaded during
the build. The SWT libraries for macOS and Windows were not downloaded because
I ran the build on Linux.
-
PR: https://git.openjdk.java.net/jfx/pull/437
E-mail addresses at openjdk.org
I noticed that my last commit used the address jgn...@openjdk.org instead of the address I normally use to create and sign commits. GitHub didn't recognize the commit as mine until I added the address to my account. Do the openjdk.org addresses receive e-mail? If not, do we just leave the address unverified in our GitHub accounts? Thanks, John
Re: RFR: 8238650: Allow to override buildDate with SOURCE_DATE_EPOCH
On Wed, 24 Mar 2021 23:39:21 GMT, Kevin Rushforth wrote: > You might retry the "contributor" command with the `@` before their GitHub > username. Thanks. I'll try that now. I checked and Bernhard is on the list of "OCAs submitted prior to 2021," but not on the OpenJDK Census list, in case that matters. By the way, should I be adding my own name, too? It looks as if sometimes people do that and sometimes not. That would put me as "Author" and also on a line with "Co-authored-by" in the commit message. - PR: https://git.openjdk.java.net/jfx/pull/422
Re: RFR: 8264010: Add Gradle dependency verification
On Wed, 24 Mar 2021 19:55:20 GMT, Kevin Rushforth wrote: > I don't yet know to handle this ... Would any of the following options work? 1. If you're using your own supplemental closed Gradle build file, create your own supplemental closed Gradle verification file, too. Before the internal build, replace the current file with your own. 2. Remove the verification file before running your internal build. In this case, though, you'll also lose its protection against software supply-chain attacks. 3. Add your internal dependency checksum entries to the public verification file and publish the updated file in the repository. I think the protection from the verification file is worth having as a default in the public repository. Gluon, Oracle, BellSoft, and anyone else building JavaFX can decide, based on their own security assessment, whether or not they want to use it. The point of including the file in the repository is to make that decision explicit. - PR: https://git.openjdk.java.net/jfx/pull/437
Re: RFR: 8264010: Add Gradle dependency verification
On Tue, 23 Mar 2021 12:07:48 GMT, Kevin Rushforth wrote: >> This pull request adds dependency verification to the Gradle builds of >> JavaFX on Linux, macOS, and Windows. It is the third of three changes that >> close the gaps in the JavaFX build security: >> >> * [JDK-8262236][1]: Configure Gradle checksum verification >> * [JDK-8263204][2]: Add Gradle Wrapper Validation Action >> * [JDK-8264010][3]: Add Gradle dependency verification >> >> "Without dependency verification it's easy for an attacker to compromise >> your supply chain," warns the [Gradle User Guide][4]. All three changes come >> from conference talks by members of the Gradle team, available as [PDF >> slides][5] or on YouTube in the following two videos: >> >> * [Cédric Champeau at Devoxx][6] in November 2019 >> * [Louis Jacomet at Jfokus][7] in February 2020 >> >> "We all run in a crazy-unsafe environment," says Louis Jacomet at the end of >> his talk. These three changes make it just a little less crazy-unsafe for >> all of us building JavaFX, regardless of our system, network, or country. >> >> [1]: https://bugs.openjdk.java.net/browse/JDK-8262236 >> [2]: https://bugs.openjdk.java.net/browse/JDK-8263204 >> [3]: https://bugs.openjdk.java.net/browse/JDK-8264010 >> >> [4]: https://docs.gradle.org/current/userguide/dependency_verification.html >> [5]: >> https://www.jfokus.se/jfokus20-preso/Protecting-your-organization-against-attacks-via-the-build-system.pdf >> [6]: https://youtu.be/GWGNp3a3hpk >> [7]: https://youtu.be/bwiafNatsf0 > > This seems like a good idea to do. I have a couple overall questions before > reviewing / testing. > > 1. Can you add some sort of README file that describes the how to update the > checksums? Also, the instructions in > [UPDATING-lucene.txt](https://github.com/openjdk/jfx/blob/master/apps/samples/Ensemble8/UPDATING-lucene.txt) > should be updated accordingly. > 2. Some of the files listed are not used directly. I presume that you added > them because they are used indirectly by other components? Are all of them > actually needed? Thanks, Kevin. I added a README file and updated the Lucene instructions, as you suggested. I'm open to any other suggestions on the wording or formatting, no matter how minor. > Some of the files listed are not used directly. I presume that you added them > because they are used indirectly by other components? Are all of them > actually needed? The Gradle command, now documented in the `gradle/README.txt` file, adds entries to the dependency verification file for all dependencies, including transitive ones. I think that's the list of everything downloaded during the builds on Linux, macOS, and Windows. I'll clear the Gradle cache and double-check it now. I'll let you know if I find anything unexpected. - PR: https://git.openjdk.java.net/jfx/pull/437
Re: RFR: 8264010: Add Gradle dependency verification [v2]
> This pull request adds dependency verification to the Gradle builds of JavaFX > on Linux, macOS, and Windows. It is the third of three changes that close the > gaps in the JavaFX build security: > > * [JDK-8262236][1]: Configure Gradle checksum verification > * [JDK-8263204][2]: Add Gradle Wrapper Validation Action > * [JDK-8264010][3]: Add Gradle dependency verification > > "Without dependency verification it's easy for an attacker to compromise your > supply chain," warns the [Gradle User Guide][4]. All three changes come from > conference talks by members of the Gradle team, available as [PDF slides][5] > or on YouTube in the following two videos: > > * [Cédric Champeau at Devoxx][6] in November 2019 > * [Louis Jacomet at Jfokus][7] in February 2020 > > "We all run in a crazy-unsafe environment," says Louis Jacomet at the end of > his talk. These three changes make it just a little less crazy-unsafe for all > of us building JavaFX, regardless of our system, network, or country. > > [1]: https://bugs.openjdk.java.net/browse/JDK-8262236 > [2]: https://bugs.openjdk.java.net/browse/JDK-8263204 > [3]: https://bugs.openjdk.java.net/browse/JDK-8264010 > > [4]: https://docs.gradle.org/current/userguide/dependency_verification.html > [5]: > https://www.jfokus.se/jfokus20-preso/Protecting-your-organization-against-attacks-via-the-build-system.pdf > [6]: https://youtu.be/GWGNp3a3hpk > [7]: https://youtu.be/bwiafNatsf0 John Neffenger has updated the pull request incrementally with one additional commit since the last revision: Add a README file and update 'UPDATING-lucene.txt' - Changes: - all: https://git.openjdk.java.net/jfx/pull/437/files - new: https://git.openjdk.java.net/jfx/pull/437/files/2a11d401..c7ac7f62 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jfx=437=01 - incr: https://webrevs.openjdk.java.net/?repo=jfx=437=00-01 Stats: 53 lines in 2 files changed: 41 ins; 4 del; 8 mod Patch: https://git.openjdk.java.net/jfx/pull/437.diff Fetch: git fetch https://git.openjdk.java.net/jfx pull/437/head:pull/437 PR: https://git.openjdk.java.net/jfx/pull/437
Integrated: 8264061: LocalDateTimeStringConverterTest fails in Canada
On Tue, 23 Mar 2021 18:24:40 GMT, John Neffenger wrote: > As the comment in the test case mentions, "Tests require that default locale > is en_US." The default locale is required by the objects created in the > static method `implementations`, marked with the JUnit annotation > `@Parameterized.Parameters`. So the static method `initDefaultLocale`, marked > with the JUnit annotation `@BeforeClass`, calls the method to set the > required default: `Locale.setDefault(Locale.US)`. > > The problem occurs because the `@Parameterized.Parameters` method is called > before the `@BeforeClass` method, so the system locale is used as the default > instead of `Locale.US`. > > The fix is to move the call to the first instruction of the > `@Parameterized.Parameters` method so that the default is set before it's > required. This pull request has now been integrated. Changeset: 3bbcf977 Author:John Neffenger Committer: Kevin Rushforth URL: https://git.openjdk.java.net/jfx/commit/3bbcf977 Stats: 4 lines in 1 file changed: 0 ins; 3 del; 1 mod 8264061: LocalDateTimeStringConverterTest fails in Canada Reviewed-by: kcr - PR: https://git.openjdk.java.net/jfx/pull/438
RFR: 8264061: LocalDateTimeStringConverterTest fails in Canada
As the comment in the test case mentions, "Tests require that default locale is en_US." The default locale is required by the objects created in the static method `implementations`, marked with the JUnit annotation `@Parameterized.Parameters`. So the static method `initDefaultLocale`, marked with the JUnit annotation `@BeforeClass`, calls the method to set the required default: `Locale.setDefault(Locale.US)`. The problem occurs because the `@Parameterized.Parameters` method is called before the `@BeforeClass` method, so the system locale is used as the default instead of `Locale.US`. The fix is to move the call to the first instruction of the `@Parameterized.Parameters` method so that the default is set before it's required. - Commit messages: - 8264061: LocalDateTimeStringConverterTest fails in Canada Changes: https://git.openjdk.java.net/jfx/pull/438/files Webrev: https://webrevs.openjdk.java.net/?repo=jfx=438=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8264061 Stats: 4 lines in 1 file changed: 0 ins; 3 del; 1 mod Patch: https://git.openjdk.java.net/jfx/pull/438.diff Fetch: git fetch https://git.openjdk.java.net/jfx pull/438/head:pull/438 PR: https://git.openjdk.java.net/jfx/pull/438
Re: Not really a nice comment but a real issue?
On 3/23/21 12:34 AM, Johan Vos wrote: * bug-report system: I'm all in to find a more accessible way, keeping into account that would require work being done by the community (hence, us) for converting issues into high-quality JBS issues. I have two GitHub Issue Templates that gather the same information as the Oracle Web form for bug reports and feature requests. OpenJFX was the first project to move to GitHub. It could be the first to take over the bug submission, too. I know, easy for me to say! The burden would fall mainly on you and Kevin as project leads. You can look at the old sandbox repository to see the kind of reports we would get, as they're still coming in there at a steady pace. I think we could be pretty ruthless in regards to the quality. I'm losing patience with Java developers who say a bug is important but can't bother to push the "Step Into" button instead of the "Step Over" button in their debugger. But I'm fine as is, too. Maybe just let people know that they can become an Author and get access to the Java Bug System after only two commits! I was stunned when that happened. John
Re: Not really a nice comment but a real issue?
On 3/22/21 3:27 PM, Philip Race wrote: I am informed that, for no reason given, the corporate IT folks will not allow attachment upload. Thank you for looking into it, Phil. It's good at least to have a definitive answer. I brought this up on the mailing list three years ago, too: Re: More community participation in JavaFX https://mail.openjdk.java.net/pipermail/openjfx-dev/2018-February/021343.html I eventually created those bug reports, despite all the gate-keeping. In fact, I'm even starting to think there could be advantages to having pull requests as the only real way to participate in the OpenJFX project. A flood of easily-created issues on GitHub can be just as off-putting as that Oracle Web form. John
RFR: 8264010: Add Gradle dependency verification
This pull request adds dependency verification to the Gradle builds of JavaFX on Linux, macOS, and Windows. It is the third of three changes that close the gaps in the JavaFX build security: * [JDK-8262236][1]: Configure Gradle checksum verification * [JDK-8263204][2]: Add Gradle Wrapper Validation Action * [JDK-8264010][3]: Add Gradle dependency verification "Without dependency verification it's easy for an attacker to compromise your supply chain," warns the [Gradle User Guide][4]. All three changes come from conference talks by members of the Gradle team, available as [PDF slides][5] or on YouTube in the following two videos: * [Cédric Champeau at Devoxx][6] in November 2019 * [Louis Jacomet at Jfokus][7] in February 2020 "We all run in a crazy-unsafe environment," says Louis Jacomet at the end of his talk. These three changes make it just a little less crazy-unsafe for all of us building JavaFX, regardless of our system, network, or country. [1]: https://bugs.openjdk.java.net/browse/JDK-8262236 [2]: https://bugs.openjdk.java.net/browse/JDK-8263204 [3]: https://bugs.openjdk.java.net/browse/JDK-8264010 [4]: https://docs.gradle.org/current/userguide/dependency_verification.html [5]: https://www.jfokus.se/jfokus20-preso/Protecting-your-organization-against-attacks-via-the-build-system.pdf [6]: https://youtu.be/GWGNp3a3hpk [7]: https://youtu.be/bwiafNatsf0 - Commit messages: - 8264010: Add Gradle dependency verification Changes: https://git.openjdk.java.net/jfx/pull/437/files Webrev: https://webrevs.openjdk.java.net/?repo=jfx=437=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8264010 Stats: 173 lines in 1 file changed: 173 ins; 0 del; 0 mod Patch: https://git.openjdk.java.net/jfx/pull/437.diff Fetch: git fetch https://git.openjdk.java.net/jfx pull/437/head:pull/437 PR: https://git.openjdk.java.net/jfx/pull/437
Re: Not really a nice comment but a real issue?
On 3/19/21 12:37 PM, Philip Race wrote: I'm surprised credit is considered important It took me weeks to figure out some bugs and create a good report, all done as a volunteer in my spare time. Credit is my only compensation. I went as far as sneaking my name into the source code comments just to have it show up somewhere in the report where it wouldn't be removed. But FX never used the github bug tracker. It should never have been possible to submit bugs there. Oh, but it did! https://github.com/javafxports/openjdk-jfx/issues John
Re: Not really a nice comment but a real issue?
On 3/19/21 11:05 AM, Philip Race wrote: If this was important to him I don't understand why just a blog post and not a bug report .. If I had to guess, it might be because, in the age of GitHub, this is not what people expect when they try to report a bug: Report a Bug or Request a Feature https://bugreport.java.com/bugreport/ There are two main problems: 1. You can't attach images. 2. You don't get credit. I speak from experience. I spent five years frustrated with a JavaFX font bug, but it was only when I could properly format a report and include images that I bothered to open this issue: Reduce color fringes in FreeType subpixel rendering https://github.com/javafxports/openjdk-jfx/issues/229 Formatting and images shouldn't matter, but two other people tried to report that bug using the Oracle Web form, and both were closed as "Not an Issue." You can send images later by e-mail, eventually, but that's not explained anywhere. There was a brief window when the OpenJFX project accepted bug reports on GitHub, but now it's back to the Oracle Web form. That brief window is the reason I'm a contributor to the project now. I understand the need for gate-keeping. We just shouldn't be too surprised when people decide that the gate's too high. I think Oracle could fix the two problems and keep the Web form, and we might get more quality bug reports instead of frustrated blog posts. I also think that it would help a lot to enable the JIRA markup[1] in the Java Bug System as Apache NetBeans has done. As an example, I can't imagine trying to report a bug like the following without formatting or in-line images: Attaching JavaFX Javadoc and Sources https://issues.apache.org/jira/browse/NETBEANS-3296 John [1] https://jira.atlassian.com/secure/WikiRendererHelpAction.jspa?section=all
RFR: 8238650: Allow to override buildDate with SOURCE_DATE_EPOCH
This is a continuation of the [pull request][1] started by @bmwiedemann in January 2020. After this change is integrated, I can follow up immediately with additional pull requests that get us much closer to providing fully reproducible builds. Motivation The only conclusive way to verify a software package is to reproduce it. That's the main point of the Linux Foundation article [Preventing Supply Chain Attacks like SolarWinds][2] by David Wheeler, Director of Open Source Supply Chain Security. "In the longer term," he writes, "I know of only one strong countermeasure for this kind of attack: verified reproducible builds." It's not enough anymore to trust the person, organization, or company that publishes a software package. David Wheeler explains, "Assuming a system can 'never be broken into' is a failing strategy." As I see it, any project that doesn't yet allow for reproducible builds is on a list of possible attack vectors. I'd like to get OpenJFX off that list. This is a huge undertaking involving the entire open-source community. Just [Debian, for example][3], has over 30,000 source packages to build in a reproducible manner. Remarkably, it's almost 96 percent complete. The OpenJFX package is one of three percent still failing. Our first step towards helping in this goal is to support the [`SOURCE_DATE_EPOCH` specification][4]. Implementation When you want to build 30,000 packages in a reproducible manner, [command-line flags][5] unique to each package aren't so helpful. The environment variable needs to be set, anyway, for the tools invoked by Gradle to pick it up. We could allow for a Gradle property in addition to the environment variable. The Gradle property would override the default current date and export the environment variable, and the environment variable would override the command-line property. I think it makes more sense in the OpenJFX build to support the environment variable directly. With these considerations, I added the support just as recommended by the example for "Java / gradle" on the Reproducible Builds [`SOURCE_DATE_EPOCH`][6] page. For comparison, the [OpenJDK build][7] does the reverse, using the *configure* script option `--with-source-date` to export the `SOURCE_DATE_EPOCH` environment variable. [1]: https://github.com/openjdk/jfx/pull/99 [2]: https://www.linuxfoundation.org/en/blog/preventing-supply-chain-attacks-like-solarwinds [3]: https://tests.reproducible-builds.org/debian/reproducible.html [4]: https://reproducible-builds.org/specs/source-date-epoch/ [5]: https://wiki.debian.org/ReproducibleBuilds/StandardEnvironmentVariables#More_detailed_discussion [6]: https://reproducible-builds.org/docs/source-date-epoch/ [7]: https://github.com/openjdk/jdk/commit/1a16a4b6 - Commit messages: - 8238650: Allow to override buildDate with SOURCE_DATE_EPOCH Changes: https://git.openjdk.java.net/jfx/pull/422/files Webrev: https://webrevs.openjdk.java.net/?repo=jfx=422=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8238650 Stats: 6 lines in 1 file changed: 6 ins; 0 del; 0 mod Patch: https://git.openjdk.java.net/jfx/pull/422.diff Fetch: git fetch https://git.openjdk.java.net/jfx pull/422/head:pull/422 PR: https://git.openjdk.java.net/jfx/pull/422
Integrated: 8263204: Add Gradle Wrapper Validation Action
On Mon, 8 Mar 2021 20:34:05 GMT, John Neffenger wrote: > See the [Gradle Wrapper Validation > Action](https://github.com/marketplace/actions/gradle-wrapper-validation) for > details on this pull request. I'll test the changes with the following > sequence of commits: > > 1. This commit adds a tampered Gradle Wrapper JAR file, which should go > undetected. > 2. The next commit will add the Official Gradle Wrapper Validation Action, > which should detect the tampered file. > 3. The final commit will remove the tampered file and replace it with the > original Gradle 4.8 Wrapper. This pull request has now been integrated. Changeset: 75b4c15c Author:John Neffenger Committer: Kevin Rushforth URL: https://git.openjdk.java.net/jfx/commit/75b4c15c Stats: 14 lines in 1 file changed: 11 ins; 3 del; 0 mod 8263204: Add Gradle Wrapper Validation Action Reviewed-by: kcr - PR: https://git.openjdk.java.net/jfx/pull/419
Re: RFR: 8263204: Add Gradle Wrapper Validation Action
On Mon, 8 Mar 2021 22:37:10 GMT, John Neffenger wrote: >> If it isn't too hard to make the running of the other three jobs dependent >> on the wrapper validation job, that would be good, as long as the validation >> job is fast. The other three still need to run in parallel with each other. >> >> Alternatively, you could make the wrapper validation a task that is >> replicated in each of the three jobs. > > Okay, this is looking better! The validation took only 23 seconds, and GitHub > shows the following results: > >  > > The screenshot below shows the notification I received by e-mail: > >  > > Now we'll see whether the three build jobs still run in parallel when the > wrapper validation succeeds. My next commit will restore the Gradle Wrapper > JAR file. Looks good: two preliminary checks followed conditionally by the three concurrent build jobs:  Once the build jobs finish, this pull request is finally ready for review. Thanks for your timely help, Kevin. - PR: https://git.openjdk.java.net/jfx/pull/419
Re: RFR: 8263204: Add Gradle Wrapper Validation Action [v4]
> See the [Gradle Wrapper Validation > Action](https://github.com/marketplace/actions/gradle-wrapper-validation) for > details on this pull request. I'll test the changes with the following > sequence of commits: > > 1. This commit adds a tampered Gradle Wrapper JAR file, which should go > undetected. > 2. The next commit will add the Official Gradle Wrapper Validation Action, > which should detect the tampered file. > 3. The final commit will remove the tampered file and replace it with the > original Gradle 4.8 Wrapper. John Neffenger has updated the pull request incrementally with one additional commit since the last revision: Restore the Gradle version 4.8 Wrapper JAR file - Changes: - all: https://git.openjdk.java.net/jfx/pull/419/files - new: https://git.openjdk.java.net/jfx/pull/419/files/b2a737ad..9a4b0215 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jfx=419=03 - incr: https://webrevs.openjdk.java.net/?repo=jfx=419=02-03 Stats: 0 lines in 1 file changed: 0 ins; 0 del; 0 mod Patch: https://git.openjdk.java.net/jfx/pull/419.diff Fetch: git fetch https://git.openjdk.java.net/jfx pull/419/head:pull/419 PR: https://git.openjdk.java.net/jfx/pull/419
Re: RFR: 8263204: Add Gradle Wrapper Validation Action
On Mon, 8 Mar 2021 22:13:09 GMT, Kevin Rushforth wrote: >> Thanks, Kevin. I'll merge the two workflow files. >> >> The test results aren't what I expected: >> >>  >> >> I assumed the wrapper validation would fail and prevent the builds from >> running. Should I try to make the three build steps dependent on the success >> of the wrapper validation? There might be a way to make the steps sequential >> and conditional. > > If it isn't too hard to make the running of the other three jobs dependent on > the wrapper validation job, that would be good, as long as the validation job > is fast. The other three still need to run in parallel with each other. > > Alternatively, you could make the wrapper validation a task that is > replicated in each of the three jobs. Okay, this is looking better! The validation took only 23 seconds, and GitHub shows the following results:  The screenshot below shows the notification I received by e-mail:  Now we'll see whether the three build jobs still run in parallel when the wrapper validation succeeds. My next commit will restore the Gradle Wrapper JAR file. - PR: https://git.openjdk.java.net/jfx/pull/419
Re: RFR: 8263204: Add Gradle Wrapper Validation Action [v3]
> See the [Gradle Wrapper Validation > Action](https://github.com/marketplace/actions/gradle-wrapper-validation) for > details on this pull request. I'll test the changes with the following > sequence of commits: > > 1. This commit adds a tampered Gradle Wrapper JAR file, which should go > undetected. > 2. The next commit will add the Official Gradle Wrapper Validation Action, > which should detect the tampered file. > 3. The final commit will remove the tampered file and replace it with the > original Gradle 4.8 Wrapper. John Neffenger has updated the pull request incrementally with one additional commit since the last revision: Make build jobs dependent on wrapper validation Consolidate the two GitHub workflow files and make the three build jobs dependent on the success of the Gradle Wrapper validation. - Changes: - all: https://git.openjdk.java.net/jfx/pull/419/files - new: https://git.openjdk.java.net/jfx/pull/419/files/886a54a4..b2a737ad Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jfx=419=02 - incr: https://webrevs.openjdk.java.net/?repo=jfx=419=01-02 Stats: 24 lines in 2 files changed: 11 ins; 13 del; 0 mod Patch: https://git.openjdk.java.net/jfx/pull/419.diff Fetch: git fetch https://git.openjdk.java.net/jfx pull/419/head:pull/419 PR: https://git.openjdk.java.net/jfx/pull/419
Re: RFR: 8263204: Add Gradle Wrapper Validation Action
On Mon, 8 Mar 2021 21:44:03 GMT, Kevin Rushforth wrote: >> So far, so good. The tampered file was not detected: >> >>  >> >> The next commit will add the Official Gradle Wrapper Validation Action. > > It might be better to include the validation task in the same > [`submit.yml`](https://github.com/openjdk/jfx/blob/master/.github/workflows/submit.yml) > file as the pre-submit tests, as a separate job. That way it will get the > same set of conditions triggering it as the other pre-submit jobs. In > particular, we don't use the "on pull_request" trigger for our github actions > run, since all actions triggered on any pull request in any repo in the > openjdk org will be run in the context of the openjdk organization and we > would blow our limits too quickly. Also, this should be limited to the set of > branches that `submit.yml` uses. > > If there is a good reason to keep it in a separate file, then I would at > least duplicate this part from submit.yml: > > on: > # Run GitHub actions on every push to all branches except the main > production branches, also > # exclude any branch starting with "WIP". > push: > branches-ignore: > - master > - main > - 'jfx[0-9]+' > - 'WIP*' Thanks, Kevin. I'll merge the two workflow files. The test results aren't what I expected:  I assumed the wrapper validation would fail and prevent the builds from running. Should I try to make the three build steps dependent on the success of the wrapper validation? There might be a way to make the steps sequential and conditional. - PR: https://git.openjdk.java.net/jfx/pull/419
Re: RFR: 8263204: Add Gradle Wrapper Validation Action
On Mon, 8 Mar 2021 20:38:09 GMT, Kevin Rushforth wrote: >> See the [Gradle Wrapper Validation >> Action](https://github.com/marketplace/actions/gradle-wrapper-validation) >> for details on this pull request. I'll test the changes with the following >> sequence of commits: >> >> 1. This commit adds a tampered Gradle Wrapper JAR file, which should go >> undetected. >> 2. The next commit will add the Official Gradle Wrapper Validation Action, >> which should detect the tampered file. >> 3. The final commit will remove the tampered file and replace it with the >> original Gradle 4.8 Wrapper. > >> 1. This commit adds a tampered Gradle Wrapper JAR file, which should go >> undetected. >> 2. The next commit will add the Official Gradle Wrapper Validation Action, >> which should detect the tampered file. >> 3. The final commit will remove the tampered file and replace it with the >> original Gradle 4.8 Wrapper. > > This sounds like a good plan to test it. So far, so good. The tampered file was not detected:  The next commit will add the Official Gradle Wrapper Validation Action. - PR: https://git.openjdk.java.net/jfx/pull/419
Re: RFR: 8263204: Add Gradle Wrapper Validation Action [v2]
> See the [Gradle Wrapper Validation > Action](https://github.com/marketplace/actions/gradle-wrapper-validation) for > details on this pull request. I'll test the changes with the following > sequence of commits: > > 1. This commit adds a tampered Gradle Wrapper JAR file, which should go > undetected. > 2. The next commit will add the Official Gradle Wrapper Validation Action, > which should detect the tampered file. > 3. The final commit will remove the tampered file and replace it with the > original Gradle 4.8 Wrapper. John Neffenger has updated the pull request incrementally with one additional commit since the last revision: Add the Official Gradle Wrapper Validation Action - Changes: - all: https://git.openjdk.java.net/jfx/pull/419/files - new: https://git.openjdk.java.net/jfx/pull/419/files/710bb40e..886a54a4 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jfx=419=01 - incr: https://webrevs.openjdk.java.net/?repo=jfx=419=00-01 Stats: 10 lines in 1 file changed: 10 ins; 0 del; 0 mod Patch: https://git.openjdk.java.net/jfx/pull/419.diff Fetch: git fetch https://git.openjdk.java.net/jfx pull/419/head:pull/419 PR: https://git.openjdk.java.net/jfx/pull/419
