On Tue, Apr 16, 2024 at 9:20 AM Jiangli Zhou <jiangliz...@google.com> wrote: > > On Tue, Apr 16, 2024 at 8:30 AM Magnus Ihse Bursie > <magnus.ihse.bur...@oracle.com> wrote: > > > > Jiangli, > > > > First of all: I tried building the leyden branch "hermetic-java-runtime" > > but failed. :-( I tried some various attempts to work around the failures, > > but the javastatic file I ended up with were unable to start. > > Magnus, I recall you mentioned that you had build issues during one of > the zoom meetings earlier this year (didn't hear any follow-up on it). > Let's get this resolved for you first. Can you please share the build > failures/errors and runtime errors that you run into? > > Same questions about your build environment, as you asked below. > > > > > What kind of environment are you using to build this branch? (OS, compiler, > > libc version, etc) Do you have any special configure flags that are > > required? > > For the leyden hermetic-java-runtime branch, I've only been building > on Linux. I've used both gcc and clang. > > Yes, the branch requires `--with-static-java=yes`, e.g. > > bash configure --with-boot-jdk=<jdk_path> > --with-debug-level=slowdebug --with-static-java=yes > > I documented the build information in one of the comment messages. > That's probably not the best place. I'll look for a better place. > > Let's discuss more on the rest of the topics below during our meeting > today. We can update on the mailing list about the discussion, if > others are interested.
For anyone who's interested in the topics, https://mail.openjdk.org/pipermail/leyden-dev/2024-April/000643.html is a separate email thread with today's meeting summary from @Magnus Ihse Bursie and responses from me. Best, Jiangli > > Best, > Jiangli > > > > > I would very much like to test it out myself to check this static java > > launcher that you are creating. > > > > On 2024-04-16 03:01, Jiangli Zhou wrote: > > > > The main purpose of StaticLink.gmk is to support the static-java-image > > make target, which can be used to perform the actual static linking > > step using libjvm.a and JDK static libraries. That currently doesn't > > exist in the JDK mainline. Creating a "fully" statically linked Java > > launcher is the first step (out of many) towards supporting > > static/hermetic Java. > > > > I am still not sure it is the best design to have this in a separate file. > > You are in some way "just" creating a new launcher. > > > > As part of cleaning/refactoring/integrating for the static linking > > step, we want to agree and decide/accept on the following: > > > > - Support the "fully" statically linked java launcher for testing and > > demoing the capability of static JDK support, e.g. > > - Support running jtreg testing using the "fully" statically linked > > Java launcher > > > > So how do you want to test native jtreg tests? By compiling the jtreg tests > > to native .a files and statically link with those as well, or to make this > > a hybrid mode where a statically linked launcher loads the jtreg tests > > dynamically? > > > > - Set up tests in github workflow to help detect any breaking > > changes for static support, e.g. new symbol issues introduced by any > > changes. There were some earlier discussions on this with Ron and Alan > > during the zoom meetings. > > > > If we do this correct, we should not have to worry about symbol conflicts. > > (Local symbols should all be hidden before creation of the static > > libraries.) That said, creating static libraries needs to be regularly > > tested or it will break. > > > > - Which JDK native libraries to be statically linked with the new > > launcher target? E.g. StaticLink.gmk currently excludes libjsound.a, > > libawt_xawt.a, etc from statically linked with the launcher. > > > > That sounds rather arbitrarily. I would assume that a statically linked > > launcher should include native libraries from all included modules. If you > > are not interested in java.desktop native libraries, then you'd need to > > exclude java.desktop from the build. But my assumption here is that we need > > to build a solution that works with all modules included in the JDK, and > > that is what must be tested in GHA. > > > > - Do we want more than one statically linked launcher target, based on > > the set of linked native libraries? > > > > Do you mean that you want like "static-java-java.base-only", > > "static-java-eveything-but-java.desktop" etc? That does not sound like > > something that belongs in the makefiles. If you want to build a JDK that > > can only support a certain subset of modules, you need to specify those > > module in your build scripts as input to configure. > > > > (II) What missing: > > - Static linking step as mentioned above > > > > (III) What needs to be improved (require cleanups and refactoring, and > > you mentioned some of those in your response as well): > > - Support building both the static libraries and dynamic libraries > > using the same set of .o files, instead of separately compiled .o > > files. That helps improve build speed and reduce memory overhead for > > building JDK. Your current refactoring work aims to help that. > > - Clean up the usages of STATIC_BUILD macro. Most of the usages are in > > test code. > > - Other runtime fixes/enhancements in the leyden > > https://github.com/openjdk/leyden/tree/hermetic-java-runtime branch > > > > I think most work mentioned in III has dependencies on II. We need a > > workable base to be able to build the "fully" statically linked > > launcher for building and testing the work mentioned in III, when > > integrating any of those to the JDK mainline. The makefile refactoring > > work can be done in parallel but does not need to be completed before > > we add the static linking step in JDK mainline. > > > > I'm not sure I understand why you consider (II) to be a prerequisite for > > the (III) issues. From my point of view, they are all mutually independent > > changes that needs to be done, and if anything, I think getting in code to > > create a statically linked launcher will probably be easier if we get > > further along on the other issues. > > > > In particular, I'd like to see changes to the STATIC_BUILD macro come in > > first. Ideally, we should not send in such a macro at all. The JNI_OnLoad > > stuff is bugging me mostly. > > > > As of today, the leyden > > https://github.com/openjdk/leyden/tree/hermetic-java-runtime branch > > can build a "fully" statically linked Java launcher. The issue of > > compiling the dynamic and static libraries .o files separately is not > > a blocker. It's good to have it resolved at some point of time. > > > > Weeeell, yes and no... It is not *technically* a blocker, but unless we can > > reuse the .o files, including static builds in the standard testing on GHA > > will effectively double the build time, and that is definitely not > > acceptable. And the alternative is to bring in static builds without having > > it regularly tested, which I don't think is acceptable either. > > > > So the conclusion is that getting static and dynamic libraries built from > > the same .o files is in effect a blocker for Hermetic Java. > > > > This, in turn, require several changes: > > > > 1) The linking code needs to be cleaned up, and all technical debt needs to > > be resolved. This is what I have been doing since I started working on > > static builds for Hermetic Java. JDK-8329704 (which was integrated > > yesterday) was the first major milestone of this cleanup. Now, the path > > were to find a library created by the JDK (static or dynamic) is > > encapsulated in ResolveLibPath. This is currently a monster, but at least > > all knowledge is collected in a single location, instead of spread over the > > code base. Getting this simplified is the next step. > > > > 2) We need to stop passing the STATIC_BUILD define when compiling. This is > > partially addressed in your PR, where you have replaced #ifdef STATIC_BUILD > > with a dynamic lookup. But there is also the problem of JNI/JVMTI entry > > points. I have been pondering how we can compile the code in a way so we > > support both dynamic and static name resolution, and I think I have a > > solution. > > > > This is unfortunately quite complex, and I have started a discussion with > > Alan if it is possible to update the JNI spec so that both static and > > dynamic entry points can have the form "JNI_OnLoad_<library-name>". > > Ideally, I'd like to see us push for this with as much effort as possible. > > If we got this in place, static builds would be much easier, and the > > changes required for Hermetic Java even smaller. > > > > Thumbs up! That seems to be a good direction. Currently in the leyden > > branch, it first looks up the unique > > JNI_OnLoad<_lib_name>|Agent_OnLoad<_lib_name> etc for built-in > > libraries, then search for the dynamic libraries using the > > conventional naming when necessary. e.g.: > > > > https://github.com/openjdk/leyden/commit/a5c886d2e85a0ff0c3712a5488ae61d8c9d7ba1a > > https://github.com/openjdk/leyden/commit/1da8e3240e0bd27366d19f2e7dde386e46015135 > > > > When spec supports JNI_OnLoad_<library-name> and etc. for dynamic > > libraries, we may still need to support the conventional naming > > without the <_lib_name> part for existing libraries out there. > > > > That looks interesting. Basically, this is very similar to what I imagine > > needs to be done in the mainline. However, I don't think we can just change > > this code like that without a CSR? Or are we allowed to treat JDK-internal > > libraries different than the specification states? > > > > I would say getting this part into mainline should be the main focus right > > now, not the StaticLink.gmk file. And that includes getting any CSR > > approvals etc. > > > > Thank you for taking this on! Potentially we could consider taking the > > objcopy to localizing hotspot symbols on unix-like platforms, based on > > https://github.com/openjdk/jdk/pull/17456 discussions. Additional > > testing is still needed to verify the solution. > > > > We don't need any additional testing on that; proof of concept shows it > > works. It just needs to be implemented. With JDK-8330261 (pushed today), > > the ground is now prepared to get it done. It is next on my todo-list. > > > > It might be a good idea to follow up on the static linking discussion > > in tomorrow's zoom meeting (hope you'll be able to join tomorrow). > > > > I'll join but might be somewhat late due to family commitments. > > > > /Magnus
