On 2020-03-23 21:19, Mandy Chung wrote:
Hi Magnus,
Modularizing the build tools is a good move.
Thanks!
This patch suggests to place the build tools under
src/$MODULE/share/tools/$PACKAGE/*.java
I think the modular source location of the build tools needs more
discussion, including jigsaw-dev for this discussion.
Ok, I've pruned the recipient list down to just jigsaw-dev and
build-dev. Let's keep this thread for discussing where to put the code
in the source tree, and when we agree on that I can make an updated
webrev and get buy-in from the component owners.
The JDK source as specified in JEP 201 is under:
src/$MODULE/{share,$OS}/classes/$PACKAGE/*.java
Compiling the source files from the `src` directory are the
intermediate input to build the resulting image. Build tools are
used to generate additional intermediate input (that is not part of
the `src` directory) to build the image. So I wonder if
make/$MODULE/share/tools or make/tools/$MODULE may be better location
for the build tools.
That's certainly a possibility. Ever since the big jigsaw restructuring,
Erik and I have discussed how to modularize the build system. I did not
even like the original jigsaw push, but there was not enough time to fix
the build system, and as a compromise we opened a bunch of JBS issues to
fix this in a follow-up fashion. Two are still remaining: JDK-8055192
and JDK-8055193, and JDK-8241463 is a part of that, even though it seems
to have not been written down in a bug report at that time.
At the core, we'd like to "invert" the current structure where we have
files like:
make/lib/Lib-java.base.gmk
make/lib/Lib-java.desktop.gmk
make/gensrc/Gensrc-java.base.gmk
make/gensrc/Gensrc-java.desktop.gmk
... etc
and instead have like:
make/modules/java.base/lib.gmk
make/modules/java.base/gensrc.gmk
make/modules/java.desktop/lib.gmk
make/modules/java.desktop/gensrc.gmk
However, this stuff is clearly a core part of the build system, so there
is no question that this belongs under "make".
In contrast, the build tools are only interacting with the build system
on the surface. There is some kind of an API here -- the build tools
have some kind of calling convention, wants some input and produces some
output, which is consumed by the build system. But the actual workings
of the build tools have 100% to do with the component. In some cases,
the original developer found it more suitable to create a tool for
generating programmatically a number of classes, rather than doing that
once, for hand, and check in the code. (I'm a bit skeptical towards
those build tools, btw, even though I understand the reason.) But this
clearly is just another way to express the core functionality of that
module. If anything is causing confusion right now, it's when the
component teams do not know about or take responsibility for the build
tools.
So I think the build tools are just like much of the rest of the product
-- the build system needs to be aware of them and how they work, but
they are really the expression -- and responsibility -- of the component
owners. And thus, I believe they are much better suited in the src tree.
Furthermore, we have JDK-8055193, about modularizing the make/data
directory. This is something I care even more about. And here I'm quite
firm in my conviction that this has nothing to do in the make directory.
It's just a remnant of the old thinking of "Oh, I don't know where to
put this. Let's just continue using 'make' as our misc trashcan". Things
like make/data/macosxicons should move to
src/java.desktop/data/macosxicons, and make/data/publicsuffixlist to
src/java.base/data/publicsuffixlist, and so forth. There's absolutely
nothing here that has anything to do with the build system. (With the
possible exception of product-wide stuff, I know these exists among the
tools, I'm not sure about the data.)
And since there is a strong correlation between the data files and the
build tools -- many build tools are custom made to process things in the
data directory -- I think it makes much sense to move the source code of
the tools along with them.
We already have collected everything else that belongs to a module under
src/$module/share. Apart from classes, and native, we have:
* conf
* lib
* legal
* man
for those modules that require them.
My suggestion is that we add, for those module that require them:
* data
* tools
If the name of the latter is not good enough, I'm open for suggestions.
Maybe "build-tools", "buildtools", "tools-src", "tools-classes",
"classes-tools"...? My idea for picking "tools" was that it was short,
and seemed to fit in well with the style of the other names. An
additional benefit was that it was not as limiting as "buildtools", and
thus allowing for the modules to put other kinds of tools there that
might not be needed at build time -- for instance, tools that are needed
every once in a while to update some checked-in files, like
make/script/generate-symbol-data.sh (which really belongs to jdk.compiler).
And regarding JEP 201, as far as I can tell the additional "man" and
"lib" directories do not seem to be reflected in JEP 201. I do not know
if this is considered an oversight, or just reflecting the fact that the
directory structure will continue to evolve after JEP 201 were
delivered. But if you think JEP 201 needs to be updated, fine, I'll
gladly help with that.
/Magnus
Mandy
On 3/23/20 12:03 PM, Magnus Ihse Bursie wrote:
The build tools (small java tools that are run during the build to
generate source code, or data, needed in the JDK) have historically
been placed in the "make" directory. This maybe made sense long time
ago, but does not do so anymore.
Instead, the build tools source code should move the the module that
needs them. For instance, compilefontconfig should move to
java.desktop, etc.
There are multiple reasons for this:
* Currently we build *all* build tools at once, which mean that we
cannot compile java.base until e.g. the compilefontconfig tool is
compiled, even though it is not needed.
* If a build tool, e.g. compilefontconfig is modified, all build
tools are recompiled, which triggers a rebuild of more or less the
entire JDK. This makes development of the build tools unnecessary
tedious.
* When the build tools are modified, the group owning the
corresponding module is the proper review instance, not the build
team. But since they reside under "make", the review mails often
include build-dev, but this is mostly noise for us. With this move,
the ownership is made clear.
In this patch, I have not modified how and when the build tools are
compiled, but this shuffle is the prerequisite for continuing with
that in a follow-up patch.
I have also moved the build tools to the org.openjdk.buildtools.*
package name space (inspired by Skara), instead of the strangely
named build.tools.* name space.
A few build tools are not moved in this patch. Two of them,
charsetmapping and cldrconverter, are shared between two modules. (I
think they should move to modules nevertheless, but they need some
more thought to make sure I do this right.) The rest are tools that
are needed for the build in general, like linking or javadoc support.
I'll move this to a better location too, but in a separate patch.
Bug: https://bugs.openjdk.java.net/browse/JDK-8241463
WebRev:
http://cr.openjdk.java.net/~ihse/JDK-8241463-move-build-tools-to-modules/webrev.01
/Magnus
