I work on the build system for Firefox and have some experience that
might be worth sharing.
We (Mozilla) evaluated GYP as the replacement for Makefile.in files.
While we generally love the concept of GYP (assemble data structures
then transform them N ways to whatever build backend/tool format you
want), we didn't really care for GYP's frontend file format. A "static"
file format like JSON, YAML, XML, etc just doesn't feel right,
especially when you need to do things like "if X & Y then do these things."
After a significant conversation on one of our main developer lists [1]
(many parts relevant to this post), we flirted with ini-like manifests
and Lua (it was invented as a config file language after all) but
eventually chose sandboxed Python. Once we decided to go with a real
programming language, the choice of Python was easy, as that is what
most of our tools are written in.
Instead of Makefile.in or .gyp files, we have what are called
"moz.build" files. These are actually Python scripts executed in a
highly limited Python environment (no importing, no I/O, etc). We define
special functions and variables. At the end of each file's execution, we
extract values from the symbol table then construct data structures
representing the build configuration. At this point, we essentially have
arrived at the backend portion of GYP.
See [2] for slides with some examples. A technical readme exists at [3].
The main code lives at [4] and [5]. The sandboxing code [4] is the real
bread and butter. With relative ease it can be extracted from the
Firefox source tree (we'd like to eventually have it live on PyPI).
Unfortunately, we haven't yet landed the initial switchover, so I can't
tell you how the experience has been received. Having personally
converted large portions of our source tree to use moz.build files, I
can say that it feels much better writing "real" code as opposed to
shoehorning procedural logic into something like YAML.
We have aspirations of eventually hooking our moz.build-derived data
structures into GYP (why reinvent the build backend generation wheel).
If that works out well, we speculated we could contribute the Python
sandboxing code to GYP as an alternate frontend file format or at least
make it available as a standalone package. But this is so far out that
it's only wishful thinking. If WebKit or some other large GYP-using
project were interested, I'd love to explore possibilities of unifying
solutions.
I don't actively follow this mailing list, so please CC me on any
replies. You can also find me as gps on irc.mozilla.org (try #build).
Gregory Szorc
[email protected]
[1]
https://groups.google.com/d/topic/mozilla.dev.platform/SACOnl-avMs/discussion
[2]
http://gregoryszorc.com/presentations/2012-11-29-firefox-build-system/#41
[3]
https://hg.mozilla.org/mozilla-central/file/1eab3e0d7f53/python/mozbuild/mozbuild/frontend/README.rst
[4]
https://hg.mozilla.org/mozilla-central/file/1eab3e0d7f53/python/mozbuild/mozbuild/frontend/sandbox.py
[5]
https://hg.mozilla.org/mozilla-central/file/1eab3e0d7f53/python/mozbuild/mozbuild/frontend/reader.py
On 2/5/13 2:01 PM, Ryosuke Niwa wrote:
On Tue, Feb 5, 2013 at 1:36 PM, Mark Mentovai <[email protected]
<mailto:[email protected]>> wrote:
In the end, I don’t think that the actual syntax (as opposed to the
structured data that it contains) is all that important. GYP input
files are fairly simple structured data consisting of very few
types: dicts, lists, strings, and infrequent bools and ints. Any
extant or nascent format that can handle these is a candidate. If
there’s some kind of convergence on a beautiful one that’s easy to
maintain and quick to parse, there’s no reason GYP can’t be taught
to read it, even alongside the existing JSON-esque format that’s fed
to the Python interpreter. If this is the only thing standing
between a project and its adopting GYP, then it’s a really easy one
to get over.
The syntax has always been the biggest problem when it comes to adopting
GYP in WebKit.
Let's take the following as an example as picked by Maciej
{
'action_name': 'XMLNames',
'inputs': [
'../dom/make_names.pl <http://make_names.pl>',
'../xml/xmlattrs.in <http://xmlattrs.in>',
],
'outputs': [
'<(SHARED_INTERMEDIATE_DIR)/webkit/XMLNames.cpp',
'<(SHARED_INTERMEDIATE_DIR)/webkit/XMLNames.h',
],
'action': [
'python',
'scripts/action_makenames.py',
'<@(_outputs)',
'--',
'<@(_inputs)',
'--',
'--extraDefines', '<(feature_defines)'
],
'msvs_cygwin_shell': 1,
},
Using a YAML-like syntax, we can rewrite it as:
XMLName:
inputs:
<(SHARED_INTERMEDIATE_DIR)/../dom/make_names.pl
<http://make_names.pl>
<(SHARED_INTERMEDIATE_DIR)/../xml/xmlattrs.in <http://xmlattrs.in>
outputs:
XMLNames.cpp
XMLNames.h
action:
python scripts/action_makenames.py <@(_outputs) -- <@(_inputs)
-- --extraDefines <(feature_defines)
msvs_cygwin_shell: True
To me, this YAML-like syntax reads significantly better.
Now, I don't like all the magic strings like
"<(SHARED_INTERMEDIATE_DIR)/../" and "<@(_outputs) -- <@(_inputs) --
--extraDefines <(feature_defines)". Can someone with build system
knowlede prettify them?
- R. Niwa
_______________________________________________
webkit-dev mailing list
[email protected]
https://lists.webkit.org/mailman/listinfo/webkit-dev
_______________________________________________
webkit-dev mailing list
[email protected]
https://lists.webkit.org/mailman/listinfo/webkit-dev
