Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On 2013-02-03, at 21:20, Maciej Stachowiak m...@apple.com wrote: I should mention that there's a lot of interest right now at Apple in the possibility of switching to Gyp. I’ve filed https://bugs.webkit.org/show_bug.cgi?id=109248 to track initial work in getting gyp set up for the Mac build. I’m initially attempting to generate Xcode projects that closely match our existing projects in order to minimize unintentional changes in build behavior. - Mark ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On Monday, February 04, 2013 10:57:46 AM Maciej Stachowiak wrote:
On Feb 4, 2013, at 10:46 AM, Mark Mentovai m...@chromium.org wrote:
GYP was written in Python to address point (b). Python was already part of
the baseline requirements on all platforms, so we already had Python
available everywhere we needed it. There are no dependencies on external
binaries, and no compiled code needs to be checked in anywhere or
maintained as part of a base image.
As for point (a), you can easily have a top-level Makefile not generated
by GYP that says “run GYP to produce the build files for whatever
environment you like and then pass control to that build system to do the
rest of the build. Developers who like it can use ninja for their own
builds, and your bots can use Xcode or make if that’s a requirement (or
if ninja doesn’t meet your requirements given point (b)).
Checking in the generated Xcode projects is another alternative. The
Makefile might be better for the reasons you suggest, though.
I'm reasonably confident at this point that Gyp can meet our hard
requirements. Our remaining issues are finding time to do it and
comprehensibility/readability of the syntax.
On the topic of the Gyp syntax:
I find it pretty readable, but I wonder if you guys are open to the idea of
relaxing the parser a bit to also allow for the omission of quotes and commas
for the dictionary keys. I feel a syntax like
{
includes: [
...
]
variables: [
]
}
is slightly less error prone to type and you could probably still support the
stricter language, too.
Simon
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
The parser (and the grammar) works the way it does because it’s just
Python—the whole thing can be slurped in quickly by the interpreter and
made available to GYP as data. It’s faster than a parser written in Python
for a similar language would be, and it’d likely take a binary Python
module to approach or exceed this speed. Developers don’t like waiting for
tools.
With that in mind, I’d be open to experimenting with alternative (probably
optional) formats.
On Tue, Feb 5, 2013 at 7:48 AM, Simon Hausmann simon.hausm...@digia.comwrote:
On Monday, February 04, 2013 10:57:46 AM Maciej Stachowiak wrote:
On Feb 4, 2013, at 10:46 AM, Mark Mentovai m...@chromium.org wrote:
GYP was written in Python to address point (b). Python was already
part of
the baseline requirements on all platforms, so we already had Python
available everywhere we needed it. There are no dependencies on
external
binaries, and no compiled code needs to be checked in anywhere or
maintained as part of a base image.
As for point (a), you can easily have a top-level Makefile not
generated
by GYP that says “run GYP to produce the build files for whatever
environment you like and then pass control to that build system to do
the
rest of the build. Developers who like it can use ninja for their own
builds, and your bots can use Xcode or make if that’s a requirement (or
if ninja doesn’t meet your requirements given point (b)).
Checking in the generated Xcode projects is another alternative. The
Makefile might be better for the reasons you suggest, though.
I'm reasonably confident at this point that Gyp can meet our hard
requirements. Our remaining issues are finding time to do it and
comprehensibility/readability of the syntax.
On the topic of the Gyp syntax:
I find it pretty readable, but I wonder if you guys are open to the idea of
relaxing the parser a bit to also allow for the omission of quotes and
commas
for the dictionary keys. I feel a syntax like
{
includes: [
...
]
variables: [
]
}
is slightly less error prone to type and you could probably still support
the
stricter language, too.
Simon
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On Tue, Feb 5, 2013 at 6:09 AM, Mark Mentovai m...@chromium.org wrote: The parser (and the grammar) works the way it does because it’s just Python This works great for people who like Python syntax but not for someone like myself who dislikes Python syntax. I also find it particularly annoying that people can use whatever Python constructs they want to use in GYP. It dramatically reduces language portability because you need to support quite a few Python constructs and quirks in order to correctly parse GYP. I personally would have much preferred for it be a simple JSON file. - R. Niwa ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
You’re not supposed to use arbitrary Python, it’s highly discouraged. We have a linter that keeps you from doing things you’re not supposed to do (like this), but it slows things down, so it’s not part of the “standard” GYP run that developers normally use. It can run as a pre-commit script or test on the bot or something else. Used as we’re using it, GYP basically is simple JSON, except the rules about commas are looser and # introduces comments. On Tue, Feb 5, 2013 at 2:47 PM, Ryosuke Niwa rn...@webkit.org wrote: On Tue, Feb 5, 2013 at 6:09 AM, Mark Mentovai m...@chromium.org wrote: The parser (and the grammar) works the way it does because it’s just Python This works great for people who like Python syntax but not for someone like myself who dislikes Python syntax. I also find it particularly annoying that people can use whatever Python constructs they want to use in GYP. It dramatically reduces language portability because you need to support quite a few Python constructs and quirks in order to correctly parse GYP. I personally would have much preferred for it be a simple JSON file. - R. Niwa ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
I'm curious if YAML was ever considered? I have very limited experience with YAML, except for Google App Engine config files. It's very python parse-able? :) On Tue, Feb 5, 2013 at 11:55 AM, Mark Mentovai m...@chromium.org wrote: You’re not supposed to use arbitrary Python, it’s highly discouraged. We have a linter that keeps you from doing things you’re not supposed to do (like this), but it slows things down, so it’s not part of the “standard” GYP run that developers normally use. It can run as a pre-commit script or test on the bot or something else. Used as we’re using it, GYP basically is simple JSON, except the rules about commas are looser and # introduces comments. On Tue, Feb 5, 2013 at 2:47 PM, Ryosuke Niwa rn...@webkit.org wrote: On Tue, Feb 5, 2013 at 6:09 AM, Mark Mentovai m...@chromium.org wrote: The parser (and the grammar) works the way it does because it’s just Python This works great for people who like Python syntax but not for someone like myself who dislikes Python syntax. I also find it particularly annoying that people can use whatever Python constructs they want to use in GYP. It dramatically reduces language portability because you need to support quite a few Python constructs and quirks in order to correctly parse GYP. I personally would have much preferred for it be a simple JSON file. - R. Niwa ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
http://www.haskell.org/haskellwiki/Wadlers_Law On Tue, Feb 5, 2013 at 12:16 PM, Eric Seidel e...@webkit.org wrote: I'm curious if YAML was ever considered? I have very limited experience with YAML, except for Google App Engine config files. It's very python parse-able? :) On Tue, Feb 5, 2013 at 11:55 AM, Mark Mentovai m...@chromium.org wrote: You’re not supposed to use arbitrary Python, it’s highly discouraged. We have a linter that keeps you from doing things you’re not supposed to do (like this), but it slows things down, so it’s not part of the “standard” GYP run that developers normally use. It can run as a pre-commit script or test on the bot or something else. Used as we’re using it, GYP basically is simple JSON, except the rules about commas are looser and # introduces comments. On Tue, Feb 5, 2013 at 2:47 PM, Ryosuke Niwa rn...@webkit.org wrote: On Tue, Feb 5, 2013 at 6:09 AM, Mark Mentovai m...@chromium.org wrote: The parser (and the grammar) works the way it does because it’s just Python This works great for people who like Python syntax but not for someone like myself who dislikes Python syntax. I also find it particularly annoying that people can use whatever Python constructs they want to use in GYP. It dramatically reduces language portability because you need to support quite a few Python constructs and quirks in order to correctly parse GYP. I personally would have much preferred for it be a simple JSON file. - R. Niwa ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On Tue, Feb 5, 2013 at 12:20 PM, Nico Weber tha...@chromium.org wrote: http://www.haskell.org/haskellwiki/Wadlers_Law I've started working on an experimental gyp build for GTK+, so I have first-hand experience as a gyp newbie editing .gyp and .gypi files. While the aesthetics of the files haven't bothered me much, I find that it's very easy to introduce syntax errors. I often forget a comma or use the wrong closing parenthesis. Since the files are deeply nested, it's easy to lose track. Most frustratingly the error message from the tool aren't very useful for finding the problem. I usually resort to matching parenthesis with vim's % command and squinting. Other than this, I love it so far. The build file generation is very quick (admittedly I have only a small part of WebKit building) and it seems to handle source generating dependencies really well. This is one of the most painful issues we have with autotools. I also look forward to building with ninja. :) --Martin ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On Feb 5, 2013, at 6:09 AM, Mark Mentovai m...@chromium.org wrote:
The parser (and the grammar) works the way it does because it’s just
Python—the whole thing can be slurped in quickly by the interpreter and made
available to GYP as data. It’s faster than a parser written in Python for a
similar language would be, and it’d likely take a binary Python module to
approach or exceed this speed. Developers don’t like waiting for tools.
Is that a result based on testing, or just a projectiom? My guess would be that
most incremental builds do not add or remove any files, and therefore do not
need to regenerate the project files at all, so the speed of parsing GYP files
is unlikely to have a major impact, unless it's a truly massive difference.
I have to say that the syntax weight of gyp files (quotes, commas, braces,
brackets) makes them feel harder to read and edit than other build file formats
designed for human consumption (obviously Xcode is worse).
Compare this bit of gyp for generating a derived source file:
{
'action_name': 'XMLNames',
'inputs': [
'../dom/make_names.pl',
'../xml/xmlattrs.in',
],
'outputs': [
'(SHARED_INTERMEDIATE_DIR)/webkit/XMLNames.cpp',
'(SHARED_INTERMEDIATE_DIR)/webkit/XMLNames.h',
],
'action': [
'python',
'scripts/action_makenames.py',
'@(_outputs)',
'--',
'@(_inputs)',
'--',
'--extraDefines', '(feature_defines)'
],
'msvs_cygwin_shell': 1,
},
vs the equivalent bit of DerivedSources.make:
XMLNames.cpp : dom/make_names.pl xml/xmlattrs.in
perl -I $(WebCore)/bindings/scripts $ --attrs
$(WebCore)/xml/xmlattrs.in
Or GNUMakefile.am:
DerivedSources/WebCore/XMLNames.cpp DerivedSources/WebCore/XMLNames.h:
$(WebCore)/dom/make_names.pl $(WebCore)/xml/xmlattrs.in
$(AM_V_GEN)$(PERL) -I$(WebCore)/bindings/scripts $ --attrs
$(WebCore)/xml/xmlattrs.in --outputDir $(GENSOURCES_WEBCORE)
Or DerivedSources.pri:
xmlnames.output = XMLNames.cpp
xmlnames.input = XML_NAMES
xmlnames.script = $$PWD/dom/make_names.pl
xmlnames.commands = perl -I$$PWD/bindings/scripts $$xmlnames.script --attrs
$$PWD/xml/xmlattrs.in --preprocessor \$${QMAKE_MOC} -E\ --outputDir
${QMAKE_FUNC_FILE_OUT_PATH}
GENERATORS += xmlnames
Gyp kind of seems like an outlier in heaviness of the syntax compared to the
other build systems.The irony is that Python itself doesn't use a lot of
punctuation in its syntax, so actual Python code looks much cleaner. I like to
read Python code, but I can't say I enjoy reading the Gyp syntax. This makes me
wonder if a Python DSL might be more legible than simply using a dictionary.
Alternately, maybe it turns out that parsing a custom language is not that huge
a hit in practice.
Regards,
Maciej
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On Tue, Feb 5, 2013 at 12:59 PM, Maciej Stachowiak m...@apple.com wrote: I have to say that the syntax weight of gyp files (quotes, commas, braces, brackets) makes them feel harder to read and edit than other build file formats designed for human consumption (obviously Xcode is worse). As much as I don't like Xcode project file, they are not to be edited by hand. It is easy to edit Xcode project file because there is a graphical tool that makes the task easy. There is no such tool for gyp so its syntax is important. TestExpectations is a success story of making unreadable files, readable. We should consider it for gyp if it were to become the standard tool for everyone. Benjamin ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
The run time comparisons were based on actual comparison with available
Python modules at the time. JSON was substantially slower than having the
Python interpreter slurp it up on its own. In fact, GYP started out reading
JSON input and we switched it to Python once we learned this. It was four
years ago, and I don’t have the numbers handy anymore.
In the end, I don’t think that the actual syntax (as opposed to the
structured data that it contains) is all that important. GYP input files
are fairly simple structured data consisting of very few types: dicts,
lists, strings, and infrequent bools and ints. Any extant or nascent format
that can handle these is a candidate. If there’s some kind of convergence
on a beautiful one that’s easy to maintain and quick to parse, there’s no
reason GYP can’t be taught to read it, even alongside the existing
JSON-esque format that’s fed to the Python interpreter. If this is the only
thing standing between a project and its adopting GYP, then it’s a really
easy one to get over.
On Tue, Feb 5, 2013 at 3:59 PM, Maciej Stachowiak m...@apple.com wrote:
On Feb 5, 2013, at 6:09 AM, Mark Mentovai m...@chromium.org wrote:
The parser (and the grammar) works the way it does because it’s just
Python—the whole thing can be slurped in quickly by the interpreter and
made available to GYP as data. It’s faster than a parser written in Python
for a similar language would be, and it’d likely take a binary Python
module to approach or exceed this speed. Developers don’t like waiting for
tools.
Is that a result based on testing, or just a projectiom? My guess would be
that most incremental builds do not add or remove any files, and therefore
do not need to regenerate the project files at all, so the speed of parsing
GYP files is unlikely to have a major impact, unless it's a truly massive
difference.
I have to say that the syntax weight of gyp files (quotes, commas, braces,
brackets) makes them feel harder to read and edit than other build file
formats designed for human consumption (obviously Xcode is worse).
Compare this bit of gyp for generating a derived source file:
{
'action_name': 'XMLNames',
'inputs': [
'../dom/make_names.pl',
'../xml/xmlattrs.in',
],
'outputs': [
'(SHARED_INTERMEDIATE_DIR)/webkit/XMLNames.cpp',
'(SHARED_INTERMEDIATE_DIR)/webkit/XMLNames.h',
],
'action': [
'python',
'scripts/action_makenames.py',
'@(_outputs)',
'--',
'@(_inputs)',
'--',
'--extraDefines', '(feature_defines)'
],
'msvs_cygwin_shell': 1,
},
vs the equivalent bit of DerivedSources.make:
XMLNames.cpp : dom/make_names.pl xml/xmlattrs.in
perl -I $(WebCore)/bindings/scripts $ --attrs $(WebCore)/xml/
xmlattrs.in
Or GNUMakefile.am:
DerivedSources/WebCore/XMLNames.cpp DerivedSources/WebCore/XMLNames.h:
$(WebCore)/dom/make_names.pl $(WebCore)/xml/xmlattrs.in
$(AM_V_GEN)$(PERL) -I$(WebCore)/bindings/scripts $ --attrs
$(WebCore)/xml/xmlattrs.in --outputDir $(GENSOURCES_WEBCORE)
Or DerivedSources.pri:
xmlnames.output = XMLNames.cpp
xmlnames.input = XML_NAMES
xmlnames.script = $$PWD/dom/make_names.pl
xmlnames.commands = perl -I$$PWD/bindings/scripts $$xmlnames.script
--attrs $$PWD/xml/xmlattrs.in --preprocessor \$${QMAKE_MOC} -E\
--outputDir ${QMAKE_FUNC_FILE_OUT_PATH}
GENERATORS += xmlnames
Gyp kind of seems like an outlier in heaviness of the syntax compared to
the other build systems.The irony is that Python itself doesn't use a lot
of punctuation in its syntax, so actual Python code looks much cleaner. I
like to read Python code, but I can't say I enjoy reading the Gyp syntax.
This makes me wonder if a Python DSL might be more legible than simply
using a dictionary.
Alternately, maybe it turns out that parsing a custom language is not that
huge a hit in practice.
Regards,
Maciej
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On 02/05/2013 12:59 PM, Maciej Stachowiak wrote: I have to say that the syntax weight of gyp files (quotes, commas, braces, brackets) makes them feel harder to read and edit than other build file formats designed for human consumption (obviously Xcode is worse). Ant is pretty verbose as well, partially due to the XML syntax. It is sometimes generated by tools (IDEs) and sometime written by hand. (Because Ant is implemented in Java and primarily aimed at building Java projects, it's not a good match for WebKit, regardless.) -- --Per Bothner per.both...@oracle.com p...@bothner.com http://per.bothner.com/ ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
I have looked at YAML off and on over the years, and I'm not sure that it would be much of an improvement in this case. I do believe that dropping the strict python syntax could make some things easier to read. I don't have a fully-baked proposal in mind, and I don't know what the perf hit would be. I will also note that -- like in many languages -- it's easy to write hard-to-read files. GYP has never had much of a style guide, and it probably could greatly benefit from one. It is important to make a fair apples-to-apples comparison for whatever we end up looking at. I suspect that given any tool or language, you can find some things that are easy to express and some that are harder. Even the existing chromium gyp files are not necessarily a model of clean code or good style. Also, one thing that's important to realize about GYP's syntax is that it is very set-based; the order things appear in the files is much less important than in many other languages (especially in how they handle conditionals, i.e., if/then/else). This has many advantages, but does also have some disadvantages. -- Dirk On Tue, Feb 5, 2013 at 12:16 PM, Eric Seidel e...@webkit.org wrote: I'm curious if YAML was ever considered? I have very limited experience with YAML, except for Google App Engine config files. It's very python parse-able? :) On Tue, Feb 5, 2013 at 11:55 AM, Mark Mentovai m...@chromium.org wrote: You’re not supposed to use arbitrary Python, it’s highly discouraged. We have a linter that keeps you from doing things you’re not supposed to do (like this), but it slows things down, so it’s not part of the “standard” GYP run that developers normally use. It can run as a pre-commit script or test on the bot or something else. Used as we’re using it, GYP basically is simple JSON, except the rules about commas are looser and # introduces comments. On Tue, Feb 5, 2013 at 2:47 PM, Ryosuke Niwa rn...@webkit.org wrote: On Tue, Feb 5, 2013 at 6:09 AM, Mark Mentovai m...@chromium.org wrote: The parser (and the grammar) works the way it does because it’s just Python This works great for people who like Python syntax but not for someone like myself who dislikes Python syntax. I also find it particularly annoying that people can use whatever Python constructs they want to use in GYP. It dramatically reduces language portability because you need to support quite a few Python constructs and quirks in order to correctly parse GYP. I personally would have much preferred for it be a simple JSON file. - R. Niwa ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On Tue, Feb 5, 2013 at 5:33 PM, Dirk Pranke dpra...@chromium.org wrote: I have looked at YAML off and on over the years, and I'm not sure that it would be much of an improvement in this case. I do believe that dropping the strict python syntax could make some things easier to read. I don't have a fully-baked proposal in mind, and I don't know what the perf hit would be. I will also note that -- like in many languages -- it's easy to write hard-to-read files. GYP has never had much of a style guide, and it probably could greatly benefit from one. I'm hopeful that we can come up with a syntax that doesn't require style guide. I'd even go as far as to say we shouldn't adopt a syntax that needs a style guide. - R. Niwa ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
GYP was written in Python to address point (b). Python was already part of the baseline requirements on all platforms, so we already had Python available everywhere we needed it. There are no dependencies on external binaries, and no compiled code needs to be checked in anywhere or maintained as part of a base image. As for point (a), you can easily have a top-level Makefile not generated by GYP that says “run GYP to produce the build files for whatever environment you like and then pass control to that build system to do the rest of the build. Developers who like it can use ninja for their own builds, and your bots can use Xcode or make if that’s a requirement (or if ninja doesn’t meet your requirements given point (b)). I’d be happy to discuss this further with anyone who’s interested in moving in this direction. On Mon, Feb 4, 2013 at 12:20 AM, Maciej Stachowiak m...@apple.com wrote: On Feb 3, 2013, at 8:18 PM, Mike Lawther mikelawt...@chromium.org wrote: Hi Maciej! On 31 January 2013 11:15, Maciej Stachowiak m...@apple.com wrote: It's far simplest for us if: (a) There is an Xcode project (or a Makefile) that builds the Mac port checked in to source control. (b) The generated project invokes only tools that are part of the default Mac OS X install. For b), does 'default' include an Xcode install? From my memory of setting up a Mac dev box Xcode was needed to compile. The limitation in (a) is all about how the build fleet is managed, and yes, the machines in question have the Mac OS X user install as well as developer tools installed, and it is ok to rely on that. To clarify: the things that are not kosher are submitting a binary tool as part of the module, downloading things as part of the build, or relying on something to be installed out of band before building. It is ok for the build to rely on scripts, so long as the relevant interpreter is part of the default install. Any such script would have to be in each directory under Source/ since these build separately without access to the source of the others. I should mention that there's a lot of interest right now at Apple in the possibility of switching to Gyp. We think it's likely closer to what we need than CMake, and we suspect there is probably somewhat better probability of modifying it if necessary. If we find someone at Apple who is able to work on such a project in the short term (not completely impossible), would any Google folks be interested in collaborating? Ideally we could use assistance from someone who (a) understands how the Chromium WebKit Gyp-based build works; and (b) is willing and able to change Gyp if necessary. Regards, Maciej mike On 31 January 2013 11:15, Maciej Stachowiak m...@apple.com wrote: On Jan 30, 2013, at 3:24 PM, Dirk Pranke dpra...@chromium.org wrote: On Wed, Jan 30, 2013 at 1:50 PM, Filip Pizlo fpi...@apple.com wrote: Thanks for sharing this. On Jan 30, 2013, at 1:28 PM, Eric Seidel e...@webkit.org wrote: I wish we only had one build system (it were easy to add/remove/move files). I believe changes like http://trac.webkit.org/changeset/74849 are an unhealthy sign for the project. Adam is not the only person who has chosen to empty files instead of removing them. The pain of updating 8 build system is so great, we jump through hoops to avoid it. Which means it took us months to move JavaScriptCore/wtf to WTF, and will take us years to kill WebCore/platform. +1 This is a hard problem. It is a problem worth solving. Do you have more thoughts on this, particularly since you know quite well how both Xcode and gyp work? I suspect this is one of those things that it would be hard to achieve consensus on since there are so many stakeholders. But it may be fruitful to have a what if discussion about what this might look like. I think we can solve this problem if we agree that we want to. I think we haven't in the past mostly because we couldn't reach a consensus that it was worth solving enough to really try. I would love to see this fixed and would be glad to work on it. I think we should at least pursue this far enough to fully understand what our options are and what the costs and tradeoffs might be; does anyone disagree, and is anyone else willing to help pitch in? I think there are several possible ways we could solve this. One would be to switch to a common meta-build system. My understanding is that Apple's internal production build processes impose certain constraints here that I don't fully understand, but I know we've discussed the possibility of checking in generated project files as a workaround. Maybe there are other options as well to those constraints? I would love to discuss this further w/ someone from Apple ... It's far simplest for us if: (a) There is an Xcode project (or a Makefile) that builds the Mac port checked in to source control. (b)
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On Feb 4, 2013, at 10:46 AM, Mark Mentovai m...@chromium.org wrote: GYP was written in Python to address point (b). Python was already part of the baseline requirements on all platforms, so we already had Python available everywhere we needed it. There are no dependencies on external binaries, and no compiled code needs to be checked in anywhere or maintained as part of a base image. As for point (a), you can easily have a top-level Makefile not generated by GYP that says “run GYP to produce the build files for whatever environment you like and then pass control to that build system to do the rest of the build. Developers who like it can use ninja for their own builds, and your bots can use Xcode or make if that’s a requirement (or if ninja doesn’t meet your requirements given point (b)). Checking in the generated Xcode projects is another alternative. The Makefile might be better for the reasons you suggest, though. I'm reasonably confident at this point that Gyp can meet our hard requirements. Our remaining issues are finding time to do it and comprehensibility/readability of the syntax. I’d be happy to discuss this further with anyone who’s interested in moving in this direction. That would be much appreciated. I'll ping you once we have somebody to work on this. Regards, Maciej ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On 2013-02-04, at 10:57, Maciej Stachowiak m...@apple.com wrote: On Feb 4, 2013, at 10:46 AM, Mark Mentovai m...@chromium.org wrote: GYP was written in Python to address point (b). Python was already part of the baseline requirements on all platforms, so we already had Python available everywhere we needed it. There are no dependencies on external binaries, and no compiled code needs to be checked in anywhere or maintained as part of a base image. As for point (a), you can easily have a top-level Makefile not generated by GYP that says “run GYP to produce the build files for whatever environment you like and then pass control to that build system to do the rest of the build. Developers who like it can use ninja for their own builds, and your bots can use Xcode or make if that’s a requirement (or if ninja doesn’t meet your requirements given point (b)). Checking in the generated Xcode projects is another alternative. The Makefile might be better for the reasons you suggest, though. It’s not immediately obvious to me that a Makefile that ran GYP would be suitable for our production builds. I think it would be materially the same as building with something other than Xcode, in that it would limit the integration with the rest of the Apple build infrastructure. - Mark ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On Sun, Feb 3, 2013 at 9:20 PM, Maciej Stachowiak m...@apple.com wrote: On Feb 3, 2013, at 8:18 PM, Mike Lawther mikelawt...@chromium.org wrote: Hi Maciej! On 31 January 2013 11:15, Maciej Stachowiak m...@apple.com wrote: It's far simplest for us if: (a) There is an Xcode project (or a Makefile) that builds the Mac port checked in to source control. (b) The generated project invokes only tools that are part of the default Mac OS X install. For b), does 'default' include an Xcode install? From my memory of setting up a Mac dev box Xcode was needed to compile. The limitation in (a) is all about how the build fleet is managed, and yes, the machines in question have the Mac OS X user install as well as developer tools installed, and it is ok to rely on that. To clarify: the things that are not kosher are submitting a binary tool as part of the module, downloading things as part of the build, or relying on something to be installed out of band before building. It is ok for the build to rely on scripts, so long as the relevant interpreter is part of the default install. Any such script would have to be in each directory under Source/ since these build separately without access to the source of the others. I should mention that there's a lot of interest right now at Apple in the possibility of switching to Gyp. We think it's likely closer to what we need than CMake, and we suspect there is probably somewhat better probability of modifying it if necessary. If we find someone at Apple who is able to work on such a project in the short term (not completely impossible), would any Google folks be interested in collaborating? Ideally we could use assistance from someone who (a) understands how the Chromium WebKit Gyp-based build works; and (b) is willing and able to change Gyp if necessary. Mark is the gyp-guru, but I would be happy to help out as well. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
Hi Maciej! On 31 January 2013 11:15, Maciej Stachowiak m...@apple.com wrote: It's far simplest for us if: (a) There is an Xcode project (or a Makefile) that builds the Mac port checked in to source control. (b) The generated project invokes only tools that are part of the default Mac OS X install. For b), does 'default' include an Xcode install? From my memory of setting up a Mac dev box Xcode was needed to compile. mike On 31 January 2013 11:15, Maciej Stachowiak m...@apple.com wrote: On Jan 30, 2013, at 3:24 PM, Dirk Pranke dpra...@chromium.org wrote: On Wed, Jan 30, 2013 at 1:50 PM, Filip Pizlo fpi...@apple.com wrote: Thanks for sharing this. On Jan 30, 2013, at 1:28 PM, Eric Seidel e...@webkit.org wrote: I wish we only had one build system (it were easy to add/remove/move files). I believe changes like http://trac.webkit.org/changeset/74849 are an unhealthy sign for the project. Adam is not the only person who has chosen to empty files instead of removing them. The pain of updating 8 build system is so great, we jump through hoops to avoid it. Which means it took us months to move JavaScriptCore/wtf to WTF, and will take us years to kill WebCore/platform. +1 This is a hard problem. It is a problem worth solving. Do you have more thoughts on this, particularly since you know quite well how both Xcode and gyp work? I suspect this is one of those things that it would be hard to achieve consensus on since there are so many stakeholders. But it may be fruitful to have a what if discussion about what this might look like. I think we can solve this problem if we agree that we want to. I think we haven't in the past mostly because we couldn't reach a consensus that it was worth solving enough to really try. I would love to see this fixed and would be glad to work on it. I think we should at least pursue this far enough to fully understand what our options are and what the costs and tradeoffs might be; does anyone disagree, and is anyone else willing to help pitch in? I think there are several possible ways we could solve this. One would be to switch to a common meta-build system. My understanding is that Apple's internal production build processes impose certain constraints here that I don't fully understand, but I know we've discussed the possibility of checking in generated project files as a workaround. Maybe there are other options as well to those constraints? I would love to discuss this further w/ someone from Apple ... It's far simplest for us if: (a) There is an Xcode project (or a Makefile) that builds the Mac port checked in to source control. (b) The generated project invokes only tools that are part of the default Mac OS X install. It may not be completely impossible to violate these requirements but it will require a lot of bureaucracy. (Also, just to get this out of the way, I don't think gyp needs to be the solution). Another alternative would be to write a script that did support at least the common use cases (add/move/delete files). There have been attempts in the past, but they have foundered on at least some perceived skepticism over how well this would work w/ XCode. That said, I don't think we've really pushed this to see. At some point this script might turn into a meta-meta-build system, which might be silly but also be the shortest path to the finish line. I suggest if there is interest in this we can start a new thread to discuss further ... My preference would be to use a common meta-build system with a comfortably human-readable and human-editable syntax, and checked in generated project files for those ports that need them. I think a key to making this work is to get Chromium and the Apple Mac port onto a common build system, which will probably require both Google and Apple ponying up at least one person to work on this project for a reasonable period of time. I think the plausible meta-build-systems to use would be CMake and Gyp, but in both cases it may be necessary to modify them to work well for everyone. I'd also like to add that I think a key related issue to common build system is common feature configuration. The many different ways ports control their feature flags is super confusing. I've long wanted to implement common configuration management, but have not had time. Cheers, Maciej ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On Feb 3, 2013, at 8:18 PM, Mike Lawther mikelawt...@chromium.org wrote: Hi Maciej! On 31 January 2013 11:15, Maciej Stachowiak m...@apple.com wrote: It's far simplest for us if: (a) There is an Xcode project (or a Makefile) that builds the Mac port checked in to source control. (b) The generated project invokes only tools that are part of the default Mac OS X install. For b), does 'default' include an Xcode install? From my memory of setting up a Mac dev box Xcode was needed to compile. The limitation in (a) is all about how the build fleet is managed, and yes, the machines in question have the Mac OS X user install as well as developer tools installed, and it is ok to rely on that. To clarify: the things that are not kosher are submitting a binary tool as part of the module, downloading things as part of the build, or relying on something to be installed out of band before building. It is ok for the build to rely on scripts, so long as the relevant interpreter is part of the default install. Any such script would have to be in each directory under Source/ since these build separately without access to the source of the others. I should mention that there's a lot of interest right now at Apple in the possibility of switching to Gyp. We think it's likely closer to what we need than CMake, and we suspect there is probably somewhat better probability of modifying it if necessary. If we find someone at Apple who is able to work on such a project in the short term (not completely impossible), would any Google folks be interested in collaborating? Ideally we could use assistance from someone who (a) understands how the Chromium WebKit Gyp-based build works; and (b) is willing and able to change Gyp if necessary. Regards, Maciej mike On 31 January 2013 11:15, Maciej Stachowiak m...@apple.com wrote: On Jan 30, 2013, at 3:24 PM, Dirk Pranke dpra...@chromium.org wrote: On Wed, Jan 30, 2013 at 1:50 PM, Filip Pizlo fpi...@apple.com wrote: Thanks for sharing this. On Jan 30, 2013, at 1:28 PM, Eric Seidel e...@webkit.org wrote: I wish we only had one build system (it were easy to add/remove/move files). I believe changes like http://trac.webkit.org/changeset/74849 are an unhealthy sign for the project. Adam is not the only person who has chosen to empty files instead of removing them. The pain of updating 8 build system is so great, we jump through hoops to avoid it. Which means it took us months to move JavaScriptCore/wtf to WTF, and will take us years to kill WebCore/platform. +1 This is a hard problem. It is a problem worth solving. Do you have more thoughts on this, particularly since you know quite well how both Xcode and gyp work? I suspect this is one of those things that it would be hard to achieve consensus on since there are so many stakeholders. But it may be fruitful to have a what if discussion about what this might look like. I think we can solve this problem if we agree that we want to. I think we haven't in the past mostly because we couldn't reach a consensus that it was worth solving enough to really try. I would love to see this fixed and would be glad to work on it. I think we should at least pursue this far enough to fully understand what our options are and what the costs and tradeoffs might be; does anyone disagree, and is anyone else willing to help pitch in? I think there are several possible ways we could solve this. One would be to switch to a common meta-build system. My understanding is that Apple's internal production build processes impose certain constraints here that I don't fully understand, but I know we've discussed the possibility of checking in generated project files as a workaround. Maybe there are other options as well to those constraints? I would love to discuss this further w/ someone from Apple ... It's far simplest for us if: (a) There is an Xcode project (or a Makefile) that builds the Mac port checked in to source control. (b) The generated project invokes only tools that are part of the default Mac OS X install. It may not be completely impossible to violate these requirements but it will require a lot of bureaucracy. (Also, just to get this out of the way, I don't think gyp needs to be the solution). Another alternative would be to write a script that did support at least the common use cases (add/move/delete files). There have been attempts in the past, but they have foundered on at least some perceived skepticism over how well this would work w/ XCode. That said, I don't think we've really pushed this to see. At some point this script might turn into a meta-meta-build system, which might be silly but also be the shortest path to the finish line. I suggest if there is interest in this we can start a new thread to discuss further ... My preference would be to use a common
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
I think one important aspect of build systems was not considered yet int this conversation: speed. The time an incremental build takes has a great effect on developer productivity. I don't think any of the meta-build systems we use does a great job here - although I only have experiences with qmake, cmake and autotools (and I don't have an SSD that could help somewhat). The technic I found useful here is to avoid calling build-webkit always and instead just rebuild the subproject you have edited, so I think it is important to have a build system that supports it. Let me share my experiences here. With qmake nowadays this work perfectly. The developer build is producing a shared library for every subdir (WTF, JavaScriptCore, WebCore, WebKit2), which means you only need to call make in the specific subdirectory (i.e. if I only touched WebKit2 files I do make -C WebKitBuild/Release/Source/WebKit2 which is pretty quick). Still WebCore is so big that make is quite slowly find out the files you actually edited and need to be rebuilt. What could help here is to devide WebCore into smaller parts, like the ongoing work of extracting Platform. Maybe the next logical candidate could be svg (I don't have real knowledge about how feasible it is). Note that I don't come up with qmake because I would like to recommend it as the one and only build system (in fact it has a ridiculously inconvenient syntax, and a lot of bugs), just as an example. With Cmake fast incremental rebuilds are also possible, maybe in a bit more complicated way. When working with the EFL port I found a quick rule for WebKit2 in the generated makefile. Although the makefiles are usually call back to Cmake, and make is not faster than build-webkit, if you use the special fast target, which is something like eflWebKit/fast (i.e. make -C WebKitBuild/Release/Source/WebKit2 eflWebKit/fast), it will not check dependencies but just rebuild the files that have changed. I did not find a similar thing for WebCore, I guess because it is not built as a shared library. What I dislike in Cmake is that I am disappointed by how slow a normal incremental build with it (i.e. build-webkit). qmake is not faster at all, but it generate plain makefiles that typically no call back to qmake if not specified explicitly to do so, and directly calling make is faster, yet it can handle most of the non-trivial changes, for example editing a generator file. I don't know gyp, so I wonder about how would it do in this comparison (but I guess it generates simple makefiles as well, so it's similar than qmake in this manner.) I hope I added something to this conversation that is worth to consider with my late nightly brain dump. -kbalazs ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
Ninja has extremely fast incremental builds and can be generated by GYP. Here are some stats from a year ago: https://plus.google.com/101038813433650812235/posts/irc26fhRtPC Ninja has gotten even faster since then. If you're interested in trying it out, you can play around with incremental builds of the Chromium port on Mac or Linux. Adam On Fri, Feb 1, 2013 at 4:58 PM, Balazs Kelemen kbal...@webkit.org wrote: I think one important aspect of build systems was not considered yet int this conversation: speed. The time an incremental build takes has a great effect on developer productivity. I don't think any of the meta-build systems we use does a great job here - although I only have experiences with qmake, cmake and autotools (and I don't have an SSD that could help somewhat). The technic I found useful here is to avoid calling build-webkit always and instead just rebuild the subproject you have edited, so I think it is important to have a build system that supports it. Let me share my experiences here. With qmake nowadays this work perfectly. The developer build is producing a shared library for every subdir (WTF, JavaScriptCore, WebCore, WebKit2), which means you only need to call make in the specific subdirectory (i.e. if I only touched WebKit2 files I do make -C WebKitBuild/Release/Source/WebKit2 which is pretty quick). Still WebCore is so big that make is quite slowly find out the files you actually edited and need to be rebuilt. What could help here is to devide WebCore into smaller parts, like the ongoing work of extracting Platform. Maybe the next logical candidate could be svg (I don't have real knowledge about how feasible it is). Note that I don't come up with qmake because I would like to recommend it as the one and only build system (in fact it has a ridiculously inconvenient syntax, and a lot of bugs), just as an example. With Cmake fast incremental rebuilds are also possible, maybe in a bit more complicated way. When working with the EFL port I found a quick rule for WebKit2 in the generated makefile. Although the makefiles are usually call back to Cmake, and make is not faster than build-webkit, if you use the special fast target, which is something like eflWebKit/fast (i.e. make -C WebKitBuild/Release/Source/WebKit2 eflWebKit/fast), it will not check dependencies but just rebuild the files that have changed. I did not find a similar thing for WebCore, I guess because it is not built as a shared library. What I dislike in Cmake is that I am disappointed by how slow a normal incremental build with it (i.e. build-webkit). qmake is not faster at all, but it generate plain makefiles that typically no call back to qmake if not specified explicitly to do so, and directly calling make is faster, yet it can handle most of the non-trivial changes, for example editing a generator file. I don't know gyp, so I wonder about how would it do in this comparison (but I guess it generates simple makefiles as well, so it's similar than qmake in this manner.) I hope I added something to this conversation that is worth to consider with my late nightly brain dump. -kbalazs ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev On Fri, Feb 1, 2013 at 4:58 PM, Balazs Kelemen kbal...@webkit.org wrote: I think one important aspect of build systems was not considered yet int this conversation: speed. The time an incremental build takes has a great effect on developer productivity. I don't think any of the meta-build systems we use does a great job here - although I only have experiences with qmake, cmake and autotools (and I don't have an SSD that could help somewhat). The technic I found useful here is to avoid calling build-webkit always and instead just rebuild the subproject you have edited, so I think it is important to have a build system that supports it. Let me share my experiences here. With qmake nowadays this work perfectly. The developer build is producing a shared library for every subdir (WTF, JavaScriptCore, WebCore, WebKit2), which means you only need to call make in the specific subdirectory (i.e. if I only touched WebKit2 files I do make -C WebKitBuild/Release/Source/WebKit2 which is pretty quick). Still WebCore is so big that make is quite slowly find out the files you actually edited and need to be rebuilt. What could help here is to devide WebCore into smaller parts, like the ongoing work of extracting Platform. Maybe the next logical candidate could be svg (I don't have real knowledge about how feasible it is). Note that I don't come up with qmake because I would like to recommend it as the one and only build system (in fact it has a ridiculously inconvenient syntax, and a lot of bugs), just as an example. With Cmake fast incremental rebuilds are also possible, maybe in a bit more complicated way. When
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On Sat, Feb 2, 2013 at 4:58 PM, Adam Barth aba...@webkit.org wrote: Ninja has extremely fast incremental builds and can be generated by GYP. Here are some stats from a year ago: https://plus.google.com/101038813433650812235/posts/irc26fhRtPC Ninja has gotten even faster since then. If you're interested in trying it out, you can play around with incremental builds of the Chromium port on Mac or Linux. You can also look at the build output from the chromium bots. Empty build in 1s: http://build.webkit.org/builders/Chromium%20Linux%20Release/builds/66807/steps/compile-webkit/logs/stdio Build with a few files changed in 15s: http://build.webkit.org/builders/Chromium%20Linux%20Release/builds/66800/steps/compile-webkit/logs/stdio …and this is on fairly slow bots. On my SSD-equipped laptop, I can do incremental rebuilds of all of chrome after touching one (cpp or mm) file in 2-6s. Adam On Fri, Feb 1, 2013 at 4:58 PM, Balazs Kelemen kbal...@webkit.org wrote: I think one important aspect of build systems was not considered yet int this conversation: speed. The time an incremental build takes has a great effect on developer productivity. I don't think any of the meta-build systems we use does a great job here - although I only have experiences with qmake, cmake and autotools (and I don't have an SSD that could help somewhat). The technic I found useful here is to avoid calling build-webkit always and instead just rebuild the subproject you have edited, so I think it is important to have a build system that supports it. Let me share my experiences here. With qmake nowadays this work perfectly. The developer build is producing a shared library for every subdir (WTF, JavaScriptCore, WebCore, WebKit2), which means you only need to call make in the specific subdirectory (i.e. if I only touched WebKit2 files I do make -C WebKitBuild/Release/Source/WebKit2 which is pretty quick). Still WebCore is so big that make is quite slowly find out the files you actually edited and need to be rebuilt. What could help here is to devide WebCore into smaller parts, like the ongoing work of extracting Platform. Maybe the next logical candidate could be svg (I don't have real knowledge about how feasible it is). Note that I don't come up with qmake because I would like to recommend it as the one and only build system (in fact it has a ridiculously inconvenient syntax, and a lot of bugs), just as an example. With Cmake fast incremental rebuilds are also possible, maybe in a bit more complicated way. When working with the EFL port I found a quick rule for WebKit2 in the generated makefile. Although the makefiles are usually call back to Cmake, and make is not faster than build-webkit, if you use the special fast target, which is something like eflWebKit/fast (i.e. make -C WebKitBuild/Release/Source/WebKit2 eflWebKit/fast), it will not check dependencies but just rebuild the files that have changed. I did not find a similar thing for WebCore, I guess because it is not built as a shared library. What I dislike in Cmake is that I am disappointed by how slow a normal incremental build with it (i.e. build-webkit). qmake is not faster at all, but it generate plain makefiles that typically no call back to qmake if not specified explicitly to do so, and directly calling make is faster, yet it can handle most of the non-trivial changes, for example editing a generator file. I don't know gyp, so I wonder about how would it do in this comparison (but I guess it generates simple makefiles as well, so it's similar than qmake in this manner.) I hope I added something to this conversation that is worth to consider with my late nightly brain dump. -kbalazs ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev On Fri, Feb 1, 2013 at 4:58 PM, Balazs Kelemen kbal...@webkit.org wrote: I think one important aspect of build systems was not considered yet int this conversation: speed. The time an incremental build takes has a great effect on developer productivity. I don't think any of the meta-build systems we use does a great job here - although I only have experiences with qmake, cmake and autotools (and I don't have an SSD that could help somewhat). The technic I found useful here is to avoid calling build-webkit always and instead just rebuild the subproject you have edited, so I think it is important to have a build system that supports it. Let me share my experiences here. With qmake nowadays this work perfectly. The developer build is producing a shared library for every subdir (WTF, JavaScriptCore, WebCore, WebKit2), which means you only need to call make in the specific subdirectory (i.e. if I only touched WebKit2 files I do make -C WebKitBuild/Release/Source/WebKit2 which is pretty quick). Still WebCore is so big that make is quite slowly find out the files you
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
+1 Ninja is beyond-words amazing. http://martine.github.com/ninja/ For better or worse, it is not designed to use human-editable build files, but rather to be used by a meta build system, like GYP or CMake. So using ninja is really an orthogonal discussion to the single build system discussion for WebKit. :) Were the WebKit project to convert to using a single meta-build system, ninja would become an option many users might choose. I'm told most Chromium hackers have GYP set to output ninja files these days, with the exception of some folks who still want the MSVC build environment. For WebKit ports already using CMake, they should definitely try ninja today! Anyway, my wish was not about arguing for a specific build solution. I'm instead noting that for the project to continue to move quickly, we need to stop needing to edit 8 build systems for every file move/addition. Whether that's GYP or CMake or something else, I don't really care. Adam and I tried GYP-for-WebKIt a while back. But any of these solutions will require buy-in from Apple, as they will have to do the largest amount of work converting to use something other than XCode. On Sat, Feb 2, 2013 at 8:20 PM, Nico Weber tha...@chromium.org wrote: On Sat, Feb 2, 2013 at 4:58 PM, Adam Barth aba...@webkit.org wrote: Ninja has extremely fast incremental builds and can be generated by GYP. Here are some stats from a year ago: https://plus.google.com/101038813433650812235/posts/irc26fhRtPC Ninja has gotten even faster since then. If you're interested in trying it out, you can play around with incremental builds of the Chromium port on Mac or Linux. You can also look at the build output from the chromium bots. Empty build in 1s: http://build.webkit.org/builders/Chromium%20Linux%20Release/builds/66807/steps/compile-webkit/logs/stdio Build with a few files changed in 15s: http://build.webkit.org/builders/Chromium%20Linux%20Release/builds/66800/steps/compile-webkit/logs/stdio …and this is on fairly slow bots. On my SSD-equipped laptop, I can do incremental rebuilds of all of chrome after touching one (cpp or mm) file in 2-6s. Adam On Fri, Feb 1, 2013 at 4:58 PM, Balazs Kelemen kbal...@webkit.org wrote: I think one important aspect of build systems was not considered yet int this conversation: speed. The time an incremental build takes has a great effect on developer productivity. I don't think any of the meta-build systems we use does a great job here - although I only have experiences with qmake, cmake and autotools (and I don't have an SSD that could help somewhat). The technic I found useful here is to avoid calling build-webkit always and instead just rebuild the subproject you have edited, so I think it is important to have a build system that supports it. Let me share my experiences here. With qmake nowadays this work perfectly. The developer build is producing a shared library for every subdir (WTF, JavaScriptCore, WebCore, WebKit2), which means you only need to call make in the specific subdirectory (i.e. if I only touched WebKit2 files I do make -C WebKitBuild/Release/Source/WebKit2 which is pretty quick). Still WebCore is so big that make is quite slowly find out the files you actually edited and need to be rebuilt. What could help here is to devide WebCore into smaller parts, like the ongoing work of extracting Platform. Maybe the next logical candidate could be svg (I don't have real knowledge about how feasible it is). Note that I don't come up with qmake because I would like to recommend it as the one and only build system (in fact it has a ridiculously inconvenient syntax, and a lot of bugs), just as an example. With Cmake fast incremental rebuilds are also possible, maybe in a bit more complicated way. When working with the EFL port I found a quick rule for WebKit2 in the generated makefile. Although the makefiles are usually call back to Cmake, and make is not faster than build-webkit, if you use the special fast target, which is something like eflWebKit/fast (i.e. make -C WebKitBuild/Release/Source/WebKit2 eflWebKit/fast), it will not check dependencies but just rebuild the files that have changed. I did not find a similar thing for WebCore, I guess because it is not built as a shared library. What I dislike in Cmake is that I am disappointed by how slow a normal incremental build with it (i.e. build-webkit). qmake is not faster at all, but it generate plain makefiles that typically no call back to qmake if not specified explicitly to do so, and directly calling make is faster, yet it can handle most of the non-trivial changes, for example editing a generator file. I don't know gyp, so I wonder about how would it do in this comparison (but I guess it generates simple makefiles as well, so it's similar than qmake in this manner.) I hope I added something to this conversation that is worth to consider with my late nightly brain dump. -kbalazs
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On 2013-01-30, at 17:14, Dirk Pranke dpra...@chromium.org wrote: On Wed, Jan 30, 2013 at 4:15 PM, Maciej Stachowiak m...@apple.com wrote: On Jan 30, 2013, at 3:24 PM, Dirk Pranke dpra...@chromium.org wrote: On Wed, Jan 30, 2013 at 1:50 PM, Filip Pizlo fpi...@apple.com wrote: Thanks for sharing this. On Jan 30, 2013, at 1:28 PM, Eric Seidel e...@webkit.org wrote: I wish we only had one build system (it were easy to add/remove/move files). I believe changes like http://trac.webkit.org/changeset/74849 are an unhealthy sign for the project. Adam is not the only person who has chosen to empty files instead of removing them. The pain of updating 8 build system is so great, we jump through hoops to avoid it. Which means it took us months to move JavaScriptCore/wtf to WTF, and will take us years to kill WebCore/platform. +1 This is a hard problem. It is a problem worth solving. Do you have more thoughts on this, particularly since you know quite well how both Xcode and gyp work? I suspect this is one of those things that it would be hard to achieve consensus on since there are so many stakeholders. But it may be fruitful to have a what if discussion about what this might look like. I think we can solve this problem if we agree that we want to. I think we haven't in the past mostly because we couldn't reach a consensus that it was worth solving enough to really try. I would love to see this fixed and would be glad to work on it. I think we should at least pursue this far enough to fully understand what our options are and what the costs and tradeoffs might be; does anyone disagree, and is anyone else willing to help pitch in? I think there are several possible ways we could solve this. One would be to switch to a common meta-build system. My understanding is that Apple's internal production build processes impose certain constraints here that I don't fully understand, but I know we've discussed the possibility of checking in generated project files as a workaround. Maybe there are other options as well to those constraints? I would love to discuss this further w/ someone from Apple ... It's far simplest for us if: (a) There is an Xcode project (or a Makefile) that builds the Mac port checked in to source control. (b) The generated project invokes only tools that are part of the default Mac OS X install. It may not be completely impossible to violate these requirements but it will require a lot of bureaucracy. (Also, just to get this out of the way, I don't think gyp needs to be the solution). Another alternative would be to write a script that did support at least the common use cases (add/move/delete files). There have been attempts in the past, but they have foundered on at least some perceived skepticism over how well this would work w/ XCode. That said, I don't think we've really pushed this to see. At some point this script might turn into a meta-meta-build system, which might be silly but also be the shortest path to the finish line. I suggest if there is interest in this we can start a new thread to discuss further ... My preference would be to use a common meta-build system with a comfortably human-readable and human-editable syntax, and checked in generated project files for those ports that need them. I think a key to making this work is to get Chromium and the Apple Mac port onto a common build system, which will probably require both Google and Apple ponying up at least one person to work on this project for a reasonable period of time. I think the plausible meta-build-systems to use would be CMake and Gyp, but in both cases it may be necessary to modify them to work well for everyone. Premake might also be an option, though I wouldn't necessarily vote for it. Gyp's syntax is ... awkward ... but apart from that might just work for checking in generated apple mac xcode projects. We should try it (shouldn't be too hard, since abarth had it working at one point). I think Eric and/or Adam had JavaScriptCore building with gyp at one point. I’m not sure if they ever got to the other projects. I would consider changing or improving gyp's syntax to be on the table if it was needed to reach the goal. For what it’s worth, I also find the gyp syntax to be unpleasant. It feels as though it was optimized for being processed by a machine rather than for being written and maintained by humans. CMake was originally considered a non-starter for chromium since the XCode and VS projects it produced didn't look or feel like native projects. However, we've since switched to ninja for most things so I'm less sure how important this is now. I've never been a huge fan of the CMake syntax but I haven't used it enough to really have an informed opinion. Generating well-structured Xcode projects is still something that is important for any meta build system that we would consider
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On Thu, Jan 31, 2013 at 12:25 AM, Mark Rowe mr...@apple.com wrote: On 2013-01-30, at 17:14, Dirk Pranke dpra...@chromium.org wrote: On Wed, Jan 30, 2013 at 4:15 PM, Maciej Stachowiak m...@apple.com wrote: On Jan 30, 2013, at 3:24 PM, Dirk Pranke dpra...@chromium.org wrote: On Wed, Jan 30, 2013 at 1:50 PM, Filip Pizlo fpi...@apple.com wrote: Thanks for sharing this. On Jan 30, 2013, at 1:28 PM, Eric Seidel e...@webkit.org wrote: I wish we only had one build system (it were easy to add/remove/move files). I believe changes like http://trac.webkit.org/changeset/74849 are an unhealthy sign for the project. Adam is not the only person who has chosen to empty files instead of removing them. The pain of updating 8 build system is so great, we jump through hoops to avoid it. Which means it took us months to move JavaScriptCore/wtf to WTF, and will take us years to kill WebCore/platform. +1 This is a hard problem. It is a problem worth solving. Do you have more thoughts on this, particularly since you know quite well how both Xcode and gyp work? I suspect this is one of those things that it would be hard to achieve consensus on since there are so many stakeholders. But it may be fruitful to have a what if discussion about what this might look like. I think we can solve this problem if we agree that we want to. I think we haven't in the past mostly because we couldn't reach a consensus that it was worth solving enough to really try. I would love to see this fixed and would be glad to work on it. I think we should at least pursue this far enough to fully understand what our options are and what the costs and tradeoffs might be; does anyone disagree, and is anyone else willing to help pitch in? I think there are several possible ways we could solve this. One would be to switch to a common meta-build system. My understanding is that Apple's internal production build processes impose certain constraints here that I don't fully understand, but I know we've discussed the possibility of checking in generated project files as a workaround. Maybe there are other options as well to those constraints? I would love to discuss this further w/ someone from Apple ... It's far simplest for us if: (a) There is an Xcode project (or a Makefile) that builds the Mac port checked in to source control. (b) The generated project invokes only tools that are part of the default Mac OS X install. It may not be completely impossible to violate these requirements but it will require a lot of bureaucracy. (Also, just to get this out of the way, I don't think gyp needs to be the solution). Another alternative would be to write a script that did support at least the common use cases (add/move/delete files). There have been attempts in the past, but they have foundered on at least some perceived skepticism over how well this would work w/ XCode. That said, I don't think we've really pushed this to see. At some point this script might turn into a meta-meta-build system, which might be silly but also be the shortest path to the finish line. I suggest if there is interest in this we can start a new thread to discuss further ... My preference would be to use a common meta-build system with a comfortably human-readable and human-editable syntax, and checked in generated project files for those ports that need them. I think a key to making this work is to get Chromium and the Apple Mac port onto a common build system, which will probably require both Google and Apple ponying up at least one person to work on this project for a reasonable period of time. I think the plausible meta-build-systems to use would be CMake and Gyp, but in both cases it may be necessary to modify them to work well for everyone. Premake might also be an option, though I wouldn't necessarily vote for it. Gyp's syntax is ... awkward ... but apart from that might just work for checking in generated apple mac xcode projects. We should try it (shouldn't be too hard, since abarth had it working at one point). I think Eric and/or Adam had JavaScriptCore building with gyp at one point. I’m not sure if they ever got to the other projects. We had JavaScriptCore and WebCore working. (We also had JavaScriptGlue, but that doesn't exist anymore.) I don't remember if we had WebKit/mac working. (WebKit2 didn't exist at the time.) I would consider changing or improving gyp's syntax to be on the table if it was needed to reach the goal. For what it’s worth, I also find the gyp syntax to be unpleasant. It feels as though it was optimized for being processed by a machine rather than for being written and maintained by humans. Unlike xcodeproj files. :) CMake was originally considered a non-starter for chromium since the XCode and VS projects it produced didn't look or feel like native projects. However, we've since switched to ninja for most things so I'm less sure how
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
Hi, Am 31.01.2013 um 09:25 schrieb Mark Rowe: CMake was originally considered a non-starter for chromium since the XCode and VS projects it produced didn't look or feel like native projects. However, we've since switched to ninja for most things so I'm less sure how important this is now. I've never been a huge fan of the CMake syntax but I haven't used it enough to really have an informed opinion. Generating well-structured Xcode projects is still something that is important for any meta build system that we would consider adopting for the Mac. It would be substantially more challenging for us to support an alternative system for building our production builds. If there are substantial advantages to using ninja for engineering builds then we may wish to support both to get the best behavior in each environment. The non-natively structured Xcode projects is one thing that turned me off CMake when I looked at it in the past. I’m not sure if this has improved since then. If not, then I don’t think CMake is worth spending much time on. If you want to commit the generated projects CMake isn't an option anyway, since it uses absolute paths and also requires a cmake binary on the machine used for compiling (to support all CMake features across all generators). This tight integration is usually a big win, since the CMakeLists.txt files are part of the build system they can be changed directly in the IDE and update the project without invoking external tools. Only for my personal interest: What do you mean exactly with non-natively structured Xcode projects? CMake aims to support a native IDE feel as far as possible. Would be great if you explain that to me (off-list) to get CMake improved for XCode? Regarding (b) The generated project invokes only tools that are part of the default Mac OS X install: invoking tools that are checked into the WK repo is also possible, right, since we invoke scripts now? So, hypothetically, could we check in a copy of the ninja binary and build with that? Checking in binaries isn’t an option for us, and isn’t a particularly scalable approach anyway given the number of platforms that WebKit can build on. If we require an external tool as a dependency to build WebKit from source then we’d need to check in the source for the tool and build it prior to building WebKit proper. This obviously introduces more complexity so it would be preferable to keep the dependencies to a minimum. Maybe you can check in the ninja source and compile it during build. It only requires python (and hopefully installed build tools) to get built. -- Patrick ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On 2013-01-31, at 00:48, Adam Barth aba...@webkit.org wrote: I would consider changing or improving gyp's syntax to be on the table if it was needed to reach the goal. For what it’s worth, I also find the gyp syntax to be unpleasant. It feels as though it was optimized for being processed by a machine rather than for being written and maintained by humans. Unlike xcodeproj files. :) Don’t get me wrong, Xcode projects suck for hand-editing too. However, they’re not intended to be edited by hand. Gyp files are, and so the expected level of human readability is much higher. - Mark ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On Thu, Jan 31, 2013 at 9:53 AM, Mark Rowe mr...@apple.com wrote: On 2013-01-31, at 00:48, Adam Barth aba...@webkit.org wrote: I would consider changing or improving gyp's syntax to be on the table if it was needed to reach the goal. For what it’s worth, I also find the gyp syntax to be unpleasant. It feels as though it was optimized for being processed by a machine rather than for being written and maintained by humans. Unlike xcodeproj files. :) Don’t get me wrong, Xcode projects suck for hand-editing too. However, they’re not intended to be edited by hand. Gyp files are, and so the expected level of human readability is much higher. Many of us are actually editing the Xcode projects by hand, either because they don't have Xcode or don't know how to use it. (Yes, that includes coming up with a bunch of new UUIDs by hand) best -jochen ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On Thu, Jan 31, 2013 at 5:49 PM, Patrick Gansterer par...@paroga.comwrote: Hi, Am 31.01.2013 um 09:25 schrieb Mark Rowe: CMake was originally considered a non-starter for chromium since the XCode and VS projects it produced didn't look or feel like native projects. However, we've since switched to ninja for most things so I'm less sure how important this is now. I've never been a huge fan of the CMake syntax but I haven't used it enough to really have an informed opinion. Generating well-structured Xcode projects is still something that is important for any meta build system that we would consider adopting for the Mac. It would be substantially more challenging for us to support an alternative system for building our production builds. If there are substantial advantages to using ninja for engineering builds then we may wish to support both to get the best behavior in each environment. The non-natively structured Xcode projects is one thing that turned me off CMake when I looked at it in the past. I’m not sure if this has improved since then. If not, then I don’t think CMake is worth spending much time on. If you want to commit the generated projects CMake isn't an option anyway, since it uses absolute paths and also requires a cmake binary on the machine used for compiling (to support all CMake features across all generators). This tight integration is usually a big win, since the CMakeLists.txt files are part of the build system they can be changed directly in the IDE and update the project without invoking external tools. Only for my personal interest: What do you mean exactly with non-natively structured Xcode projects? CMake aims to support a native IDE feel as far as possible. Would be great if you explain that to me (off-list) to get CMake improved for XCode? Regarding (b) The generated project invokes only tools that are part of the default Mac OS X install: invoking tools that are checked into the WK repo is also possible, right, since we invoke scripts now? So, hypothetically, could we check in a copy of the ninja binary and build with that? Checking in binaries isn’t an option for us, and isn’t a particularly scalable approach anyway given the number of platforms that WebKit can build on. If we require an external tool as a dependency to build WebKit from source then we’d need to check in the source for the tool and build it prior to building WebKit proper. This obviously introduces more complexity so it would be preferable to keep the dependencies to a minimum. Maybe you can check in the ninja source and compile it during build. It only requires python (and hopefully installed build tools) to get built. In my understanding, it doesn't matter whether Apple Mac port supports ninja or not. We could use GNU make if some meta-build system is adopted because Mac OS has it installed. The problem here is that the neither CMake and GYP isn't easy to adopt for reasons discussed in this thread. My personal feeling is that we could build a simple meta build system which specifically targetsMac WebKit and XCode. It could be just a little more than a templating system which generates the list of files plus their UUIDs. The tool doesn't need to be so general like CMake/GYP. Many tricky configuration things could be in the hard-coded boilerplate. It could just focus on generating file list and UUIDs. It's something like project.pbxproj.in and its preprocessor. It won't be a direct step toward the unified build system. But we'll figure out the next step once the build is represented by a simple plain text. -- morrita -- Patrick ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
In my understanding, it doesn't matter whether Apple Mac port supports ninja or not. We could use GNU make if some meta-build system is adopted because Mac OS has it installed. The problem here is that the neither CMake and GYP isn't easy to adopt for reasons discussed in this thread. My personal feeling is that we could build a simple meta build system which specifically targetsMac WebKit and XCode. It could be just a little more than a templating system which generates the list of files plus their UUIDs. The tool doesn't need to be so general like CMake/GYP. Many tricky configuration things could be in the hard-coded boilerplate. It could just focus on generating file list and UUIDs. It's something like project.pbxproj.in and its preprocessor. It won't be a direct step toward the unified build system. But we'll figure out the next step once the build is represented by a simple plain text. -- morrita On Thu, Jan 31, 2013 at 5:49 PM, Patrick Gansterer par...@paroga.comwrote: Hi, Am 31.01.2013 um 09:25 schrieb Mark Rowe: CMake was originally considered a non-starter for chromium since the XCode and VS projects it produced didn't look or feel like native projects. However, we've since switched to ninja for most things so I'm less sure how important this is now. I've never been a huge fan of the CMake syntax but I haven't used it enough to really have an informed opinion. Generating well-structured Xcode projects is still something that is important for any meta build system that we would consider adopting for the Mac. It would be substantially more challenging for us to support an alternative system for building our production builds. If there are substantial advantages to using ninja for engineering builds then we may wish to support both to get the best behavior in each environment. The non-natively structured Xcode projects is one thing that turned me off CMake when I looked at it in the past. I’m not sure if this has improved since then. If not, then I don’t think CMake is worth spending much time on. If you want to commit the generated projects CMake isn't an option anyway, since it uses absolute paths and also requires a cmake binary on the machine used for compiling (to support all CMake features across all generators). This tight integration is usually a big win, since the CMakeLists.txt files are part of the build system they can be changed directly in the IDE and update the project without invoking external tools. Only for my personal interest: What do you mean exactly with non-natively structured Xcode projects? CMake aims to support a native IDE feel as far as possible. Would be great if you explain that to me (off-list) to get CMake improved for XCode? Regarding (b) The generated project invokes only tools that are part of the default Mac OS X install: invoking tools that are checked into the WK repo is also possible, right, since we invoke scripts now? So, hypothetically, could we check in a copy of the ninja binary and build with that? Checking in binaries isn’t an option for us, and isn’t a particularly scalable approach anyway given the number of platforms that WebKit can build on. If we require an external tool as a dependency to build WebKit from source then we’d need to check in the source for the tool and build it prior to building WebKit proper. This obviously introduces more complexity so it would be preferable to keep the dependencies to a minimum. Maybe you can check in the ninja source and compile it during build. It only requires python (and hopefully installed build tools) to get built. -- Patrick ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On 2013-01-31, at 00:59, Jochen Eisinger joc...@chromium.org wrote: On Thu, Jan 31, 2013 at 9:53 AM, Mark Rowe mr...@apple.com wrote: On 2013-01-31, at 00:48, Adam Barth aba...@webkit.org wrote: I would consider changing or improving gyp's syntax to be on the table if it was needed to reach the goal. For what it’s worth, I also find the gyp syntax to be unpleasant. It feels as though it was optimized for being processed by a machine rather than for being written and maintained by humans. Unlike xcodeproj files. :) Don’t get me wrong, Xcode projects suck for hand-editing too. However, they’re not intended to be edited by hand. Gyp files are, and so the expected level of human readability is much higher. Many of us are actually editing the Xcode projects by hand, either because they don't have Xcode or don't know how to use it. (Yes, that includes coming up with a bunch of new UUIDs by hand) I wasn’t trying to suggest that current situation is a good one, only that if it would be easier to get momentum on switching to something like gyp if the replacement’s syntax was friendlier. Particularly when the people that need to be convinced to switch, and who’ll have to adapt their workflow, are those that are editing the project files in a nice GUI. - Mark ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On 2013-01-31, at 01:07, Hajime Morrita morr...@chromium.org wrote: In my understanding, it doesn't matter whether Apple Mac port supports ninja or not. We could use GNU make if some meta-build system is adopted because Mac OS has it installed. The problem here is that the neither CMake and GYP isn't easy to adopt for reasons discussed in this thread. As I mentioned in an earlier email, we need to keep the Mac port building via Xcode for our production builds. My personal feeling is that we could build a simple meta build system which specifically targetsMac WebKit and XCode. It could be just a little more than a templating system which generates the list of files plus their UUIDs. The tool doesn't need to be so general like CMake/GYP. Many tricky configuration things could be in the hard-coded boilerplate. It could just focus on generating file list and UUIDs. It's something like project.pbxproj.in and its preprocessor. It won't be a direct step toward the unified build system. But we'll figure out the next step once the build is represented by a simple plain text. I’ve experimented with this in the past and you’re right that it shouldn’t be particularly difficult to do. However, I suspect that the task would be similar in scope to defining an improved syntax for gyp. And if the syntax is the primary sticking point with gyp then it’d seem preferable to tackle initially. I should also clarify that I don’t think gyp’s current syntax is a showstoppper for adoption. I’d just like to see it improved. - Mark ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On Thu, Jan 31, 2013 at 10:12 AM, Mark Rowe mr...@apple.com wrote: On 2013-01-31, at 00:59, Jochen Eisinger joc...@chromium.org wrote: On Thu, Jan 31, 2013 at 9:53 AM, Mark Rowe mr...@apple.com wrote: On 2013-01-31, at 00:48, Adam Barth aba...@webkit.org wrote: I would consider changing or improving gyp's syntax to be on the table if it was needed to reach the goal. For what it’s worth, I also find the gyp syntax to be unpleasant. It feels as though it was optimized for being processed by a machine rather than for being written and maintained by humans. Unlike xcodeproj files. :) Don’t get me wrong, Xcode projects suck for hand-editing too. However, they’re not intended to be edited by hand. Gyp files are, and so the expected level of human readability is much higher. Many of us are actually editing the Xcode projects by hand, either because they don't have Xcode or don't know how to use it. (Yes, that includes coming up with a bunch of new UUIDs by hand) I wasn’t trying to suggest that current situation is a good one, only that if it would be easier to get momentum on switching to something like gyp if the replacement’s syntax was friendlier. Particularly when the people that need to be convinced to switch, and who’ll have to adapt their workflow, are those that are editing the project files in a nice GUI. Agreed. Another option is to add a webkit-patch command for modifying the build files. That way, the syntax doesn't need to be overly human friendly. There was also some attempt to write a tool to add files automatically: https://bugs.webkit.org/show_bug.cgi?id=61772 I would expect that such a tool becomes easier if it would only modify one source of truth and generates all other artifacts such as Xcode projects from it. best -jochen - Mark ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On Thu, Jan 31, 2013 at 1:17 AM, Jochen Eisinger joc...@chromium.orgwrote: Another option is to add a webkit-patch command for modifying the build files. That way, the syntax doesn't need to be overly human friendly. There was also some attempt to write a tool to add files automatically: https://bugs.webkit.org/show_bug.cgi?id=61772 I would expect that such a tool becomes easier if it would only modify one source of truth and generates all other artifacts such as Xcode projects from it. I don't want build file's syntax to be so human unfriendly that I need a tool for it. Often times, these syntax problems can be improved dramatically by simple changes. e.g. we had a similar discussion about TestExpectation syntax, and I'm much happier with the new syntax even though the new syntax is functionally equivalent to the old one, and two syntaxes are very similar. On Thu, Jan 31, 2013 at 1:17 AM, Mark Rowe mr...@apple.com wrote: I’ve experimented with this in the past and you’re right that it shouldn’t be particularly difficult to do. However, I suspect that the task would be similar in scope to defining an improved syntax for gyp. And if the syntax is the primary sticking point with gyp then it’d seem preferable to tackle initially. Yeah. In fact, we can just come up with whatever syntax we like and convert it to the existing gyp format if the syntax was the biggest issue. - R. Niwa ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
Am 31.01.2013 um 10:37 schrieb Ryosuke Niwa: On Thu, Jan 31, 2013 at 1:17 AM, Jochen Eisinger joc...@chromium.org wrote: Another option is to add a webkit-patch command for modifying the build files. That way, the syntax doesn't need to be overly human friendly. There was also some attempt to write a tool to add files automatically: https://bugs.webkit.org/show_bug.cgi?id=61772 I would expect that such a tool becomes easier if it would only modify one source of truth and generates all other artifacts such as Xcode projects from it. I don't want build file's syntax to be so human unfriendly that I need a tool for it. Often times, these syntax problems can be improved dramatically by simple changes. e.g. we had a similar discussion about TestExpectation syntax, and I'm much happier with the new syntax even though the new syntax is functionally equivalent to the old one, and two syntaxes are very similar. On Thu, Jan 31, 2013 at 1:17 AM, Mark Rowe mr...@apple.com wrote: I’ve experimented with this in the past and you’re right that it shouldn’t be particularly difficult to do. However, I suspect that the task would be similar in scope to defining an improved syntax for gyp. And if the syntax is the primary sticking point with gyp then it’d seem preferable to tackle initially. Yeah. In fact, we can just come up with whatever syntax we like and convert it to the existing gyp format if the syntax was the biggest issue. Do we want to define the whole build system (including information how to invoke the generators) with the new system, or is a simple list for the input files sufficient? IMHO adding a new generator build step happens very rarely. So maybe we can spit the input file list (mainly *.cpp and *.idl) into new files. Then GYP and CMake can read them and generate the build system out of them directly (like they to already today) instead of listing the files in the *.gpyi and *.cmake. This might work for other systems like qmake too. For XCode we can maybe have a template XCode project and generate the work XCode project with a script. This script then only need to fill in the files from the input file list into the template XCode project. Defining the feature flags can be done like Maciej suggested with Port.h files. -- Patrick___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On Thu, Jan 31, 2013 at 6:56 AM, Patrick Gansterer par...@paroga.com wrote: Am 31.01.2013 um 10:37 schrieb Ryosuke Niwa: On Thu, Jan 31, 2013 at 1:17 AM, Jochen Eisinger joc...@chromium.org wrote: Another option is to add a webkit-patch command for modifying the build files. That way, the syntax doesn't need to be overly human friendly. There was also some attempt to write a tool to add files automatically: https://bugs.webkit.org/show_bug.cgi?id=61772 I would expect that such a tool becomes easier if it would only modify one source of truth and generates all other artifacts such as Xcode projects from it. I don't want build file's syntax to be so human unfriendly that I need a tool for it. Often times, these syntax problems can be improved dramatically by simple changes. e.g. we had a similar discussion about TestExpectation syntax, and I'm much happier with the new syntax even though the new syntax is functionally equivalent to the old one, and two syntaxes are very similar. On Thu, Jan 31, 2013 at 1:17 AM, Mark Rowe mr...@apple.com wrote: I’ve experimented with this in the past and you’re right that it shouldn’t be particularly difficult to do. However, I suspect that the task would be similar in scope to defining an improved syntax for gyp. And if the syntax is the primary sticking point with gyp then it’d seem preferable to tackle initially. Yeah. In fact, we can just come up with whatever syntax we like and convert it to the existing gyp format if the syntax was the biggest issue. Do we want to define the whole build system (including information how to invoke the generators) with the new system, or is a simple list for the input files sufficient? IMHO adding a new generator build step happens very rarely. So maybe we can spit the input file list (mainly *.cpp and *.idl) into new files. Then GYP and CMake can read them and generate the build system out of them directly (like they to already today) instead of listing the files in the *.gpyi and *.cmake. This might work for other systems like qmake too. For XCode we can maybe have a template XCode project and generate the work XCode project with a script. This script then only need to fill in the files from the input file list into the template XCode project. Defining the feature flags can be done like Maciej suggested with Port.h files. My 2 cents. One advantage CMake has over other proposals is that it's already working for 2 ports (and potentially 4). It is an open source project so we could potentially contribute to it to add or fix what is needed. One other good point for CMake is that it's widely used in the industry and it is backed by a company. When KDE switched over CMake the guys behind CMake were very very responsive, I believe they will be too if we plan to switch to CMake. The more famous projects they have running CMake, the better it is for them. So if we need to improve the Xcode support then I bet we can count on them. CMake has also some support in various IDE, and if not then the native solution is a fallback. Sure the syntax is maybe not the best but it no worst than Gyp, Xcode, Makefiles, qmake or some perl script. We already live with all these syntax and people are also used to edit the CMake related project. Perfect build system do not exist it's a fact. On the other hand I don't want to loose the native support like Xcode. I don't know if many are using it but I find incredibly convenient to open the Xcode workspace of WebKit, setup the two little things instructed in the wiki and press cmd+b and it just works, it builds, it integrate with Xcode (so you get the neat features of pretty output compiles errors with jumping, ...) and I press cmd+r and it launch MiniBrowser or something else to debug from within the IDE. This is what makes the Mac port a very great port to work on. -- Patrick ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev -- Software Engineer @ Intel Open Source Technology Center ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On Thu, Jan 31, 2013 at 10:37 AM, Ryosuke Niwa rn...@webkit.org wrote: On Thu, Jan 31, 2013 at 1:17 AM, Jochen Eisinger joc...@chromium.orgwrote: Another option is to add a webkit-patch command for modifying the build files. That way, the syntax doesn't need to be overly human friendly. There was also some attempt to write a tool to add files automatically: https://bugs.webkit.org/show_bug.cgi?id=61772 I would expect that such a tool becomes easier if it would only modify one source of truth and generates all other artifacts such as Xcode projects from it. I don't want build file's syntax to be so human unfriendly that I need a tool for it. Often times, these syntax problems can be improved dramatically by simple changes. e.g. we had a similar discussion about TestExpectation syntax, and I'm much happier with the new syntax even though the new syntax is functionally equivalent to the old one, and two syntaxes are very similar. I totally agree. I guess I just failed at finding the right words. -jochen On Thu, Jan 31, 2013 at 1:17 AM, Mark Rowe mr...@apple.com wrote: I’ve experimented with this in the past and you’re right that it shouldn’t be particularly difficult to do. However, I suspect that the task would be similar in scope to defining an improved syntax for gyp. And if the syntax is the primary sticking point with gyp then it’d seem preferable to tackle initially. Yeah. In fact, we can just come up with whatever syntax we like and convert it to the existing gyp format if the syntax was the biggest issue. - R. Niwa ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On Jan 31, 2013, at 12:49 AM, Patrick Gansterer par...@paroga.com wrote: Hi, Am 31.01.2013 um 09:25 schrieb Mark Rowe: Regarding (b) The generated project invokes only tools that are part of the default Mac OS X install: invoking tools that are checked into the WK repo is also possible, right, since we invoke scripts now? So, hypothetically, could we check in a copy of the ninja binary and build with that? Checking in binaries isn’t an option for us, and isn’t a particularly scalable approach anyway given the number of platforms that WebKit can build on. If we require an external tool as a dependency to build WebKit from source then we’d need to check in the source for the tool and build it prior to building WebKit proper. This obviously introduces more complexity so it would be preferable to keep the dependencies to a minimum. Maybe you can check in the ninja source and compile it during build. It only requires python (and hopefully installed build tools) to get built. Depending on how this is done it might be possible, but as I understand it gyp can build ok without ninja, so it doesn't seem strictly necessary. Regards, Maciej ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On Jan 31, 2013, at 1:56 AM, Patrick Gansterer par...@paroga.com wrote: Am 31.01.2013 um 10:37 schrieb Ryosuke Niwa: On Thu, Jan 31, 2013 at 1:17 AM, Jochen Eisinger joc...@chromium.org wrote: Another option is to add a webkit-patch command for modifying the build files. That way, the syntax doesn't need to be overly human friendly. There was also some attempt to write a tool to add files automatically: https://bugs.webkit.org/show_bug.cgi?id=61772 I would expect that such a tool becomes easier if it would only modify one source of truth and generates all other artifacts such as Xcode projects from it. I don't want build file's syntax to be so human unfriendly that I need a tool for it. Often times, these syntax problems can be improved dramatically by simple changes. e.g. we had a similar discussion about TestExpectation syntax, and I'm much happier with the new syntax even though the new syntax is functionally equivalent to the old one, and two syntaxes are very similar. On Thu, Jan 31, 2013 at 1:17 AM, Mark Rowe mr...@apple.com wrote: I’ve experimented with this in the past and you’re right that it shouldn’t be particularly difficult to do. However, I suspect that the task would be similar in scope to defining an improved syntax for gyp. And if the syntax is the primary sticking point with gyp then it’d seem preferable to tackle initially. Yeah. In fact, we can just come up with whatever syntax we like and convert it to the existing gyp format if the syntax was the biggest issue. Do we want to define the whole build system (including information how to invoke the generators) with the new system, or is a simple list for the input files sufficient? IMHO adding a new generator build step happens very rarely. So maybe we can spit the input file list (mainly *.cpp and *.idl) into new files. Then GYP and CMake can read them and generate the build system out of them directly (like they to already today) instead of listing the files in the *.gpyi and *.cmake. This might work for other systems like qmake too. For XCode we can maybe have a template XCode project and generate the work XCode project with a script. This script then only need to fill in the files from the input file list into the template XCode project. Defining the feature flags can be done like Maciej suggested with Port.h files. I think it would be better to adapt an existing meta build system to our needs than to make one from scratch, unless we find that completely impractical. In particular, gyp and cmake both know how to handle generated sources, and while it may not be super common to make a new type of generated source, it's bad for hackability of the project of doing so is super hard. We get a lot of hackability benefits from using various kinds of generated sources, many first introduced in the days when we had a lot fewer build systems. So in my mind, they are already ahead of a hypothetical simple system. Cheers, Maciej ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
Am 31.01.2013 um 17:17 schrieb Maciej Stachowiak: On Jan 31, 2013, at 1:56 AM, Patrick Gansterer par...@paroga.com wrote: Am 31.01.2013 um 10:37 schrieb Ryosuke Niwa: On Thu, Jan 31, 2013 at 1:17 AM, Jochen Eisinger joc...@chromium.org wrote: Another option is to add a webkit-patch command for modifying the build files. That way, the syntax doesn't need to be overly human friendly. There was also some attempt to write a tool to add files automatically: https://bugs.webkit.org/show_bug.cgi?id=61772 I would expect that such a tool becomes easier if it would only modify one source of truth and generates all other artifacts such as Xcode projects from it. I don't want build file's syntax to be so human unfriendly that I need a tool for it. Often times, these syntax problems can be improved dramatically by simple changes. e.g. we had a similar discussion about TestExpectation syntax, and I'm much happier with the new syntax even though the new syntax is functionally equivalent to the old one, and two syntaxes are very similar. On Thu, Jan 31, 2013 at 1:17 AM, Mark Rowe mr...@apple.com wrote: I’ve experimented with this in the past and you’re right that it shouldn’t be particularly difficult to do. However, I suspect that the task would be similar in scope to defining an improved syntax for gyp. And if the syntax is the primary sticking point with gyp then it’d seem preferable to tackle initially. Yeah. In fact, we can just come up with whatever syntax we like and convert it to the existing gyp format if the syntax was the biggest issue. Do we want to define the whole build system (including information how to invoke the generators) with the new system, or is a simple list for the input files sufficient? IMHO adding a new generator build step happens very rarely. So maybe we can spit the input file list (mainly *.cpp and *.idl) into new files. Then GYP and CMake can read them and generate the build system out of them directly (like they to already today) instead of listing the files in the *.gpyi and *.cmake. This might work for other systems like qmake too. For XCode we can maybe have a template XCode project and generate the work XCode project with a script. This script then only need to fill in the files from the input file list into the template XCode project. Defining the feature flags can be done like Maciej suggested with Port.h files. I think it would be better to adapt an existing meta build system to our needs than to make one from scratch, unless we find that completely impractical. In particular, gyp and cmake both know how to handle generated sources, and while it may not be super common to make a new type of generated source, it's bad for hackability of the project of doing so is super hard. We get a lot of hackability benefits from using various kinds of generated sources, many first introduced in the days when we had a lot fewer build systems. So in my mind, they are already ahead of a hypothetical simple system. Do you want to kick the requirements of the smaller ports from trunk or do you think that e.g. a qmake generate for GYP makes sense? AFAIK e.g. QtWebKit is shipped with Qt, which uses qmake as build system, where CMake/GYP is not an option. I completely agree that creating a new meta meta build system isn't a good idea, but sharing the common parts (which reduce the daily productivity) might be a step in the right direction. Using simple text files which contain the list of files (like the gpyi files already do today) isn't a new build system. It only offers the existing meta build systems (CMake, GYP, autotools, qmake) to use a common base. The remaining build systems can be ported to one of these systems or be adopted to use this file lists too. -- Patrick___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
30.01.2013, в 17:14, Dirk Pranke dpra...@chromium.org написал(а): CMake was originally considered a non-starter for chromium since the XCode and VS projects it produced didn't look or feel like native projects. I think that this is the key consideration here. Whatever the pains of modifying multiple build systems files are, reading and - to a lesser extent - modifying actual code is the most important activity to optimize for. It's not so helpful to have an easy way to add code if the only reason you are adding it is that you could not easily navigate the code base to find an already existing solution. To me personally, this means that Xcode project files need to be fully usable. - WBR, Alexey Proskuryakov ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On Wednesday, January 30, 2013 04:15:48 PM Maciej Stachowiak wrote: On Jan 30, 2013, at 3:24 PM, Dirk Pranke dpra...@chromium.org wrote: On Wed, Jan 30, 2013 at 1:50 PM, Filip Pizlo fpi...@apple.com wrote: Thanks for sharing this. On Jan 30, 2013, at 1:28 PM, Eric Seidel e...@webkit.org wrote: I wish we only had one build system (it were easy to add/remove/move files). I believe changes like http://trac.webkit.org/changeset/74849 are an unhealthy sign for the project. Adam is not the only person who has chosen to empty files instead of removing them. The pain of updating 8 build system is so great, we jump through hoops to avoid it. Which means it took us months to move JavaScriptCore/wtf to WTF, and will take us years to kill WebCore/platform. +1 This is a hard problem. It is a problem worth solving. Do you have more thoughts on this, particularly since you know quite well how both Xcode and gyp work? I suspect this is one of those things that it would be hard to achieve consensus on since there are so many stakeholders. But it may be fruitful to have a what if discussion about what this might look like. I think we can solve this problem if we agree that we want to. I think we haven't in the past mostly because we couldn't reach a consensus that it was worth solving enough to really try. I would love to see this fixed and would be glad to work on it. I think we should at least pursue this far enough to fully understand what our options are and what the costs and tradeoffs might be; does anyone disagree, and is anyone else willing to help pitch in? I think there are several possible ways we could solve this. One would be to switch to a common meta-build system. My understanding is that Apple's internal production build processes impose certain constraints here that I don't fully understand, but I know we've discussed the possibility of checking in generated project files as a workaround. Maybe there are other options as well to those constraints? I would love to discuss this further w/ someone from Apple ... It's far simplest for us if: (a) There is an Xcode project (or a Makefile) that builds the Mac port checked in to source control. (b) The generated project invokes only tools that are part of the default Mac OS X install. It may not be completely impossible to violate these requirements but it will require a lot of bureaucracy. (Also, just to get this out of the way, I don't think gyp needs to be the solution). Another alternative would be to write a script that did support at least the common use cases (add/move/delete files). There have been attempts in the past, but they have foundered on at least some perceived skepticism over how well this would work w/ XCode. That said, I don't think we've really pushed this to see. At some point this script might turn into a meta-meta-build system, which might be silly but also be the shortest path to the finish line. I suggest if there is interest in this we can start a new thread to discuss further ... My preference would be to use a common meta-build system with a comfortably human-readable and human-editable syntax, and checked in generated project files for those ports that need them. I think a key to making this work is to get Chromium and the Apple Mac port onto a common build system, which will probably require both Google and Apple ponying up at least one person to work on this project for a reasonable period of time. I think the plausible meta-build-systems to use would be CMake and Gyp, but in both cases it may be necessary to modify them to work well for everyone. I'd also like to add that I think a key related issue to common build system is common feature configuration. The many different ways ports control their feature flags is super confusing. I've long wanted to implement common configuration management, but have not had time. I have a patch trying to solve this issue for CMake based ports[1], the patch still on going, but even a change affecting just 2-3 ports using the same build system is a bit hard to get a consensus, so you can imagine how hard will be to get a consensus over all WebKit ports. [1] https://bugs.webkit.org/show_bug.cgi?id=103162 Cheers, Maciej ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev signature.asc Description: This is a digitally signed message part. ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On Thu, Jan 31, 2013 at 11:48 AM, Hugo Parente Lima hugo.l...@openbossa.org wrote: On Wednesday, January 30, 2013 04:15:48 PM Maciej Stachowiak wrote: On Jan 30, 2013, at 3:24 PM, Dirk Pranke dpra...@chromium.org wrote: On Wed, Jan 30, 2013 at 1:50 PM, Filip Pizlo fpi...@apple.com wrote: Thanks for sharing this. On Jan 30, 2013, at 1:28 PM, Eric Seidel e...@webkit.org wrote: I wish we only had one build system (it were easy to add/remove/move files). I believe changes like http://trac.webkit.org/changeset/74849 are an unhealthy sign for the project. Adam is not the only person who has chosen to empty files instead of removing them. The pain of updating 8 build system is so great, we jump through hoops to avoid it. Which means it took us months to move JavaScriptCore/wtf to WTF, and will take us years to kill WebCore/platform. +1 This is a hard problem. It is a problem worth solving. Do you have more thoughts on this, particularly since you know quite well how both Xcode and gyp work? I suspect this is one of those things that it would be hard to achieve consensus on since there are so many stakeholders. But it may be fruitful to have a what if discussion about what this might look like. I think we can solve this problem if we agree that we want to. I think we haven't in the past mostly because we couldn't reach a consensus that it was worth solving enough to really try. I would love to see this fixed and would be glad to work on it. I think we should at least pursue this far enough to fully understand what our options are and what the costs and tradeoffs might be; does anyone disagree, and is anyone else willing to help pitch in? I think there are several possible ways we could solve this. One would be to switch to a common meta-build system. My understanding is that Apple's internal production build processes impose certain constraints here that I don't fully understand, but I know we've discussed the possibility of checking in generated project files as a workaround. Maybe there are other options as well to those constraints? I would love to discuss this further w/ someone from Apple ... It's far simplest for us if: (a) There is an Xcode project (or a Makefile) that builds the Mac port checked in to source control. (b) The generated project invokes only tools that are part of the default Mac OS X install. It may not be completely impossible to violate these requirements but it will require a lot of bureaucracy. (Also, just to get this out of the way, I don't think gyp needs to be the solution). Another alternative would be to write a script that did support at least the common use cases (add/move/delete files). There have been attempts in the past, but they have foundered on at least some perceived skepticism over how well this would work w/ XCode. That said, I don't think we've really pushed this to see. At some point this script might turn into a meta-meta-build system, which might be silly but also be the shortest path to the finish line. I suggest if there is interest in this we can start a new thread to discuss further ... My preference would be to use a common meta-build system with a comfortably human-readable and human-editable syntax, and checked in generated project files for those ports that need them. I think a key to making this work is to get Chromium and the Apple Mac port onto a common build system, which will probably require both Google and Apple ponying up at least one person to work on this project for a reasonable period of time. I think the plausible meta-build-systems to use would be CMake and Gyp, but in both cases it may be necessary to modify them to work well for everyone. I'd also like to add that I think a key related issue to common build system is common feature configuration. The many different ways ports control their feature flags is super confusing. I've long wanted to implement common configuration management, but have not had time. I have a patch trying to solve this issue for CMake based ports[1], the patch still on going, but even a change affecting just 2-3 ports using the same build system is a bit hard to get a consensus, so you can imagine how hard will be to get a consensus over all WebKit ports. [1] https://bugs.webkit.org/show_bug.cgi?id=103162 This is slightly off-topic, but I had thought that no one was actually using CMake; maybe I was mistaken and just none of the ports that build on webkit.org are? It looks like Blackberry and maybe a WinCE port uses CMake? Anyone else? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
Am 31.01.2013 um 21:07 schrieb Dirk Pranke: On Thu, Jan 31, 2013 at 11:48 AM, Hugo Parente Lima hugo.l...@openbossa.org wrote: On Wednesday, January 30, 2013 04:15:48 PM Maciej Stachowiak wrote: On Jan 30, 2013, at 3:24 PM, Dirk Pranke dpra...@chromium.org wrote: On Wed, Jan 30, 2013 at 1:50 PM, Filip Pizlo fpi...@apple.com wrote: Thanks for sharing this. On Jan 30, 2013, at 1:28 PM, Eric Seidel e...@webkit.org wrote: I wish we only had one build system (it were easy to add/remove/move files). I believe changes like http://trac.webkit.org/changeset/74849 are an unhealthy sign for the project. Adam is not the only person who has chosen to empty files instead of removing them. The pain of updating 8 build system is so great, we jump through hoops to avoid it. Which means it took us months to move JavaScriptCore/wtf to WTF, and will take us years to kill WebCore/platform. +1 This is a hard problem. It is a problem worth solving. Do you have more thoughts on this, particularly since you know quite well how both Xcode and gyp work? I suspect this is one of those things that it would be hard to achieve consensus on since there are so many stakeholders. But it may be fruitful to have a what if discussion about what this might look like. I think we can solve this problem if we agree that we want to. I think we haven't in the past mostly because we couldn't reach a consensus that it was worth solving enough to really try. I would love to see this fixed and would be glad to work on it. I think we should at least pursue this far enough to fully understand what our options are and what the costs and tradeoffs might be; does anyone disagree, and is anyone else willing to help pitch in? I think there are several possible ways we could solve this. One would be to switch to a common meta-build system. My understanding is that Apple's internal production build processes impose certain constraints here that I don't fully understand, but I know we've discussed the possibility of checking in generated project files as a workaround. Maybe there are other options as well to those constraints? I would love to discuss this further w/ someone from Apple ... It's far simplest for us if: (a) There is an Xcode project (or a Makefile) that builds the Mac port checked in to source control. (b) The generated project invokes only tools that are part of the default Mac OS X install. It may not be completely impossible to violate these requirements but it will require a lot of bureaucracy. (Also, just to get this out of the way, I don't think gyp needs to be the solution). Another alternative would be to write a script that did support at least the common use cases (add/move/delete files). There have been attempts in the past, but they have foundered on at least some perceived skepticism over how well this would work w/ XCode. That said, I don't think we've really pushed this to see. At some point this script might turn into a meta-meta-build system, which might be silly but also be the shortest path to the finish line. I suggest if there is interest in this we can start a new thread to discuss further ... My preference would be to use a common meta-build system with a comfortably human-readable and human-editable syntax, and checked in generated project files for those ports that need them. I think a key to making this work is to get Chromium and the Apple Mac port onto a common build system, which will probably require both Google and Apple ponying up at least one person to work on this project for a reasonable period of time. I think the plausible meta-build-systems to use would be CMake and Gyp, but in both cases it may be necessary to modify them to work well for everyone. I'd also like to add that I think a key related issue to common build system is common feature configuration. The many different ways ports control their feature flags is super confusing. I've long wanted to implement common configuration management, but have not had time. I have a patch trying to solve this issue for CMake based ports[1], the patch still on going, but even a change affecting just 2-3 ports using the same build system is a bit hard to get a consensus, so you can imagine how hard will be to get a consensus over all WebKit ports. [1] https://bugs.webkit.org/show_bug.cgi?id=103162 This is slightly off-topic, but I had thought that no one was actually using CMake; maybe I was mistaken and just none of the ports that build on webkit.org are? It looks like Blackberry and maybe a WinCE port uses CMake? Anyone else? EFL uses CMake too. 4 EFL bots @ http://build.webkit.org 1 WinCE bot @ http://build.webkit.org 1 EFL bot as EWS -- Patrick ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On Thu, Jan 31, 2013 at 12:10 PM, Patrick Gansterer par...@paroga.com wrote: Am 31.01.2013 um 21:07 schrieb Dirk Pranke: On Thu, Jan 31, 2013 at 11:48 AM, Hugo Parente Lima hugo.l...@openbossa.org wrote: On Wednesday, January 30, 2013 04:15:48 PM Maciej Stachowiak wrote: On Jan 30, 2013, at 3:24 PM, Dirk Pranke dpra...@chromium.org wrote: On Wed, Jan 30, 2013 at 1:50 PM, Filip Pizlo fpi...@apple.com wrote: Thanks for sharing this. On Jan 30, 2013, at 1:28 PM, Eric Seidel e...@webkit.org wrote: I wish we only had one build system (it were easy to add/remove/move files). I believe changes like http://trac.webkit.org/changeset/74849 are an unhealthy sign for the project. Adam is not the only person who has chosen to empty files instead of removing them. The pain of updating 8 build system is so great, we jump through hoops to avoid it. Which means it took us months to move JavaScriptCore/wtf to WTF, and will take us years to kill WebCore/platform. +1 This is a hard problem. It is a problem worth solving. Do you have more thoughts on this, particularly since you know quite well how both Xcode and gyp work? I suspect this is one of those things that it would be hard to achieve consensus on since there are so many stakeholders. But it may be fruitful to have a what if discussion about what this might look like. I think we can solve this problem if we agree that we want to. I think we haven't in the past mostly because we couldn't reach a consensus that it was worth solving enough to really try. I would love to see this fixed and would be glad to work on it. I think we should at least pursue this far enough to fully understand what our options are and what the costs and tradeoffs might be; does anyone disagree, and is anyone else willing to help pitch in? I think there are several possible ways we could solve this. One would be to switch to a common meta-build system. My understanding is that Apple's internal production build processes impose certain constraints here that I don't fully understand, but I know we've discussed the possibility of checking in generated project files as a workaround. Maybe there are other options as well to those constraints? I would love to discuss this further w/ someone from Apple ... It's far simplest for us if: (a) There is an Xcode project (or a Makefile) that builds the Mac port checked in to source control. (b) The generated project invokes only tools that are part of the default Mac OS X install. It may not be completely impossible to violate these requirements but it will require a lot of bureaucracy. (Also, just to get this out of the way, I don't think gyp needs to be the solution). Another alternative would be to write a script that did support at least the common use cases (add/move/delete files). There have been attempts in the past, but they have foundered on at least some perceived skepticism over how well this would work w/ XCode. That said, I don't think we've really pushed this to see. At some point this script might turn into a meta-meta-build system, which might be silly but also be the shortest path to the finish line. I suggest if there is interest in this we can start a new thread to discuss further ... My preference would be to use a common meta-build system with a comfortably human-readable and human-editable syntax, and checked in generated project files for those ports that need them. I think a key to making this work is to get Chromium and the Apple Mac port onto a common build system, which will probably require both Google and Apple ponying up at least one person to work on this project for a reasonable period of time. I think the plausible meta-build-systems to use would be CMake and Gyp, but in both cases it may be necessary to modify them to work well for everyone. I'd also like to add that I think a key related issue to common build system is common feature configuration. The many different ways ports control their feature flags is super confusing. I've long wanted to implement common configuration management, but have not had time. I have a patch trying to solve this issue for CMake based ports[1], the patch still on going, but even a change affecting just 2-3 ports using the same build system is a bit hard to get a consensus, so you can imagine how hard will be to get a consensus over all WebKit ports. [1] https://bugs.webkit.org/show_bug.cgi?id=103162 This is slightly off-topic, but I had thought that no one was actually using CMake; maybe I was mistaken and just none of the ports that build on webkit.org are? It looks like Blackberry and maybe a WinCE port uses CMake? Anyone else? EFL uses CMake too. 4 EFL bots @ http://build.webkit.org 1 WinCE bot @ http://build.webkit.org 1 EFL bot as EWS Ah, I thought EFL was using Autotools. Thanks for the correction. -- Dirk ___ webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On Thursday, January 31, 2013 06:14:20 PM Patrick Gansterer wrote: Am 31.01.2013 um 17:17 schrieb Maciej Stachowiak: On Jan 31, 2013, at 1:56 AM, Patrick Gansterer par...@paroga.com wrote: Am 31.01.2013 um 10:37 schrieb Ryosuke Niwa: On Thu, Jan 31, 2013 at 1:17 AM, Jochen Eisinger joc...@chromium.org wrote: Another option is to add a webkit-patch command for modifying the build files. That way, the syntax doesn't need to be overly human friendly. There was also some attempt to write a tool to add files automatically: https://bugs.webkit.org/show_bug.cgi?ida772 I would expect that such a tool becomes easier if it would only modify one source of truth and generates all other artifacts such as Xcode projects from it. I don't want build file's syntax to be so human unfriendly that I need a tool for it. Often times, these syntax problems can be improved dramatically by simple changes. e.g. we had a similar discussion about TestExpectation syntax, and I'm much happier with the new syntax even though the new syntax is functionally equivalent to the old one, and two syntaxes are very similar. On Thu, Jan 31, 2013 at 1:17 AM, Mark Rowe mr...@apple.com wrote: Iâve experimented with this in the past and youâre right that it shouldnât be particularly difficult to do. However, I suspect that the task would be similar in scope to defining an improved syntax for gyp. And if the syntax is the primary sticking point with gyp then itâd seem preferable to tackle initially. Yeah. In fact, we can just come up with whatever syntax we like and convert it to the existing gyp format if the syntax was the biggest issue. Do we want to define the whole build system (including inform tion how to invoke the generators) with the new system, or is a simple list for the input files sufficient? IMHO adding a new generator build step happens very rarely. So maybe we can spit the input file list (mainly *.cpp and *.idl) into new files. Then GYP and CMake can read them and generate the build system out of them directly (like they to already today) instead of listing the files in the *.gpyi and *.cmake. This might work for other systems like qmake too. For XCode we can maybe have a template XCode project and generate the work XCode project with a script. This script then only need to fill in the files from the input file list into the template XCode project. Defining the feature flags can be done like Maciej suggested with Port.h files. I think it would be better to adapt an existing meta build system to our needs than to make one from scratch, unless we find that completely impractical. I particular, gyp and cmake both know how to handle generated sources, and while it may not be super common to make a new type of generated source, it's bad for hackability of the project of doing so is super hard. We get a lot of hackability benefits from using various kinds of generated sources, many first introduced in the days when we had a lot fewer build systems. So in my mind, they are already ahead of a hypothetical simple system. Do you want to kick the requirements of the smaller ports from trunk or do you think that e.g. a qmake generate for GYP makes sense? AFAIK e.g. QtWebKit is shipped with Qt, which uses qmake as build system, where CMake/GYP is not an option. I completely agree that creating a new meta meta build system isn't a good idea, but sharing the common parts (which reduce the daily productivity) might be a step in the right direction. Using simple text files which contain the list of files (like the gpyi files already do tod y) isn't a new build system. It only offers the existing meta build systems (CMake, GYP, autotools, qmake) to use a common base. I agree, common build system on WebKit is an utopia, create a new meta build system isn't a good idea, so the only plausible option is to unify way we add/remove/modify files from the build. Create a script to do this on all build system is yet another utopia, so again, the only option is to have plain text files. Going a bit further on this idea, would be nice to also have conditionals on these plain text files to cover the use case of e.g. add foo.cpp to the build when ENABLE_FOO or WTF_USE_FOO is on would be even better, a script would be called passing all ENABLE, USE, HAVE, etc.. flags and returning a list of files, the only question is if all current build system would support such dynamic source file list, CMake does :-) The remaining build systems can be ported to one of these systems or be adopted to use this file lists too. -- Patrick pgpVpcWAo2RvL.pgp Description: This is a digitally signed message part. ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On Thu, Jan 31, 2013 at 9:14 AM, Patrick Gansterer par...@paroga.com wrote: Am 31.01.2013 um 17:17 schrieb Maciej Stachowiak: On Jan 31, 2013, at 1:56 AM, Patrick Gansterer par...@paroga.com wrote: Am 31.01.2013 um 10:37 schrieb Ryosuke Niwa: On Thu, Jan 31, 2013 at 1:17 AM, Jochen Eisinger joc...@chromium.org wrote: Another option is to add a webkit-patch command for modifying the build files. That way, the syntax doesn't need to be overly human friendly. There was also some attempt to write a tool to add files automatically: https://bugs.webkit.org/show_bug.cgi?id=61772 I would expect that such a tool becomes easier if it would only modify one source of truth and generates all other artifacts such as Xcode projects from it. I don't want build file's syntax to be so human unfriendly that I need a tool for it. Often times, these syntax problems can be improved dramatically by simple changes. e.g. we had a similar discussion about TestExpectation syntax, and I'm much happier with the new syntax even though the new syntax is functionally equivalent to the old one, and two syntaxes are very similar. On Thu, Jan 31, 2013 at 1:17 AM, Mark Rowe mr...@apple.com wrote: I’ve experimented with this in the past and you’re right that it shouldn’t be particularly difficult to do. However, I suspect that the task would be similar in scope to defining an improved syntax for gyp. And if the syntax is the primary sticking point with gyp then it’d seem preferable to tackle initially. Yeah. In fact, we can just come up with whatever syntax we like and convert it to the existing gyp format if the syntax was the biggest issue. Do we want to define the whole build system (including information how to invoke the generators) with the new system, or is a simple list for the input files sufficient? IMHO adding a new generator build step happens very rarely. So maybe we can spit the input file list (mainly *.cpp and *.idl) into new files. Then GYP and CMake can read them and generate the build system out of them directly (like they to already today) instead of listing the files in the *.gpyi and *.cmake. This might work for other systems like qmake too. For XCode we can maybe have a template XCode project and generate the work XCode project with a script. This script then only need to fill in the files from the input file list into the template XCode project. Defining the feature flags can be done like Maciej suggested with Port.h files. I think it would be better to adapt an existing meta build system to our needs than to make one from scratch, unless we find that completely impractical. In particular, gyp and cmake both know how to handle generated sources, and while it may not be super common to make a new type of generated source, it's bad for hackability of the project of doing so is super hard. We get a lot of hackability benefits from using various kinds of generated sources, many first introduced in the days when we had a lot fewer build systems. So in my mind, they are already ahead of a hypothetical simple system. Do you want to kick the requirements of the smaller ports from trunk or do you think that e.g. a qmake generate for GYP makes sense? AFAIK e.g. QtWebKit is shipped with Qt, which uses qmake as build system, where CMake/GYP is not an option. It could certainly make sense for us to add Autotools and Qmake generators to GYP; I'm less certain if a CMake generator makes much sense, but I haven't thought about it as much. I'm not super familiar with any of these three tools, so I could be dead wrong. I completely agree that creating a new meta meta build system isn't a good idea, but sharing the common parts (which reduce the daily productivity) might be a step in the right direction. Using simple text files which contain the list of files (like the gpyi files already do today) isn't a new build system. It only offers the existing meta build systems (CMake, GYP, autotools, qmake) to use a common base. The remaining build systems can be ported to one of these systems or be adopted to use this file lists too. -- Patrick I suspect that we would quickly find that we would want some sort of support for conditionals and/or file inclusion in our simple text files, at which point you basically get a meta-meta-build system :). I don't actually think such a thing is that bad of an idea, but it's all in the details. I would like to find a solution where all of the ports were able to retain integration with their native build environments one way or another. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On Thu, Jan 31, 2013 at 12:31 PM, Dirk Pranke dpra...@chromium.org wrote: On Thu, Jan 31, 2013 at 9:14 AM, Patrick Gansterer par...@paroga.com wrote: Am 31.01.2013 um 17:17 schrieb Maciej Stachowiak: On Jan 31, 2013, at 1:56 AM, Patrick Gansterer par...@paroga.com wrote: Am 31.01.2013 um 10:37 schrieb Ryosuke Niwa: On Thu, Jan 31, 2013 at 1:17 AM, Jochen Eisinger joc...@chromium.org wrote: Another option is to add a webkit-patch command for modifying the build files. That way, the syntax doesn't need to be overly human friendly. There was also some attempt to write a tool to add files automatically: https://bugs.webkit.org/show_bug.cgi?id=61772 I would expect that such a tool becomes easier if it would only modify one source of truth and generates all other artifacts such as Xcode projects from it. I don't want build file's syntax to be so human unfriendly that I need a tool for it. Often times, these syntax problems can be improved dramatically by simple changes. e.g. we had a similar discussion about TestExpectation syntax, and I'm much happier with the new syntax even though the new syntax is functionally equivalent to the old one, and two syntaxes are very similar. On Thu, Jan 31, 2013 at 1:17 AM, Mark Rowe mr...@apple.com wrote: I’ve experimented with this in the past and you’re right that it shouldn’t be particularly difficult to do. However, I suspect that the task would be similar in scope to defining an improved syntax for gyp. And if the syntax is the primary sticking point with gyp then it’d seem preferable to tackle initially. Yeah. In fact, we can just come up with whatever syntax we like and convert it to the existing gyp format if the syntax was the biggest issue. Do we want to define the whole build system (including information how to invoke the generators) with the new system, or is a simple list for the input files sufficient? IMHO adding a new generator build step happens very rarely. So maybe we can spit the input file list (mainly *.cpp and *.idl) into new files. Then GYP and CMake can read them and generate the build system out of them directly (like they to already today) instead of listing the files in the *.gpyi and *.cmake. This might work for other systems like qmake too. For XCode we can maybe have a template XCode project and generate the work XCode project with a script. This script then only need to fill in the files from the input file list into the template XCode project. Defining the feature flags can be done like Maciej suggested with Port.h files. I think it would be better to adapt an existing meta build system to our needs than to make one from scratch, unless we find that completely impractical. In particular, gyp and cmake both know how to handle generated sources, and while it may not be super common to make a new type of generated source, it's bad for hackability of the project of doing so is super hard. We get a lot of hackability benefits from using various kinds of generated sources, many first introduced in the days when we had a lot fewer build systems. So in my mind, they are already ahead of a hypothetical simple system. Do you want to kick the requirements of the smaller ports from trunk or do you think that e.g. a qmake generate for GYP makes sense? AFAIK e.g. QtWebKit is shipped with Qt, which uses qmake as build system, where CMake/GYP is not an option. It could certainly make sense for us to add Autotools and Qmake generators to GYP; I'm less certain if a CMake generator makes much sense, but I haven't thought about it as much. I'm not super familiar with any of these three tools, so I could be dead wrong. I think making Autotools and Qmake use GYP will be a huge win even if we couldn't make Xcodeproj and CMake use GYP at the end. Having GYP + CMake + Xcodeproj is much better than having Autotools and Qmake on top of that. I don't think we need to necessarily unify all build systems. Reducing the number of build systems is a worthwhile goal on its own merit. - R. Niwa ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
This is a lovely discussion to have every now and then :-) Maybe the file add/move/delete tool that someone wrote could land without xcode support for now? Then the pain of moving files is reduced to two systems (xcode and everyone else), and that's something that's at least tractable. Nico On Thu, Jan 31, 2013 at 12:31 PM, Dirk Pranke dpra...@chromium.org wrote: On Thu, Jan 31, 2013 at 9:14 AM, Patrick Gansterer par...@paroga.com wrote: Am 31.01.2013 um 17:17 schrieb Maciej Stachowiak: On Jan 31, 2013, at 1:56 AM, Patrick Gansterer par...@paroga.com wrote: Am 31.01.2013 um 10:37 schrieb Ryosuke Niwa: On Thu, Jan 31, 2013 at 1:17 AM, Jochen Eisinger joc...@chromium.org wrote: Another option is to add a webkit-patch command for modifying the build files. That way, the syntax doesn't need to be overly human friendly. There was also some attempt to write a tool to add files automatically: https://bugs.webkit.org/show_bug.cgi?id=61772 I would expect that such a tool becomes easier if it would only modify one source of truth and generates all other artifacts such as Xcode projects from it. I don't want build file's syntax to be so human unfriendly that I need a tool for it. Often times, these syntax problems can be improved dramatically by simple changes. e.g. we had a similar discussion about TestExpectation syntax, and I'm much happier with the new syntax even though the new syntax is functionally equivalent to the old one, and two syntaxes are very similar. On Thu, Jan 31, 2013 at 1:17 AM, Mark Rowe mr...@apple.com wrote: I’ve experimented with this in the past and you’re right that it shouldn’t be particularly difficult to do. However, I suspect that the task would be similar in scope to defining an improved syntax for gyp. And if the syntax is the primary sticking point with gyp then it’d seem preferable to tackle initially. Yeah. In fact, we can just come up with whatever syntax we like and convert it to the existing gyp format if the syntax was the biggest issue. Do we want to define the whole build system (including information how to invoke the generators) with the new system, or is a simple list for the input files sufficient? IMHO adding a new generator build step happens very rarely. So maybe we can spit the input file list (mainly *.cpp and *.idl) into new files. Then GYP and CMake can read them and generate the build system out of them directly (like they to already today) instead of listing the files in the *.gpyi and *.cmake. This might work for other systems like qmake too. For XCode we can maybe have a template XCode project and generate the work XCode project with a script. This script then only need to fill in the files from the input file list into the template XCode project. Defining the feature flags can be done like Maciej suggested with Port.h files. I think it would be better to adapt an existing meta build system to our needs than to make one from scratch, unless we find that completely impractical. In particular, gyp and cmake both know how to handle generated sources, and while it may not be super common to make a new type of generated source, it's bad for hackability of the project of doing so is super hard. We get a lot of hackability benefits from using various kinds of generated sources, many first introduced in the days when we had a lot fewer build systems. So in my mind, they are already ahead of a hypothetical simple system. Do you want to kick the requirements of the smaller ports from trunk or do you think that e.g. a qmake generate for GYP makes sense? AFAIK e.g. QtWebKit is shipped with Qt, which uses qmake as build system, where CMake/GYP is not an option. It could certainly make sense for us to add Autotools and Qmake generators to GYP; I'm less certain if a CMake generator makes much sense, but I haven't thought about it as much. I'm not super familiar with any of these three tools, so I could be dead wrong. I completely agree that creating a new meta meta build system isn't a good idea, but sharing the common parts (which reduce the daily productivity) might be a step in the right direction. Using simple text files which contain the list of files (like the gpyi files already do today) isn't a new build system. It only offers the existing meta build systems (CMake, GYP, autotools, qmake) to use a common base. The remaining build systems can be ported to one of these systems or be adopted to use this file lists too. -- Patrick I suspect that we would quickly find that we would want some sort of support for conditionals and/or file inclusion in our simple text files, at which point you basically get a meta-meta-build system :). I don't actually think such a thing is that bad of an idea, but it's all in the details. I would like to find a solution where all of the ports were able to retain integration with their native build environments one way or another. -- Dirk
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
See https://bugs.webkit.org/show_bug.cgi?id=61773 https://bugs.webkit.org/show_bug.cgi?id=64149 On Thu, Jan 31, 2013 at 12:36 PM, Nico Weber tha...@chromium.org wrote: This is a lovely discussion to have every now and then :-) Maybe the file add/move/delete tool that someone wrote could land without xcode support for now? Then the pain of moving files is reduced to two systems (xcode and everyone else), and that's something that's at least tractable. Nico On Thu, Jan 31, 2013 at 12:31 PM, Dirk Pranke dpra...@chromium.org wrote: On Thu, Jan 31, 2013 at 9:14 AM, Patrick Gansterer par...@paroga.com wrote: Am 31.01.2013 um 17:17 schrieb Maciej Stachowiak: On Jan 31, 2013, at 1:56 AM, Patrick Gansterer par...@paroga.com wrote: Am 31.01.2013 um 10:37 schrieb - R. Niwa : On Thu, Jan 31, 2013 at 1:17 AM, Jochen Eisinger joc...@chromium.org wrote: Another option is to add a webkit-patch command for modifying the build files. That way, the syntax doesn't need to be overly human friendly. There was also some attempt to write a tool to add files automatically: https://bugs.webkit.org/show_bug.cgi?id=61772 I would expect that such a tool becomes easier if it would only modify one source of truth and generates all other artifacts such as Xcode projects from it. I don't want build file's syntax to be so human unfriendly that I need a tool for it. Often times, these syntax problems can be improved dramatically by simple changes. e.g. we had a similar discussion about TestExpectation syntax, and I'm much happier with the new syntax even though the new syntax is functionally equivalent to the old one, and two syntaxes are very similar. On Thu, Jan 31, 2013 at 1:17 AM, Mark Rowe mr...@apple.com wrote: I’ve experimented with this in the past and you’re right that it shouldn’t be particularly difficult to do. However, I suspect that the task would be similar in scope to defining an improved syntax for gyp. And if the syntax is the primary sticking point with gyp then it’d seem preferable to tackle initially. Yeah. In fact, we can just come up with whatever syntax we like and convert it to the existing gyp format if the syntax was the biggest issue. Do we want to define the whole build system (including information how to invoke the generators) with the new system, or is a simple list for the input files sufficient? IMHO adding a new generator build step happens very rarely. So maybe we can spit the input file list (mainly *.cpp and *.idl) into new files. Then GYP and CMake can read them and generate the build system out of them directly (like they to already today) instead of listing the files in the *.gpyi and *.cmake. This might work for other systems like qmake too. For XCode we can maybe have a template XCode project and generate the work XCode project with a script. This script then only need to fill in the files from the input file list into the template XCode project. Defining the feature flags can be done like Maciej suggested with Port.h files. I think it would be better to adapt an existing meta build system to our needs than to make one from scratch, unless we find that completely impractical. In particular, gyp and cmake both know how to handle generated sources, and while it may not be super common to make a new type of generated source, it's bad for hackability of the project of doing so is super hard. We get a lot of hackability benefits from using various kinds of generated sources, many first introduced in the days when we had a lot fewer build systems. So in my mind, they are already ahead of a hypothetical simple system. Do you want to kick the requirements of the smaller ports from trunk or do you think that e.g. a qmake generate for GYP makes sense? AFAIK e.g. QtWebKit is shipped with Qt, which uses qmake as build system, where CMake/GYP is not an option. It could certainly make sense for us to add Autotools and Qmake generators to GYP; I'm less certain if a CMake generator makes much sense, but I haven't thought about it as much. I'm not super familiar with any of these three tools, so I could be dead wrong. I completely agree that creating a new meta meta build system isn't a good idea, but sharing the common parts (which reduce the daily productivity) might be a step in the right direction. Using simple text files which contain the list of files (like the gpyi files already do today) isn't a new build system. It only offers the existing meta build systems (CMake, GYP, autotools, qmake) to use a common base. The remaining build systems can be ported to one of these systems or be adopted to use this file lists too. -- Patrick I suspect that we would quickly find that we would want some sort of support for conditionals and/or file inclusion in our simple text files, at which
[webkit-dev] Common build system (was Re: WebKit Wishes)
On Jan 30, 2013, at 3:24 PM, Dirk Pranke dpra...@chromium.org wrote: On Wed, Jan 30, 2013 at 1:50 PM, Filip Pizlo fpi...@apple.com wrote: Thanks for sharing this. On Jan 30, 2013, at 1:28 PM, Eric Seidel e...@webkit.org wrote: I wish we only had one build system (it were easy to add/remove/move files). I believe changes like http://trac.webkit.org/changeset/74849 are an unhealthy sign for the project. Adam is not the only person who has chosen to empty files instead of removing them. The pain of updating 8 build system is so great, we jump through hoops to avoid it. Which means it took us months to move JavaScriptCore/wtf to WTF, and will take us years to kill WebCore/platform. +1 This is a hard problem. It is a problem worth solving. Do you have more thoughts on this, particularly since you know quite well how both Xcode and gyp work? I suspect this is one of those things that it would be hard to achieve consensus on since there are so many stakeholders. But it may be fruitful to have a what if discussion about what this might look like. I think we can solve this problem if we agree that we want to. I think we haven't in the past mostly because we couldn't reach a consensus that it was worth solving enough to really try. I would love to see this fixed and would be glad to work on it. I think we should at least pursue this far enough to fully understand what our options are and what the costs and tradeoffs might be; does anyone disagree, and is anyone else willing to help pitch in? I think there are several possible ways we could solve this. One would be to switch to a common meta-build system. My understanding is that Apple's internal production build processes impose certain constraints here that I don't fully understand, but I know we've discussed the possibility of checking in generated project files as a workaround. Maybe there are other options as well to those constraints? I would love to discuss this further w/ someone from Apple ... It's far simplest for us if: (a) There is an Xcode project (or a Makefile) that builds the Mac port checked in to source control. (b) The generated project invokes only tools that are part of the default Mac OS X install. It may not be completely impossible to violate these requirements but it will require a lot of bureaucracy. (Also, just to get this out of the way, I don't think gyp needs to be the solution). Another alternative would be to write a script that did support at least the common use cases (add/move/delete files). There have been attempts in the past, but they have foundered on at least some perceived skepticism over how well this would work w/ XCode. That said, I don't think we've really pushed this to see. At some point this script might turn into a meta-meta-build system, which might be silly but also be the shortest path to the finish line. I suggest if there is interest in this we can start a new thread to discuss further ... My preference would be to use a common meta-build system with a comfortably human-readable and human-editable syntax, and checked in generated project files for those ports that need them. I think a key to making this work is to get Chromium and the Apple Mac port onto a common build system, which will probably require both Google and Apple ponying up at least one person to work on this project for a reasonable period of time. I think the plausible meta-build-systems to use would be CMake and Gyp, but in both cases it may be necessary to modify them to work well for everyone. I'd also like to add that I think a key related issue to common build system is common feature configuration. The many different ways ports control their feature flags is super confusing. I've long wanted to implement common configuration management, but have not had time. Cheers, Maciej ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
Hi ! I'd also like to add that I think a key related issue to common build system is common feature configuration. The many different ways ports control their feature flags is super confusing. I've long wanted to implement common configuration management, but have not had time. I think this would be a nice related project, but I wouldn't necessarily want to lump the two goals together without understanding what we'd be shooting for first. From what I know, CMake's support for feature configuration is much more mature than what you can do w/ GYP. It could certainly be a good motivator, though. My recent e-mail ( https://lists.webkit.org/pipermail/webkit-dev/2013-January/023368.html) to the list was an attempt to address the feature configuration problem - assuming that that a single build system is not an option. The idea was to use C macros to describe the rules for feature configurations and figure out a way to populate them to the various build systems (potentially using a C precompiler to eval the macros). Some of feature configuration flags does not need to be exposed to the build system these can just be #include-d to the source. I do not think that this is a technical superiors solution but perhaps it worth considering to discuss as it relies on tools and syntax we happened to already agree on and I felt that what might come out of it is better what we have now. Best, Laszlo ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On Jan 30, 2013, at 5:46 PM, Laszlo Gombos laszlo.gom...@gmail.com wrote: Hi ! I'd also like to add that I think a key related issue to common build system is common feature configuration. The many different ways ports control their feature flags is super confusing. I've long wanted to implement common configuration management, but have not had time. I think this would be a nice related project, but I wouldn't necessarily want to lump the two goals together without understanding what we'd be shooting for first. From what I know, CMake's support for feature configuration is much more mature than what you can do w/ GYP. It could certainly be a good motivator, though. My recent e-mail (https://lists.webkit.org/pipermail/webkit-dev/2013-January/023368.html) to the list was an attempt to address the feature configuration problem - assuming that that a single build system is not an option. The idea was to use C macros to describe the rules for feature configurations and figure out a way to populate them to the various build systems (potentially using a C precompiler to eval the macros). Some of feature configuration flags does not need to be exposed to the build system these can just be #include-d to the source. I do not think that this is a technical superiors solution but perhaps it worth considering to discuss as it relies on tools and syntax we happened to already agree on and I felt that what might come out of it is better what we have now. My past suggestion: have a separate Port.h header for every file, which turns on its set of PLATFORM, ENABLE and USE macros. Platform.h includes exclusively definitions of things that can be inferred from the build environment (like CPU, COMPILER, OS). Port.h files could delegate. So for example we might have: mac/Port.h -- cocoa/Port.h -- apple/Port.h -- default/Port.h the default Port.h file would include the WebKit project's shared recommendations for what features should be on or off. If a flag ends up here and we don't know of any port disabling it, we might just remove it. Including the right Port.h would be handled by include path magic. That's my idea in a nutshell. It would put all configuration logic into the preprocessor instead of a mix of the preprocessor and various build systems. Regards, Maciej ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On Thu, Jan 31, 2013 at 1:15 AM, Maciej Stachowiak m...@apple.com wrote: On Jan 30, 2013, at 3:24 PM, Dirk Pranke dpra...@chromium.org wrote: On Wed, Jan 30, 2013 at 1:50 PM, Filip Pizlo fpi...@apple.com wrote: Thanks for sharing this. On Jan 30, 2013, at 1:28 PM, Eric Seidel e...@webkit.org wrote: I wish we only had one build system (it were easy to add/remove/move files). I believe changes like http://trac.webkit.org/changeset/74849 are an unhealthy sign for the project. Adam is not the only person who has chosen to empty files instead of removing them. The pain of updating 8 build system is so great, we jump through hoops to avoid it. Which means it took us months to move JavaScriptCore/wtf to WTF, and will take us years to kill WebCore/platform. +1 This is a hard problem. It is a problem worth solving. Do you have more thoughts on this, particularly since you know quite well how both Xcode and gyp work? I suspect this is one of those things that it would be hard to achieve consensus on since there are so many stakeholders. But it may be fruitful to have a what if discussion about what this might look like. I think we can solve this problem if we agree that we want to. I think we haven't in the past mostly because we couldn't reach a consensus that it was worth solving enough to really try. I would love to see this fixed and would be glad to work on it. I think we should at least pursue this far enough to fully understand what our options are and what the costs and tradeoffs might be; does anyone disagree, and is anyone else willing to help pitch in? I think there are several possible ways we could solve this. One would be to switch to a common meta-build system. My understanding is that Apple's internal production build processes impose certain constraints here that I don't fully understand, but I know we've discussed the possibility of checking in generated project files as a workaround. Maybe there are other options as well to those constraints? I would love to discuss this further w/ someone from Apple ... It's far simplest for us if: (a) There is an Xcode project (or a Makefile) that builds the Mac port checked in to source control. (b) The generated project invokes only tools that are part of the default Mac OS X install. Another desirable property would be to move to a more automatic way of handling exported symbols: Currently each port that depends on this feature has its own .exp file or similar. I think if we could move to something that would allow for changing WebCore API without having to touch all those files, e.g. by consistently using WEBCORE_EXPORT macros would be a big win best -jochen It may not be completely impossible to violate these requirements but it will require a lot of bureaucracy. (Also, just to get this out of the way, I don't think gyp needs to be the solution). Another alternative would be to write a script that did support at least the common use cases (add/move/delete files). There have been attempts in the past, but they have foundered on at least some perceived skepticism over how well this would work w/ XCode. That said, I don't think we've really pushed this to see. At some point this script might turn into a meta-meta-build system, which might be silly but also be the shortest path to the finish line. I suggest if there is interest in this we can start a new thread to discuss further ... My preference would be to use a common meta-build system with a comfortably human-readable and human-editable syntax, and checked in generated project files for those ports that need them. I think a key to making this work is to get Chromium and the Apple Mac port onto a common build system, which will probably require both Google and Apple ponying up at least one person to work on this project for a reasonable period of time. I think the plausible meta-build-systems to use would be CMake and Gyp, but in both cases it may be necessary to modify them to work well for everyone. I'd also like to add that I think a key related issue to common build system is common feature configuration. The many different ways ports control their feature flags is super confusing. I've long wanted to implement common configuration management, but have not had time. Cheers, Maciej ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
