[webkit-dev] Shrinking Git clone size (WebKit Transition to Git)

[Ryosuke Niwa]

Hi all,

While we're making the transition to Git, I think we should try to
shrink the Git clone size. Right now, it's 10GB and I suspect we can
cut it down quite a bit without sacrificing much. For example, we can
exclude any history for LayoutTests/platform/chromium* and
LayoutTests/platforn/*qt*. I bet that would reduce the clone size
considerably.

- R. Niwa
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Reducing / removing use of Makefile based DerivedSources.make

[Sam Weinig]

One direct benefit of moving away from DerivedSources.make would likely be (I 
say likely, as the details of how it works out are far from certain in my mind) 
removing at least one place that a IDL file needs to be listed as we would not 
need to explicitly the list the IDL file in DerivedSources.make, and would only 
need to ensure it was in some *input.xcfilelist.

But the idea of sharing “input” lists is very compelling. Just focusing on the 
IDL files themselves, I wonder if we could construct an input file that could 
serve both CMake and Xcode. It seems like CMake has the ability to read files 
into variables (https://cmake.org/cmake/help/latest/command/file.html#reading 
) so perhaps we 
could construct IDLs.txt that worked both as an input CMake and Xcode.

One issue that I don’t know how to solve in a good way is how to make adding 
files to the Xcode project easier. Perhaps this is somewhere that a script 
could help? Rather than adding the files to Sources.txt and then manually 
adding them to the Xcode project, perhaps we should just have a script that 
does this for us (either via manually invoking it or with the post processing 
that generates the xcfilelists?)

- Sam

> On Oct 18, 2020, at 10:38 AM, Darin Adler  wrote:
> 
> It would be wonderful to drastically cut down the repeated mentions of the 
> same filenames. People not on Xcode platforms probably focus primarily on 
> Xcode projects for this frustration. But I find it ridiculous how many places 
> I have to add things. For example, to add AbstractRange I had to add 
> filenames to:
> 
> Shared
> 
> - Sources.txt: Two source files and one IDL-generated file.
> 
> CMake:
> 
> - CMakeLists.txt: IDL file.
> - Headers.cmake: One source file and one IDL-generated file.
> 
> Xcode:
> 
> - DerivedSources.make: IDL file
> - WebCore.xcodeproj/project.pbxproj: IDL file, two source files, and Two 
> IDL-generated files.
> 
> And also, updating these checked-in generated files (that we’d need 
> regardless of build system, due to Apple build requirements):
> 
> - DerivedSources-input.xcfilelist
> - DerivedSources-output.xcfilelist
> 
> It seems like we should have a goal that you should not have to list 
> IDL-generated file names. That will be hard to accomplish in Xcode, though. 
> Maybe the reason we haven’t been so ambitious about getting this right for 
> CMake is that it’s so unreachable a goal for Xcode, given that we want the 
> files to show up in a clean way in the IDE?
> 
> And it would be good to find a way to share the lists of IDL files like we do 
> the list of source files. That seems relevant to this discussion of changing 
> DerivedSources.make.
> 
> — Darin
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Reducing / removing use of Makefile based DerivedSources.make

[Darin Adler]

It would be wonderful to drastically cut down the repeated mentions of the same 
filenames. People not on Xcode platforms probably focus primarily on Xcode 
projects for this frustration. But I find it ridiculous how many places I have 
to add things. For example, to add AbstractRange I had to add filenames to:

Shared

- Sources.txt: Two source files and one IDL-generated file.

CMake:

- CMakeLists.txt: IDL file.
- Headers.cmake: One source file and one IDL-generated file.

Xcode:

- DerivedSources.make: IDL file
- WebCore.xcodeproj/project.pbxproj: IDL file, two source files, and Two 
IDL-generated files.

And also, updating these checked-in generated files (that we’d need regardless 
of build system, due to Apple build requirements):

- DerivedSources-input.xcfilelist
- DerivedSources-output.xcfilelist

It seems like we should have a goal that you should not have to list 
IDL-generated file names. That will be hard to accomplish in Xcode, though. 
Maybe the reason we haven’t been so ambitious about getting this right for 
CMake is that it’s so unreachable a goal for Xcode, given that we want the 
files to show up in a clean way in the IDE?

And it would be good to find a way to share the lists of IDL files like we do 
the list of source files. That seems relevant to this discussion of changing 
DerivedSources.make.

— Darin
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Reducing / removing use of Makefile based DerivedSources.make

[Darin Adler]

Oops, I was in the middle of writing this and hit send by mistake
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Reducing / removing use of Makefile based DerivedSources.make

[Darin Adler]

Please note that this is a Cocoa-only, Xcode-only conversation.

As Sam pointed out, there’s a whole different “will you please switch to CMake” 
conversation. That’s not an easy switch to make for programmers working on the 
Apple platforms, but we might do it at some point if we can work out the issues 
Sam mentioned.

> On Oct 17, 2020, at 10:00 AM, Sam Weinig  wrote:
> 
> While I can’t recall the initial reasons for using Makefiles for derived 
> sources, over the years there have been a number of advantages to it that I 
> do know of. One clear advantage, that is no longer applicable, was code 
> sharing, as earlier in the project, at least the Apple Windows port did 
> utilize these Makefiles.

Pretty sure I was the one who did this. And I do recall the initial reasons. 
There were only two.

1) Code sharing

I believed at the time that the rules for building derived sources were tricky 
and hard to get right, and proposed that we would use the 
lowest-common-denominator build system, make, across all platforms. This 
strategy didn’t work because right from the beginning people working with 
various build systems used copies of the derived sources rules that I wrote 
rather than invoking make, and never tried sharing code.

I think this was a missed opportunity to share complicated code, but it 
happened a very long time ago, and I made my peace with it back then.

2) Xcode build system deficiencies

At that time, Xcode’s build system had too many problems for these kinds of 
complex rules. Many years later, that may no longer be true.

Since we’re not even trying to do the code sharing, I think it’s fine to stop 
using makefiles now.

If we’re deeply committed to switching to CMake at some point in future, one 
way to take a step in that direction would be to return to the original code 
sharing goal, and use CMake for derived sources, so we have code sharing. I 
don’t know how modular CMake is and how practical it is to use it for part, but 
not all, of the build process with Xcode. People using CMake for everything 
might be frustrated with the constraints this could place on how CMake is used.

I think an important third goal that I did not have at the start, but have now, 
is the one Ryosuke mentioned:

3) Minimize the number of places files are listed, so it’s easy to add, remove, 
and rename files of various types.

It would be wonderful to drastically cut down the repeated mentions of the same 
filenames. People not on Xcode platforms probably focus primarily on Xcode 
projects for this frustration. But I find it ridiculous how many places I have 
to add things. For example, to add AbstractRange I had to add filenames to:

Shared:

Sources.txt

CMake:

CMakeLists.txt
Headers.cmake

Xcode:

WebCore.xcodeproj/project.pbxproj

And also, the checked-in generated files (that we’d need regardless of build 
systems, due to Apple build requirements):

DerivedSources-input.xcfilelist
DerivedSources-output.xcfilelist

— Darin
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Reducing / removing use of Makefile based DerivedSources.make

[Sam Weinig]

> On Oct 17, 2020, at 12:34 PM, Fujii Hironori  > wrote:
> 
> 
> On Sun, Oct 18, 2020 at 2:00 AM Sam Weinig  > wrote:
> 
> What are your thoughts on this? 
> 
> I'm disappointed. I want to use CMake for all ports.


I hear you, it would indeed be nice to have a single build system, but I’d 
really like to keep this thread specifically about the explicit issue of 
DerivedSources.make use for the current Xcode based builds.

I encourage you to start a separate thread about moving to a CMake only world. 
There are plenty of seperate challenges there (e.g. generating identical 
output, maintaining Xcode based work flows, etc.), and I am not sure how 
practical it is, but I think enumerating the challenging and discussing them is 
important.

Thanks,
- Sam

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Reducing / removing use of Makefile based DerivedSources.make

[Ryosuke Niwa]

On Sat, Oct 17, 2020 at 10:00 AM Sam Weinig  wrote:
>
> Hi webkit-dev,
>
> I’d like to propose, and gauge feedback on, reducing (with the goal of 
> ultimately removing) the use of Makefile based DerivedSources.make.
>
> My understanding is that currently only the Xcode based ports still use 
> DerivedSources.make, as all the CMake based ones have moved derived source 
> generation to within CMake, so that should limit the scope of who this might 
> affect.
>
>
> Why do we use Makefiles today?
>
> While I can’t recall the initial reasons for using Makefiles for derived 
> sources, over the years there have been a number of advantages to it that I 
> do know of. One clear advantage, that is no longer applicable, was code 
> sharing, as earlier in the project, at least the Apple Windows port did 
> utilize these Makefiles. Another was to work around some limitations in what 
> dependencies Xcode was able to track with build rules. It seems at least some 
> of the problems with build rules are no longer an issue, as we can now 
> specify inputs to build rules, but I don’t if other problems will still be 
> there, but for some prototyping I did, nothing yet has come up.
>
>
> What would we move to?
>
> As this only affects the Xcode based ports, we would move to distinct script 
> phases and build rules in the Xcode project.
>
>
> Why make this change? What’s the benefit?
>
> There are few reasons to consider this. One advantage is simplifying the 
> build system. Rather than two dependency systems (one for Xcode, one for the 
> Makefile) we reduce it down to one. And with additional knowledge of the 
> stages and dependencies, Xcode could potentially parallelize more phases. We 
> would would also save some time by avoiding invoking make in the first place.
>
> We also have a bit of an issue with make itself, as due to system 
> requirements, we are forever stuck with Make 3.81, which is coming up on 
> being 15 years old. More than once in the last year I have tried to 
> troubleshoot makefile issues, looking for resources on the web, only to be 
> stymied because the solutions I found required newer make.
>
>
> What are the downsides?
>
> One potential downside will be that it will be a bit harder for those without 
> Xcode to add new types of derived sources. I am not sure how much a real 
> problem in practice this will be, as editing project.pbxproj files is already 
> required for just adding new files, but I want to call it out anyway.

Do we need to dig up some kind of bespoke Xcode UI to add a new IDL
file after this? I always find all those build phase things in Xcode
to be impossible to edit.

> What are your thoughts on this? Are there additional reasons we might want to 
> stick with or move away from Makefile based derived sources?

Is there some way we can use something like Sources.txt to list the
files from which derived sources are created? DerivedSources.make is
annoying to edit but the equivalent CMake file is equally annoying to
edit. If we had a simple list of files like we do for unified sources,
it would make an average WebKit contributor's life way easier.

- R. Niwa
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev