Re: about build environment development (was: declaring gbuild target dependencies)

2020-04-15 Thread Damjan Jovanovic
On Wed, Apr 15, 2020 at 3:15 PM Jim Jagielski  wrote:

>
>
> > On Apr 15, 2020, at 3:01 AM, Damjan Jovanovic  wrote:
> >
> >
> > We are also thin on new contributors, and I recall you saying they're
> > largely scared off by the current build system.
> >
>
> Two points:
>
>   1. I doubt that by the time we finish porting to a whole new build
> system, we will even have anyone *wanting* to contribute. With each delay
> and push-out on releases, and more time spent working on the build system
> instead of AOO itself, we become less and less relevant. Is that really a
> priority we should be focusing on? Are the number of people knowledgeable
> around scons really greater than what we have now? But I could be wrong,
> which leads me to #2...
>
>
What would you recommend we focus on instead then?

Ideally, new contributors wouldn't need to be knowledgeable about scons.
The build should be easier to perform, hopefully just "./configure"
followed by "scons" (and scons even implements features that can subsume
./configure too). Already, scons doesn't need the "source winenv.set.sh"
and "cd instsetoo_native" steps to build its modules.


>   2. "The conversion from gbuild to scons would largely be automated, fast
> and correct." If that is the case, let's test that theory right now.
>

This has been possible for some time. In the scons-build branch, you can do
the following:

$ cd gotoSCons
$ mvn package
$ java -cp target/gotoSCons-0.1-SNAPSHOT.jar
org.apache.openoffice.gotoSCons.GBuildConverter parsingAnalysis ../main
(per-module output)
Could parse: [MathMLDTD, UnoControls, animations, cosv, cppcanvas,
drawinglayer, eventattacher, fileaccess, i18nutil, idl, io, rdbmaker,
registry, remotebridges, sane, store, svgio, twain, ucbhelper, unixODBC,
xmlreader, xmlscript]
22 out of 105 gbuild modules are parseable

That means 22 modules can already be converted, completely and correctly.
As we add more features to the converter (AllLangResTarget, UnoApi, Junit,
GoogleTest, etc.), that 22 will increase.

The per-module gbuild files are easy to parse. Parsing the syntax takes
only 3 methods and < 100 lines of Java. The non-deterministic ones with
"ifeq ($(GUIBASE),aqua)" require some manual work, but even there, a lot
can be automated. There is some more work involved in semantic conversion:
understanding and converting specific gbuild commands, converting paths to
scons format, etc. but even so, we're on just 1913 lines of code in total
for the converter.

The hard part is to convert gbuild functions in main/solenv/gbuild into
scons, for example, the worst case scenario is AllLangResTarget, for which
this monstrous dependency tree needs to be implemented, with 4 layers of
intermediate targets and complex actions with side effects and header
dependencies (not shown):

#  AllLangResTarget(name)
#  (meta-target; delivers an empty timestamp file)
#^ ^
#   /   \
#  / \
# /   \
#  ResTarget(nameen-US,name,en-US)
ResTarget(nameen-GB,name,en-GB)
#  $(WORKDIR)/ResTarget/$(resName).res
$(WORKDIR)/ResTarget/$(resName).res
#  $(WORKDIR)/ResTarget/nameen-US.res
 $(WORKDIR)/ResTarget/nameen-GB.res
#^   ^^
#| rsc   ||
#|   ||
#  SrsTarget   SrsTarget
SrsTarget
#  $(WORKDIR)/SrsTarget/$(srsName).srs
#  $(WORKDIR)/SrsTarget/uui/res.srs
#^
#| concatenate
#+--+
#|   \
#|\
#  SrcPartTarget   SrcPartTarget
#  $(WORKDIR)/SrsPartTarget/$(srsPartName)
#  $(WORKDIR)/SrsPartTarget/uui/source/ids.src
#^   ^
#| rsc   | rsc
#|   |
# (when not translating) |   | (when translating)
#|   |
#|SrcPartMergeTarget
#|$(WORKDIR)/SrsPartMergeTarget/$(1)
#|
 $(WORKDIR)/SrsPartMergeTarget/uui/source/ids.src
#| ^
#|/
#|   / transex3
#|  / (when translating)
#  $(srsPartName)  /
#  uuid/source/ids.src
#

Damjan


Re: about build environment development (was: declaring gbuild target dependencies)

2020-04-15 Thread Jim Jagielski



> On Apr 15, 2020, at 3:01 AM, Damjan Jovanovic  wrote:
> 
> 
> We are also thin on new contributors, and I recall you saying they're
> largely scared off by the current build system.
> 

Two points:

  1. I doubt that by the time we finish porting to a whole new build system, we 
will even have anyone *wanting* to contribute. With each delay and push-out on 
releases, and more time spent working on the build system instead of AOO 
itself, we become less and less relevant. Is that really a priority we should 
be focusing on? Are the number of people knowledgeable around scons really 
greater than what we have now? But I could be wrong, which leads me to #2...

  2. "The conversion from gbuild to scons would largely be automated, fast and 
correct." If that is the case, let's test that theory right now.
-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org



Re: about build environment development (was: declaring gbuild target dependencies)

2020-04-15 Thread Damjan Jovanovic
On Wed, Apr 15, 2020 at 6:46 AM Peter Kovacs  wrote:

> If one wants to tap in our build system he needs to understand Perl,
> shell, make, ant, XML, configure, ...
>
> This is just way to complicated, especially if you want to bring in an
> IDE to ease code development.
>
>
> Damjan is not very happy with the features gmake offers. I am not sure
> where exactly the Issue is.
>
> He is opting for SCONs, with the option to extend the build system with
> python. And IMHO on Damjan
>
> Side he is quite serious about it.
>
>
> Everyone else has not expressed any opinion on this development, so I am
> not sure everyone is aware. The last discussion on this topic,
>
> consent has been strongly to make gmake work.
>

We already took that approach to its limit, and I don't believe we can go
much further. Most of gmake works by luck. The simplest things break, make
no sense, and cannot be debugged, eg. sometimes *_add_files breaks, but
*_add_file works, despite the fact the former just calls the latter. There
are already some hacks in modules to work around gmake's brokenness...


>
> Another objection is that we got some heavy negative experience report
> from the serf community about SCONS.
>
>
It wasn't the entire "serf community", just brane@ on 30 Oct 2019:
---snip---
As a far-too-long-time user of Scons (in Serf), let me just add a
caution: Scons is very, very broken and not at all well maintained.
Writing a truly cross-platform, cross-toolchain SConstruct file for
anything other than really trivial cases amounts to writing a *lot* of
platform-specific Python code to make the builds work.

If you're already moving to gmake (without autotools, and I expect using
Cygwin on Windows), then I suggest you finish the transition, then leave
well enough alone.
---snip---

My experience so far has been exactly the opposite: it is far easier to get
scons building cross-platform, than GNU make + countless shell tools which
also drag in the whole of Cygwin.

To summarize quickly:
I tried to avoid scons before and use gmake, and we already got as far as
we could.
scons is not a decision I made lightly, it's a decision I made because it's
that good.
Python isn't my favorite language, but what we gain from scons is
significant enough for me to want to learn more Python.
By using scons, Cygwin left the building without me even trying.
scons has a 20 year history, supports OS/2 and numerous other platforms,
supports Python 2 and 3, supports more MSVC versions than we do (including
Visual Studio 2019), would allow us to build almost anywhere.
It is packed full of many advanced features we need, like symlinking
libX.so -> libX.so.2, automated header dependency scanning, flex, yacc,
Java, Objective C, precompiled headers on MSVC, fully parallelizes builds
across module boundaries, can work out minimal rebuilds using checksums of
file contents instead of inode timestamps, is extensible by design, has
readable source code, can be debugged, and is under a liberal license
(MIT). I've looked, and nothing else really comes close; every other build
tool would require considerable further development (AOO has already tried
building big build systems around both dmake and GNU make; my better
experience with Ant/Maven makes me more hopeful about a richer higher-level
build tool that automates more of the work internally).
The conversion from gbuild to scons would largely be automated, fast and
correct. (We could also keep the scons files in a format that would allow
easier automated conversion to something else later.)


> Which are switching from, SCONS to cmake.
>
>
> So in the end we do not have Consent where we want to go. And currently
> it is heavily influenced by Damjan. And this is imho very thin.
>
>
Then we were always very thin on gbuild too.

You've also done some scons development.

We are also thin on new contributors, and I recall you saying they're
largely scared off by the current build system.


> I am still like the Idea most to go in the direction of ant / maven,
> despite its flaws. But I am not negative on SCONS either. My main point
> is we need something that
>
> offers a better architecture and is easier to handled and maintained.
>
>
> Also what we could try is making use of something like portage. Portage
> is pretty easy to use repostory manager used by gentoo, whioch also had
> a community prior to homebrew on mac. It is not very difficult to
> setup.  But it is build to make different build system work together. So
> we could have a build repository, that builds our dependencies. We
> reconstruct our monolith in smaller build libraries, like UNOcore,
> OOFrame, UNOGUI, OOapp, OOpython, StarBasic, OOwizards, extentionXYZ
> (Just saying something), and pick the best build system (cmake, gmake,
> ant, maven, SCONS) for each library. Also we could think on incubating
> Starbasic or UNO, as own Project if they become more interesting. Since
> Portage is made for source build, but can also handle binaries, maybe we