Re: Other build ideas (was: Re: Windows Build Environment futures

[Damjan Jovanovic]

On Fri, Oct 5, 2018 at 8:45 PM George Karalis  wrote:

>
> Hello everyone,
>
>
Hi, welcome :)


> Since am new to the project and working on understanding the build process
> this past
> week, I agree with Damjan that the build system needs that overhaul. I
> also agree that
> a MSVC upgrade and a 64-bit windows build, is of high priority. Changing
> the build
> system will take months.
>
> As of the MSVC upgrade I have figured out how the whole configure works
> and I ‘m
> now integrating MSVC++ 14.15 to oowintool script, and trying to integrate
> Windows 10
> SDK to the configure pipeline as well. After that I 'll try to figure out
> the build errors and
> fix everything to a successful build. If and when the upgrade completes,
> we should also
> investigate implementing targets - if there are none - for the build tool.
>
> e.g. To compile for windows XP, we need the Windows XP platform toolset,
> which installs
> Windows 7.1 SDK, among other things and can build for Windows XP using
> MSCV++
> 2017 and run something like: build —all —target=“winXP”.
>
>
Don't do that. Once the port to gbuild is complete, the "build" command
will disappear. Also the MSVC information is already stored in environment
variables from set_soenv(.in) -> winenv.set.sh like COM, COMID, CCVER,
CCNUMVER, etc.


> Since I am unfamiliar with UNO, I can’t provide any points there, but I
> agree with that
> separation of concerns, to keep only office functionality at the core of
> the project. Also
> another future idea would be to split - I don’t know if it's possible -
> each application,
> Writer, Calc, Impress etc. in its own separate module, so they can build
> independently
> and, maybe?, debug easier, since someone can work on features on only the
> app he/she
> wants.
>
> As for the build system, after some research, I concluded on three modern
> build pipelines,
> CMake, SCons and Meson. Personally, I am in favour of CMake, with the only
> argument
> that it’s backed by many large companies and that provides a safety for
> the future. The two
> systems are almost identical, SCons is more extensible due to Python, but
> all in all
> it’s just a weight of each build system's pros and cons. Also, how KDE
> switched  
> to CMake  was an interesting read. But,
> I think Damjan knows better about the project, so
> we can have an open discussion about which build system is better for
> OpenOffice.
>
>
CMake was already investigated by the original OOo team and found to be
lacking. Unlike CMake's module-at-a-time approach, gbuild and SCons take
the better approach of building a global project-wide list of dependencies
for each and every file, which is deterministic and always correct (unlike
the underspecified/overspecified module dependencies which give
inconsistent build results), and then optimally parallelizing the build at
a file level, instead of the artificial walls between modules limiting
build speed.


> I think that for now we should focus on upgrading our tools, new
> compilers, SDKs, because
> it's easier, it has been a week and I have done some progress already, and
> it will ease
> development a little, but ultimately changing to a modern build system can
> have tremendous
> potential for the project, modern build systems integrate perfect with
> Visual Studio,
> Xcode, working on an IDE it’s always nice .
>
>
Agreed :).


> Kind regards,
> George
>
>
Regards
Damjan

Re: Other build ideas (was: Re: Windows Build Environment futures

[George Karalis]

Hello everyone,

Since am new to the project and working on understanding the build process this 
past
week, I agree with Damjan that the build system needs that overhaul. I also 
agree that
a MSVC upgrade and a 64-bit windows build, is of high priority. Changing the 
build 
system will take months.

As of the MSVC upgrade I have figured out how the whole configure works and I ‘m
now integrating MSVC++ 14.15 to oowintool script, and trying to integrate 
Windows 10
SDK to the configure pipeline as well. After that I 'll try to figure out the 
build errors and
fix everything to a successful build. If and when the upgrade completes, we 
should also
investigate implementing targets - if there are none - for the build tool.

e.g. To compile for windows XP, we need the Windows XP platform toolset, which 
installs
Windows 7.1 SDK, among other things and can build for Windows XP using MSCV++
2017 and run something like: build —all —target=“winXP”.

Since I am unfamiliar with UNO, I can’t provide any points there, but I agree 
with that
separation of concerns, to keep only office functionality at the core of the 
project. Also
another future idea would be to split - I don’t know if it's possible - each 
application,
Writer, Calc, Impress etc. in its own separate module, so they can build 
independently 
and, maybe?, debug easier, since someone can work on features on only the app 
he/she
wants.

As for the build system, after some research, I concluded on three modern build 
pipelines,
CMake, SCons and Meson. Personally, I am in favour of CMake, with the only 
argument
that it’s backed by many large companies and that provides a safety for the 
future. The two 
systems are almost identical, SCons is more extensible due to Python, but all 
in all
it’s just a weight of each build system's pros and cons. Also, how KDE switched 
 
to CMake  was an interesting read. But, I 
think Damjan knows better about the project, so
we can have an open discussion about which build system is better for 
OpenOffice.

I think that for now we should focus on upgrading our tools, new compilers, 
SDKs, because
it's easier, it has been a week and I have done some progress already, and it 
will ease 
development a little, but ultimately changing to a modern build system can have 
tremendous 
potential for the project, modern build systems integrate perfect with Visual 
Studio,
Xcode, working on an IDE it’s always nice .

Kind regards,
George


> On 5 Oct 2018, at 20:32, Damjan Jovanovic  wrote:
> 
> The split wouldn't help us with newer MSVC as such.
> 
> What would help us there a lot, is SCons, as SCons supports many MSVC
> versions already, including MSVC 2017. It would be so wonderful if we were
> on SCons already: not only would we build with a wider range of MSVC
> versions, but we would need less work to keep up with future MSVC versions.
> But unfortunately I think it's a lot easier to move to a new MSVC, than it
> is to move to SCons. Especially given how heavy build development and
> testing would be necessary on all platforms, and we don't have a Mac or
> OS/2 or Solaris setup available...
> 
> On Fri, Oct 5, 2018 at 6:38 PM Peter Kovacs  wrote:
> 
>> Hi Damjan,
>> 
>> I o not understand the MSVC part. If we separate UNO it's own world.
>> (Independent of the Idea to spin it of into its own project)
>> 
>> will the update of the MSVC be easier?
>> 
>> 
>> All the best
>> 
>> Peter
>> 
>> 
>> On 10/5/18 10:15 AM, Damjan Jovanovic wrote:
>>> Hi
>>> 
>>> I now have a few other ideas regarding the building story.
>>> 
>>> The gbuild migration has been extraordinarily long and painful. Sadly
>> I've
>>> had to learn so much about gbuild, and implement so many new features in
>> it
>>> (symlinks, UNO IDL, Ant, bison, flex, now assembly...), that I am getting
>>> good at it now :(. A lot of what I've done would have to be reimplemented
>>> on SCons. But there are advantages to redoing stuff in SCons: for
>> example,
>>> we could run code in Python instead of calling external tools like awk,
>> tr
>>> and sed, and thus gain more portability, reduce dependencies, and ditch
>>> Cygwin.
>>> 
>>> However I consider SCons lower priority than a 64 bit Windows build, and
>> a
>>> newer MSVC (especially given our MSVC version is now unavailable!).
>>> 
>>> Also one of the things that's made work on all of those projects
>> difficult
>>> is the sheer size of AOO, the number and diversity of modules. One of my
>>> ideas is to split up AOO into 2 layers: an UNO layer at the bottom, and
>> an
>>> office layer at the top. This is something that's been in the pipeline
>> from
>>> the OOo days, and at one stage I think OOo was distributed with a
>> separate
>>> UNO installation, and there's even a file in main/ure/source/README that
>>> describes how the split would be done. These layers would be in separate
>>> directories, along the lines of ext_sources/, main/ and test/ now, and

Re: Other build ideas (was: Re: Windows Build Environment futures

[Damjan Jovanovic]

The split wouldn't help us with newer MSVC as such.

What would help us there a lot, is SCons, as SCons supports many MSVC
versions already, including MSVC 2017. It would be so wonderful if we were
on SCons already: not only would we build with a wider range of MSVC
versions, but we would need less work to keep up with future MSVC versions.
But unfortunately I think it's a lot easier to move to a new MSVC, than it
is to move to SCons. Especially given how heavy build development and
testing would be necessary on all platforms, and we don't have a Mac or
OS/2 or Solaris setup available...

On Fri, Oct 5, 2018 at 6:38 PM Peter Kovacs  wrote:

> Hi Damjan,
>
> I o not understand the MSVC part. If we separate UNO it's own world.
> (Independent of the Idea to spin it of into its own project)
>
> will the update of the MSVC be easier?
>
>
> All the best
>
> Peter
>
>
> On 10/5/18 10:15 AM, Damjan Jovanovic wrote:
> > Hi
> >
> > I now have a few other ideas regarding the building story.
> >
> > The gbuild migration has been extraordinarily long and painful. Sadly
> I've
> > had to learn so much about gbuild, and implement so many new features in
> it
> > (symlinks, UNO IDL, Ant, bison, flex, now assembly...), that I am getting
> > good at it now :(. A lot of what I've done would have to be reimplemented
> > on SCons. But there are advantages to redoing stuff in SCons: for
> example,
> > we could run code in Python instead of calling external tools like awk,
> tr
> > and sed, and thus gain more portability, reduce dependencies, and ditch
> > Cygwin.
> >
> > However I consider SCons lower priority than a 64 bit Windows build, and
> a
> > newer MSVC (especially given our MSVC version is now unavailable!).
> >
> > Also one of the things that's made work on all of those projects
> difficult
> > is the sheer size of AOO, the number and diversity of modules. One of my
> > ideas is to split up AOO into 2 layers: an UNO layer at the bottom, and
> an
> > office layer at the top. This is something that's been in the pipeline
> from
> > the OOo days, and at one stage I think OOo was distributed with a
> separate
> > UNO installation, and there's even a file in main/ure/source/README that
> > describes how the split would be done. These layers would be in separate
> > directories, along the lines of ext_sources/, main/ and test/ now, and
> > would build separately: first UNO, then office.
> >
> > How does this help? Changes can be contained. Different build systems
> could
> > be experimented with, for each. A Win64 bit UNO could be developed and
> > tested before we begin porting office modules to Win64. Ultimately, UNO
> is
> > a general framework for cross-platform multi-language component-based
> > development, similar to Microsoft's COM and Mozilla's XPCOM, and I think
> it
> > should be a completely separate project, that is hosted, developed, and
> > packaged independently of AOO (eg. http://uno.apache.org), and is a
> just a
> > compile-time and run-time dependency for us, that we download, use, and
> > ship just like zlib and libjpeg.
> >
> > I really think the world needs a cross-platform multi-language
> > component-based framework. Microsoft's COM is Windows-only, and Mozilla
> is
> > gradually getting rid of XPCOM. There are few other implementations, and
> we
> > have what seems to be the only permissively licensed one, and the only
> one
> > supporting exception handling between different languages (I think both
> COM
> > and XPCOM don't support exceptions and require return codes; the infamous
> > HRESULT with its bitfields).
> >
> > Almost all the UNO modules are using gbuild already (it's mostly office
> > that is troublesome and does lots of custom things during the build :-),
> so
> > a 100% gbuild UNO can be split off soon, and UNO can then potentially
> start
> > using SCons, a newer MSVC, and/or producing a Win64 build, before office
> > even finishes the gbuild migration.
> >
> > There is some work involved. We would have to split up configure.ac,
> > duplicate some of the build scripts, somehow deliver UNO files into the
> > office build. Quite a few of the office prj/build.lst would have to
> change
> > to not depend on the modules that move. Nothing would initially improve;
> > only after other changes could dependencies be reduced. Not needing
> Cygwin
> > requires both UNO and office to use SCons, a Win64 office also requires
> > both. The biggest initial benefit might be that since UNO doesn't change
> > often, rebuilding would be faster, as only office modules would rebuild.
> > And the office source code tree would be smaller and clearer.
> >
> > On the downside, the same version of MSVC must compile both UNO and
> office,
> > as C++ symbols from different MSVC versions are incompatible. This pretty
> > much means we have to build UNO from source inside office.
> >
> > Anyway, what do you think?
> >
> > Damjan
> >
> >
> > On Sun, Sep 23, 2018 at 10:46 AM Peter Kovacs  wrote:
> >
> >> At the 

Re: Other build ideas (was: Re: Windows Build Environment futures

[Peter Kovacs]

Hi Damjan,

I o not understand the MSVC part. If we separate UNO it's own world. 
(Independent of the Idea to spin it of into its own project)


will the update of the MSVC be easier?


All the best

Peter


On 10/5/18 10:15 AM, Damjan Jovanovic wrote:

Hi

I now have a few other ideas regarding the building story.

The gbuild migration has been extraordinarily long and painful. Sadly I've
had to learn so much about gbuild, and implement so many new features in it
(symlinks, UNO IDL, Ant, bison, flex, now assembly...), that I am getting
good at it now :(. A lot of what I've done would have to be reimplemented
on SCons. But there are advantages to redoing stuff in SCons: for example,
we could run code in Python instead of calling external tools like awk, tr
and sed, and thus gain more portability, reduce dependencies, and ditch
Cygwin.

However I consider SCons lower priority than a 64 bit Windows build, and a
newer MSVC (especially given our MSVC version is now unavailable!).

Also one of the things that's made work on all of those projects difficult
is the sheer size of AOO, the number and diversity of modules. One of my
ideas is to split up AOO into 2 layers: an UNO layer at the bottom, and an
office layer at the top. This is something that's been in the pipeline from
the OOo days, and at one stage I think OOo was distributed with a separate
UNO installation, and there's even a file in main/ure/source/README that
describes how the split would be done. These layers would be in separate
directories, along the lines of ext_sources/, main/ and test/ now, and
would build separately: first UNO, then office.

How does this help? Changes can be contained. Different build systems could
be experimented with, for each. A Win64 bit UNO could be developed and
tested before we begin porting office modules to Win64. Ultimately, UNO is
a general framework for cross-platform multi-language component-based
development, similar to Microsoft's COM and Mozilla's XPCOM, and I think it
should be a completely separate project, that is hosted, developed, and
packaged independently of AOO (eg. http://uno.apache.org), and is a just a
compile-time and run-time dependency for us, that we download, use, and
ship just like zlib and libjpeg.

I really think the world needs a cross-platform multi-language
component-based framework. Microsoft's COM is Windows-only, and Mozilla is
gradually getting rid of XPCOM. There are few other implementations, and we
have what seems to be the only permissively licensed one, and the only one
supporting exception handling between different languages (I think both COM
and XPCOM don't support exceptions and require return codes; the infamous
HRESULT with its bitfields).

Almost all the UNO modules are using gbuild already (it's mostly office
that is troublesome and does lots of custom things during the build :-), so
a 100% gbuild UNO can be split off soon, and UNO can then potentially start
using SCons, a newer MSVC, and/or producing a Win64 build, before office
even finishes the gbuild migration.

There is some work involved. We would have to split up configure.ac,
duplicate some of the build scripts, somehow deliver UNO files into the
office build. Quite a few of the office prj/build.lst would have to change
to not depend on the modules that move. Nothing would initially improve;
only after other changes could dependencies be reduced. Not needing Cygwin
requires both UNO and office to use SCons, a Win64 office also requires
both. The biggest initial benefit might be that since UNO doesn't change
often, rebuilding would be faster, as only office modules would rebuild.
And the office source code tree would be smaller and clearer.

On the downside, the same version of MSVC must compile both UNO and office,
as C++ symbols from different MSVC versions are incompatible. This pretty
much means we have to build UNO from source inside office.

Anyway, what do you think?

Damjan


On Sun, Sep 23, 2018 at 10:46 AM Peter Kovacs  wrote:


At the same time, the migration from Windows 32bit Version to support
64bit has been started. The build currently breaks somewhere. A
continuation would be awesome.

And we are moving the build environment from dmake to gmake. There are
65 modules left. This activity has a huge impact because all platforms
are effected. And according to Damjan, the low fruits have been already
migrated, and the ones left are not easy.

@Damjan, you still would like to migrate then to SCON after gmake, btw?
I would still like to, even I do not count my voice much, because I did
not drive this topic much. :(




-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Other build ideas (was: Re: Windows Build Environment futures (was: Open Office Contribution))

[Damjan Jovanovic]

Hi

I now have a few other ideas regarding the building story.

The gbuild migration has been extraordinarily long and painful. Sadly I've
had to learn so much about gbuild, and implement so many new features in it
(symlinks, UNO IDL, Ant, bison, flex, now assembly...), that I am getting
good at it now :(. A lot of what I've done would have to be reimplemented
on SCons. But there are advantages to redoing stuff in SCons: for example,
we could run code in Python instead of calling external tools like awk, tr
and sed, and thus gain more portability, reduce dependencies, and ditch
Cygwin.

However I consider SCons lower priority than a 64 bit Windows build, and a
newer MSVC (especially given our MSVC version is now unavailable!).

Also one of the things that's made work on all of those projects difficult
is the sheer size of AOO, the number and diversity of modules. One of my
ideas is to split up AOO into 2 layers: an UNO layer at the bottom, and an
office layer at the top. This is something that's been in the pipeline from
the OOo days, and at one stage I think OOo was distributed with a separate
UNO installation, and there's even a file in main/ure/source/README that
describes how the split would be done. These layers would be in separate
directories, along the lines of ext_sources/, main/ and test/ now, and
would build separately: first UNO, then office.

How does this help? Changes can be contained. Different build systems could
be experimented with, for each. A Win64 bit UNO could be developed and
tested before we begin porting office modules to Win64. Ultimately, UNO is
a general framework for cross-platform multi-language component-based
development, similar to Microsoft's COM and Mozilla's XPCOM, and I think it
should be a completely separate project, that is hosted, developed, and
packaged independently of AOO (eg. http://uno.apache.org), and is a just a
compile-time and run-time dependency for us, that we download, use, and
ship just like zlib and libjpeg.

I really think the world needs a cross-platform multi-language
component-based framework. Microsoft's COM is Windows-only, and Mozilla is
gradually getting rid of XPCOM. There are few other implementations, and we
have what seems to be the only permissively licensed one, and the only one
supporting exception handling between different languages (I think both COM
and XPCOM don't support exceptions and require return codes; the infamous
HRESULT with its bitfields).

Almost all the UNO modules are using gbuild already (it's mostly office
that is troublesome and does lots of custom things during the build :-), so
a 100% gbuild UNO can be split off soon, and UNO can then potentially start
using SCons, a newer MSVC, and/or producing a Win64 build, before office
even finishes the gbuild migration.

There is some work involved. We would have to split up configure.ac,
duplicate some of the build scripts, somehow deliver UNO files into the
office build. Quite a few of the office prj/build.lst would have to change
to not depend on the modules that move. Nothing would initially improve;
only after other changes could dependencies be reduced. Not needing Cygwin
requires both UNO and office to use SCons, a Win64 office also requires
both. The biggest initial benefit might be that since UNO doesn't change
often, rebuilding would be faster, as only office modules would rebuild.
And the office source code tree would be smaller and clearer.

On the downside, the same version of MSVC must compile both UNO and office,
as C++ symbols from different MSVC versions are incompatible. This pretty
much means we have to build UNO from source inside office.

Anyway, what do you think?

Damjan


On Sun, Sep 23, 2018 at 10:46 AM Peter Kovacs  wrote:

>
> At the same time, the migration from Windows 32bit Version to support
> 64bit has been started. The build currently breaks somewhere. A
> continuation would be awesome.
>
> And we are moving the build environment from dmake to gmake. There are
> 65 modules left. This activity has a huge impact because all platforms
> are effected. And according to Damjan, the low fruits have been already
> migrated, and the ones left are not easy.
>
> @Damjan, you still would like to migrate then to SCON after gmake, btw?
> I would still like to, even I do not count my voice much, because I did
> not drive this topic much. :(
>
>