Re: [question] build infra structure.
On 01.11.2012 18:31, jan iversen wrote: you See below please. On 1 November 2012 18:18, Andre Fischer awf@gmail.com wrote: On 01.11.2012 17:57, jan iversen wrote: See below please. THANKS for your VERY informative answer, it helped me a lot. I was of the simple idea, that we pursued a simple build process made up of gnuMake and an addon to gather for the shortcoming of gnumake in respect of cascaded makefiles. We are in the process of migrating from dmake to GNU make. When that is finished then we will have essentially one single makefile. Well, there will be one top level makefile that includes all the other makefiles. But there will not one make process that starts other makes in subprocesses. That would be evil, or so I have been told, see http://wiki.openoffice.org/**wiki/Build_System_Analysishttp://wiki.openoffice.org/wiki/Build_System_Analysis I am in the process of changing the l10n process. Currently it runs on one makefile that searches all directories, I want to change that to a target in every local makefile (build.lst). I am aware of that and am glad that you follow this approach. I am convinced that among a lot of other improvements, we could make the localization process a lot faster by a) using make (dmake or GNU make) for controlling what to do when and b) by integrating it into the build process to update the pootle data from time to time. Can I attach myself to your progress, or would you suggest that I attach my development to the current build process. my timeline is somewhat around new year. Conversion of Apache OpenOffice from dmake to gbuild is going very slow at the moment. I am afraid that you will have work with the current build process. But I am willing to help. I hope to see your presentation on video later, due to personal budget restriction (dont we all have that) I cannot participate. Sorry to hear that, I would have liked to meet you. Well if you come to FOSDEM, we can have a long chat. My problem is that I am currently only a contributor so that ticket alone is 600,- EUR. Yeah, I know what you mean. And I will think about FOSDEM. I am also prepared for google/skype videochats. That is good to know. Maybe after ApacheCon. -Andre
Re: [question] build infra structure.
Thanks for offering your help, I will definitively come back to that. Just one question, is there a design document or something where I can read how the new makefile concept is going to work ? Jan. On 2 November 2012 09:26, Andre Fischer awf@gmail.com wrote: On 01.11.2012 18:31, jan iversen wrote: you See below please. On 1 November 2012 18:18, Andre Fischer awf@gmail.com wrote: On 01.11.2012 17:57, jan iversen wrote: See below please. THANKS for your VERY informative answer, it helped me a lot. I was of the simple idea, that we pursued a simple build process made up of gnuMake and an addon to gather for the shortcoming of gnumake in respect of cascaded makefiles. We are in the process of migrating from dmake to GNU make. When that is finished then we will have essentially one single makefile. Well, there will be one top level makefile that includes all the other makefiles. But there will not one make process that starts other makes in subprocesses. That would be evil, or so I have been told, see http://wiki.openoffice.org/wiki/Build_System_Analysishttp://wiki.openoffice.org/**wiki/Build_System_Analysis htt**p://wiki.openoffice.org/wiki/**Build_System_Analysishttp://wiki.openoffice.org/wiki/Build_System_Analysis I am in the process of changing the l10n process. Currently it runs on one makefile that searches all directories, I want to change that to a target in every local makefile (build.lst). I am aware of that and am glad that you follow this approach. I am convinced that among a lot of other improvements, we could make the localization process a lot faster by a) using make (dmake or GNU make) for controlling what to do when and b) by integrating it into the build process to update the pootle data from time to time. Can I attach myself to your progress, or would you suggest that I attach my development to the current build process. my timeline is somewhat around new year. Conversion of Apache OpenOffice from dmake to gbuild is going very slow at the moment. I am afraid that you will have work with the current build process. But I am willing to help. I hope to see your presentation on video later, due to personal budget restriction (dont we all have that) I cannot participate. Sorry to hear that, I would have liked to meet you. Well if you come to FOSDEM, we can have a long chat. My problem is that I am currently only a contributor so that ticket alone is 600,- EUR. Yeah, I know what you mean. And I will think about FOSDEM. I am also prepared for google/skype videochats. That is good to know. Maybe after ApacheCon. -Andre
Re: [question] build infra structure.
On 11/2/12 9:46 AM, jan iversen wrote: Thanks for offering your help, I will definitively come back to that. Just one question, is there a design document or something where I can read how the new makefile concept is going to work ? maybe this helps you to get started http://wiki.openoffice.org/wiki/Build_Environment_Effort and all the related links on the right Juergen Jan. On 2 November 2012 09:26, Andre Fischer awf@gmail.com wrote: On 01.11.2012 18:31, jan iversen wrote: you See below please. On 1 November 2012 18:18, Andre Fischer awf@gmail.com wrote: On 01.11.2012 17:57, jan iversen wrote: See below please. THANKS for your VERY informative answer, it helped me a lot. I was of the simple idea, that we pursued a simple build process made up of gnuMake and an addon to gather for the shortcoming of gnumake in respect of cascaded makefiles. We are in the process of migrating from dmake to GNU make. When that is finished then we will have essentially one single makefile. Well, there will be one top level makefile that includes all the other makefiles. But there will not one make process that starts other makes in subprocesses. That would be evil, or so I have been told, see http://wiki.openoffice.org/wiki/Build_System_Analysishttp://wiki.openoffice.org/**wiki/Build_System_Analysis htt**p://wiki.openoffice.org/wiki/**Build_System_Analysishttp://wiki.openoffice.org/wiki/Build_System_Analysis I am in the process of changing the l10n process. Currently it runs on one makefile that searches all directories, I want to change that to a target in every local makefile (build.lst). I am aware of that and am glad that you follow this approach. I am convinced that among a lot of other improvements, we could make the localization process a lot faster by a) using make (dmake or GNU make) for controlling what to do when and b) by integrating it into the build process to update the pootle data from time to time. Can I attach myself to your progress, or would you suggest that I attach my development to the current build process. my timeline is somewhat around new year. Conversion of Apache OpenOffice from dmake to gbuild is going very slow at the moment. I am afraid that you will have work with the current build process. But I am willing to help. I hope to see your presentation on video later, due to personal budget restriction (dont we all have that) I cannot participate. Sorry to hear that, I would have liked to meet you. Well if you come to FOSDEM, we can have a long chat. My problem is that I am currently only a contributor so that ticket alone is 600,- EUR. Yeah, I know what you mean. And I will think about FOSDEM. I am also prepared for google/skype videochats. That is good to know. Maybe after ApacheCon. -Andre
Re: [question] build infra structure.
On 02.11.2012 10:53, Jürgen Schmidt wrote: On 11/2/12 9:46 AM, jan iversen wrote: Thanks for offering your help, I will definitively come back to that. Just one question, is there a design document or something where I can read how the new makefile concept is going to work ? maybe this helps you to get started http://wiki.openoffice.org/wiki/Build_Environment_Effort and all the related links on the right Juergen I found this one for the dmake system: http://wiki.openoffice.org/wiki/Dmake_and_OOo_environment Old but not completely outdated. -Andre
Re: [question] build infra structure.
On 31.10.2012 22:20, jan iversen wrote: Hi I have been searching for detailed internal information about how the build process works with build and dmake (gnumake). I have seen the relationship in the single directories (prj/build.lst prj/d.lst and makefile.mk), but I cannot find a central makefile. If I understand life, there should be a central makefile, telling e.g. how .cpp is translated to .o Pah, who needs a central makefile if he can have a Perl file instead :-) Sorry, I could not resist. I am currently preparing a talk for ApacheCon about the AOO build system and it is somewhat depressing to see how bizarre some things are. If I find the time after ApacheCon then I will turn my talk into a Wiki page or one or several blog posts. Here is the short version. First there is configure and bootstrap. But I think that you have mastered that step already. Then comes the actual building. The central makefile is main/solenv/bin/build.pl, yes, a Perl script. It reads module/prj/build.lst files to a) determine the dependency between modules and (just the first line) b) find the directories inside each module that have to be built. (all other lines) build.pl starts at main/instsetoo_native/prj/build.pl and follows the dependency to other modules. build.pl can handle multi process builds and uses the module dependency graph to build modules in the right order. It can do partial builds: build --all --from module ignores all modules before module when building AOO (in the linearization of the dependency graph) build --all called in another module than instsetoo_native builds all dependencies and stops when the current module is built. build.pl calls dmake for every module, regardless of whether they are dmake or gbuild modules. - For dmake modules it calls dmake for all directories listed in prj/build.lst - For gbuild modules it does the same but prj/build.lst only contains one entry which points to util/makefile.mk This util/makefile.mk then chains GNU make for module/Makefile gbuild modules have all their makefiles in their top level directory. One makefile per library or other main targets. Both dmake and gbuild distinguish between data and build logic. The modules usually contain only descriptions of which source files have to be compiled and which libraries are to be linked. How that is done, on all the different platforms, compilers, environment variables is handled by makefiles in solenv/incfor dmake solenv/gbuild for gbuild The last part of the build process is the creation of installation sets. It is triggered by instsetoo_native/util/makefile.mk which basically just calls solenv/bin/make_installer.pl with a cleverly selected bunch of parameters. make_installer.pl uses a larger number of Perl modules under solenv/bin/modules/installer which then do the actual work of collecting the relevant files, copying them into a temporary directory into a runnable office, and finally packing them into a package that fits the target platform. I am aware that the above is still very terse. I am happy to answer any questions (if I know the answer). Regards, Andre Can somebody please point me in the direction, or tell me if it done in a different way ? My reason for asking is that I need to add a set of new standard rules for localization (.xhlp - .po ) Thanks in advance. Jan
Re: [question] build infra structure.
See below please. THANKS for your VERY informative answer, it helped me a lot. I was of the simple idea, that we pursued a simple build process made up of gnuMake and an addon to gather for the shortcoming of gnumake in respect of cascaded makefiles. I hope to see your presentation on video later, due to personal budget restriction (dont we all have that) I cannot participate. Jan. On 1 November 2012 17:44, Andre Fischer awf@gmail.com wrote: On 31.10.2012 22:20, jan iversen wrote: Hi I have been searching for detailed internal information about how the build process works with build and dmake (gnumake). I have seen the relationship in the single directories (prj/build.lst prj/d.lst and makefile.mk), but I cannot find a central makefile. If I understand life, there should be a central makefile, telling e.g. how .cpp is translated to .o Pah, who needs a central makefile if he can have a Perl file instead :-) Sorry, I could not resist. I am currently preparing a talk for ApacheCon about the AOO build system and it is somewhat depressing to see how bizarre some things are. It´s quite OK, I learn fast :-) (and being a dane I like that kind of jokes/hints) If I find the time after ApacheCon then I will turn my talk into a Wiki page or one or several blog posts. Here is the short version. First there is configure and bootstrap. But I think that you have mastered that step already. Then comes the actual building. The central makefile is main/solenv/bin/ build.pl, yes, a Perl script. It reads module/prj/build.lst files to a) determine the dependency between modules and (just the first line) b) find the directories inside each module that have to be built. (all other lines) build.pl starts at main/instsetoo_native/prj/buil**d.pl http://build.pl and follows the dependency to other modules. build.pl can handle multi process builds and uses the module dependency graph to build modules in the right order. It can do partial builds: build --all --from module ignores all modules before module when building AOO (in the linearization of the dependency graph) build --all called in another module than instsetoo_native builds all dependencies and stops when the current module is built. build.pl calls dmake for every module, regardless of whether they are dmake or gbuild modules. - For dmake modules it calls dmake for all directories listed in prj/build.lst - For gbuild modules it does the same but prj/build.lst only contains one entry which points to util/makefile.mk This util/makefile.mk then chains GNU make for module/Makefile gbuild modules have all their makefiles in their top level directory. One makefile per library or other main targets. Why dont we just use dmake/gnumake, have a makefile in each directory which includes a master makefile ? Both dmake and gbuild distinguish between data and build logic. The modules usually contain only descriptions of which source files have to be compiled and which libraries are to be linked. How that is done, on all the different platforms, compilers, environment variables is handled by makefiles in solenv/incfor dmake solenv/gbuild for gbuild A I wrong in saying that the bulid list and delivery list could just as easily have been expressed as a target in makefile.in ??? Please forgive me, I am (as one who looks at the process with new eyes) just floating ideas ? The last part of the build process is the creation of installation sets. It is triggered by instsetoo_native/util/makefile**.mkhttp://makefile.mkwhich basically just calls solenv/bin/ make_installer.pl with a cleverly selected bunch of parameters. make_installer.pl uses a larger number of Perl modules under solenv/bin/modules/installer which then do the actual work of collecting the relevant files, copying them into a temporary directory into a runnable office, and finally packing them into a package that fits the target platform. I am aware that the above is still very terse. I am happy to answer any questions (if I know the answer). Thanks again, you actually helped me a lot Regards, Andre Can somebody please point me in the direction, or tell me if it done in a different way ? My reason for asking is that I need to add a set of new standard rules for localization (.xhlp - .po ) Thanks in advance. Jan
Re: [question] build infra structure.
On 01.11.2012 17:57, jan iversen wrote: See below please. THANKS for your VERY informative answer, it helped me a lot. I was of the simple idea, that we pursued a simple build process made up of gnuMake and an addon to gather for the shortcoming of gnumake in respect of cascaded makefiles. We are in the process of migrating from dmake to GNU make. When that is finished then we will have essentially one single makefile. Well, there will be one top level makefile that includes all the other makefiles. But there will not one make process that starts other makes in subprocesses. That would be evil, or so I have been told, see http://wiki.openoffice.org/wiki/Build_System_Analysis I hope to see your presentation on video later, due to personal budget restriction (dont we all have that) I cannot participate. Sorry to hear that, I would have liked to meet you. Jan. On 1 November 2012 17:44, Andre Fischer awf@gmail.com wrote: On 31.10.2012 22:20, jan iversen wrote: Hi I have been searching for detailed internal information about how the build process works with build and dmake (gnumake). I have seen the relationship in the single directories (prj/build.lst prj/d.lst and makefile.mk), but I cannot find a central makefile. If I understand life, there should be a central makefile, telling e.g. how .cpp is translated to .o Pah, who needs a central makefile if he can have a Perl file instead :-) Sorry, I could not resist. I am currently preparing a talk for ApacheCon about the AOO build system and it is somewhat depressing to see how bizarre some things are. It´s quite OK, I learn fast :-) (and being a dane I like that kind of jokes/hints) If I find the time after ApacheCon then I will turn my talk into a Wiki page or one or several blog posts. Here is the short version. First there is configure and bootstrap. But I think that you have mastered that step already. Then comes the actual building. The central makefile is main/solenv/bin/ build.pl, yes, a Perl script. It reads module/prj/build.lst files to a) determine the dependency between modules and (just the first line) b) find the directories inside each module that have to be built. (all other lines) build.pl starts at main/instsetoo_native/prj/buil**d.pl http://build.pl and follows the dependency to other modules. build.pl can handle multi process builds and uses the module dependency graph to build modules in the right order. It can do partial builds: build --all --from module ignores all modules before module when building AOO (in the linearization of the dependency graph) build --all called in another module than instsetoo_native builds all dependencies and stops when the current module is built. build.pl calls dmake for every module, regardless of whether they are dmake or gbuild modules. - For dmake modules it calls dmake for all directories listed in prj/build.lst - For gbuild modules it does the same but prj/build.lst only contains one entry which points to util/makefile.mk This util/makefile.mk then chains GNU make for module/Makefile gbuild modules have all their makefiles in their top level directory. One makefile per library or other main targets. Why dont we just use dmake/gnumake, have a makefile in each directory which includes a master makefile ? I guess there are historical reasons for that. And then there is the not-invented-here syndrome. I have made an experiment a few months ago in which I wrote a Perl script that reads all prj/build.lst files and creates one GNU makefile that did what build --all does. Worked like a charm. It just has not many advantages over build.pl. Especially when we proceed with the dmake to gbuild transition and will have the centeral makefile in a few months. Both dmake and gbuild distinguish between data and build logic. The modules usually contain only descriptions of which source files have to be compiled and which libraries are to be linked. How that is done, on all the different platforms, compilers, environment variables is handled by makefiles in solenv/incfor dmake solenv/gbuild for gbuild A I wrong in saying that the bulid list and delivery list could just as easily have been expressed as a target in makefile.in ??? Yes, certainly. But when you use a makefile for inter-module dependencies then you get a much finer granularity. That is one of the goals of gbuild. Change one file in VCL, run build -all in instsetoo_native and only the files that directly or indirectly depend on the modified files are built. Today you have to rebuild all modules completely that depend on VCL. Please forgive me, I am (as one who looks at the process with new eyes) just floating ideas ? That is, of course, a good thing. I am not so differently in that I did not know much about the build system when I worked at Sun and later Oracle. Our release engineers took care of it. It became necessary
Re: [question] build infra structure.
you See below please. On 1 November 2012 18:18, Andre Fischer awf@gmail.com wrote: On 01.11.2012 17:57, jan iversen wrote: See below please. THANKS for your VERY informative answer, it helped me a lot. I was of the simple idea, that we pursued a simple build process made up of gnuMake and an addon to gather for the shortcoming of gnumake in respect of cascaded makefiles. We are in the process of migrating from dmake to GNU make. When that is finished then we will have essentially one single makefile. Well, there will be one top level makefile that includes all the other makefiles. But there will not one make process that starts other makes in subprocesses. That would be evil, or so I have been told, see http://wiki.openoffice.org/**wiki/Build_System_Analysishttp://wiki.openoffice.org/wiki/Build_System_Analysis I am in the process of changing the l10n process. Currently it runs on one makefile that searches all directories, I want to change that to a target in every local makefile (build.lst). Can I attach myself to your progress, or would you suggest that I attach my development to the current build process. my timeline is somewhat around new year. I hope to see your presentation on video later, due to personal budget restriction (dont we all have that) I cannot participate. Sorry to hear that, I would have liked to meet you. Well if you come to FOSDEM, we can have a long chat. My problem is that I am currently only a contributor so that ticket alone is 600,- EUR. I am also prepared for google/skype videochats. Jan. On 1 November 2012 17:44, Andre Fischer awf@gmail.com wrote: On 31.10.2012 22:20, jan iversen wrote: Hi I have been searching for detailed internal information about how the build process works with build and dmake (gnumake). I have seen the relationship in the single directories (prj/build.lst prj/d.lst and makefile.mk), but I cannot find a central makefile. If I understand life, there should be a central makefile, telling e.g. how .cpp is translated to .o Pah, who needs a central makefile if he can have a Perl file instead :-) Sorry, I could not resist. I am currently preparing a talk for ApacheCon about the AOO build system and it is somewhat depressing to see how bizarre some things are. It´s quite OK, I learn fast :-) (and being a dane I like that kind of jokes/hints) If I find the time after ApacheCon then I will turn my talk into a Wiki page or one or several blog posts. Here is the short version. First there is configure and bootstrap. But I think that you have mastered that step already. Then comes the actual building. The central makefile is main/solenv/bin/ build.pl, yes, a Perl script. It reads module/prj/build.lst files to a) determine the dependency between modules and (just the first line) b) find the directories inside each module that have to be built. (all other lines) build.pl starts at main/instsetoo_native/prj/**buil**d.pl http://build.pl and follows the dependency to other modules. build.pl can handle multi process builds and uses the module dependency graph to build modules in the right order. It can do partial builds: build --all --from module ignores all modules before module when building AOO (in the linearization of the dependency graph) build --all called in another module than instsetoo_native builds all dependencies and stops when the current module is built. build.pl calls dmake for every module, regardless of whether they are dmake or gbuild modules. - For dmake modules it calls dmake for all directories listed in prj/build.lst - For gbuild modules it does the same but prj/build.lst only contains one entry which points to util/makefile.mk This util/makefile.mk then chains GNU make for module/Makefile gbuild modules have all their makefiles in their top level directory. One makefile per library or other main targets. Why dont we just use dmake/gnumake, have a makefile in each directory which includes a master makefile ? I guess there are historical reasons for that. And then there is the not-invented-here syndrome. I have made an experiment a few months ago in which I wrote a Perl script that reads all prj/build.lst files and creates one GNU makefile that did what build --all does. Worked like a charm. It just has not many advantages over build.pl. Especially when we proceed with the dmake to gbuild transition and will have the centeral makefile in a few months. Both dmake and gbuild distinguish between data and build logic. The modules usually contain only descriptions of which source files have to be compiled and which libraries are to be linked. How that is done, on all the different platforms, compilers, environment variables is handled by makefiles in solenv/incfor dmake solenv/gbuild for gbuild A I wrong in saying that the bulid list and delivery list
