Re: builddir vs. srcdir
Hello, On Fri, Mar 18, 2005 at 09:34:15PM +0200, Paul Pogonyshev wrote: Sorry, but I'll stick with my setup. My parser depends on my lowest- level library (`libutils') and almost the whole build-tree depends on the generated files. So, when I touched my `libutils' in any way, I would get a full-blown rebuild of the project. That's why I used `cmp' and only replaced the `.[ch]' files with the newly generated if they differed. And so I need timestamps to avoid unecessary commands run yes, this indeed sounds as a good reason for usiong stamps. Anyway, even if I don't accept your advice this time, thank you for the help. :) You are welcome. It was a nice excercise for me, too. Stepan
Re: builddir vs. srcdir
Hello, On Sat, Mar 12, 2005 at 06:56:20PM +0200, Paul Pogonyshev wrote: Everything seems to work just fine and as expected, your code, which is in whole cited below, doesn't actually know about the dependency foo.c foo.h: foo.list So if you change the .list file, the other files won't be rebuilt. This might be a dangerous bug. It doesn't know about the dependency, but the files will be rebuilt as a ``side-effect'' of the timestamp rule. I actually just tried this out. In general, stamps are always dangerous, so it's good to avoid them if possible. I'd try something like this: [file `aux/list.make'] SUFFIXES = .list .list.c: Makefile $(PARSE_LIST_COMMAND) $(PARSE_LIST_BUILD_RULE) .list.h: Makefile $(PARSE_LIST_COMMAND) $(PARSE_LIST_BUILD_RULE) PARSE_LIST_BUILD_RULE = \ $(PARSE_LIST_COMMAND) $(PARSE_LIST_FLAGS) \ `test -f '$' || echo '$(srcdir)/'`$ $*.h $*.c \ touch $*.h [usage] LIST_FILES = foo.list bar.list # Prevent problems with parallel make: foo.h: foo.c bar.h: bar.c PARSE_LIST_COMMAND = ... PARSE_LIST_FLAGS = ... noinst_LIBRARIES = libfoo.a libfoo_a_SOURCES = ... $(LIST_FILES) # This hint is needed only for included files; *.c files # are handled by normal target dependencies: BUILT_SOURCES = $(LIST_FILES:.list=.h) MOSTLYCLEANFILES = $(LIST_FILES:.list=.h) $(LIST_FILES:.list=.c) include $(top_srcdir)/aux/list.make Sorry, but I'll stick with my setup. My parser depends on my lowest- level library (`libutils') and almost the whole build-tree depends on the generated files. So, when I touched my `libutils' in any way, I would get a full-blown rebuild of the project. That's why I used `cmp' and only replaced the `.[ch]' files with the newly generated if they differed. And so I need timestamps to avoid unecessary commands run all the time (I want to see ``nothing to be done...'' when it is the case.) Let me add some comments: 1) The maintainer of Automake said that $(LIST_FILES:.list=.h) is safe, and I trust him. Oh, great, that is what I was looking for! 2) nodist_libfoo_a_SOURCES was redundant, especially the .c files. Automake knows how to transform .list to .o, and listing the intermediate files again could cause problems. With your setup, yes. Not with mine, unfortunately. 3) I try to have the BUILT_SOURCES hist as small as possible, so when the .c file is a prerequisite of the corresponding .o, there is no need to pre-build it using BUILT_SOURCES. 4) With parallel make, the PARSE_LIST_COMMAND cnnot be run twice in parallel, because of the foo.c: foo.h dependency. So we are safe. I need to add this rule in my setup too. Anyway, even if I don't accept your advice this time, thank you for the help. :) Paul
Re: builddir vs. srcdir
Hello, On Sat, Mar 12, 2005 at 06:56:20PM +0200, Paul Pogonyshev wrote: Everything seems to work just fine and as expected, your code, which is in whole cited below, doesn't actually know about the dependency foo.c foo.h: foo.list So if you change the .list file, the other files won't be rebuilt. This might be a dangerous bug. In general, stamps are always dangerous, so it's good to avoid them if possible. I'd try something like this: [file `aux/list.make'] SUFFIXES = .list .list.c: Makefile $(PARSE_LIST_COMMAND) $(PARSE_LIST_BUILD_RULE) .list.h: Makefile $(PARSE_LIST_COMMAND) $(PARSE_LIST_BUILD_RULE) PARSE_LIST_BUILD_RULE = \ $(PARSE_LIST_COMMAND) $(PARSE_LIST_FLAGS) \ `test -f '$' || echo '$(srcdir)/'`$ $*.h $*.c \ touch $*.h [usage] LIST_FILES = foo.list bar.list # Prevent problems with parallel make: foo.h: foo.c bar.h: bar.c PARSE_LIST_COMMAND = ... PARSE_LIST_FLAGS = ... noinst_LIBRARIES = libfoo.a libfoo_a_SOURCES = ... $(LIST_FILES) # This hint is needed only for included files; *.c files # are handled by normal target dependencies: BUILT_SOURCES = $(LIST_FILES:.list=.h) MOSTLYCLEANFILES = $(LIST_FILES:.list=.h) $(LIST_FILES:.list=.c) include $(top_srcdir)/aux/list.make Let me add some comments: 1) The maintainer of Automake said that $(LIST_FILES:.list=.h) is safe, and I trust him. 2) nodist_libfoo_a_SOURCES was redundant, especially the .c files. Automake knows how to transform .list to .o, and listing the intermediate files again could cause problems. 3) I try to have the BUILT_SOURCES hist as small as possible, so when the .c file is a prerequisite of the corresponding .o, there is no need to pre-build it using BUILT_SOURCES. 4) With parallel make, the PARSE_LIST_COMMAND cnnot be run twice in parallel, because of the foo.c: foo.h dependency. So we are safe. One more comment: 5) It looks that the PARSE_LIST_COMMAND will be executed twice: forst to create foo.c and then again to create foo.h. But this is actually not the case with GNU make: when command is to be called for the second time, make discovers, that the file foo.h has somehow appeared there, and that it's newer then its prerequisities, foo.list and foo.c (see the touch command in PARSE_LIST_BUILD_RULE), so there is no longer any need to rebuild it. Even if some inferior implementations of make were not that clever, the only problem would be that each pair of the files would be created twice. I think we can live with that. What if we used such an inferior make implementation for parallel build? Well, in general, this could bring problems. But I think this combination simply won't happen. Happy making, Stepan --- your code: [file `aux/list.make'] SUFFIXES = .list $(LIST_STAMP_FILES) : Makefile $(PARSE_LIST_COMMAND) # `PARSE_LIST_COMMAND' and `PARSE_LIST_FLAGS' should be set by the # includer. # PARSE_LIST = $(PARSE_LIST_COMMAND) $(PARSE_LIST_FLAGS) # We use `cmp' here to avoid unneeded recompilations of files that # depend on generated ones (only really useful for `.h' files.) # PARSE_LIST_BUILD_RULE = \ if $(PARSE_LIST) `test -f '$' || echo '$(srcdir)/'`$ \ $*.h.new $*.c.new; then\ if cmp -s $*.c.new $*.c; \ then rm -f $*.c.new; else mv -f $*.c.new $*.c; \ fi; \ if cmp -s $*.h.new $*.h; \ then rm -f $*.h.new; else mv -f $*.h.new $*.h; \ fi; \ echo timestamp $@; \ else\ (rm -f $*.c $*.c.new $*.h $*.h.new ; exit 1) \ fi .list.stamp: $(PARSE_LIST_BUILD_RULE) # Since $(LIST_GENERATED_FILES) defined by the includer don't (at # least shouldn't) have any dependencies, if this rule is being # executed, it probably means that one of the files was removed. # Then all we can do is to force rebuilding of corresponding stamp # file, which builds the required sources ``by side-effect.'' # $(LIST_GENERATED_FILES): rm -f $*.stamp; $(MAKE) $(AM_MAKEFLAGS) $*.stamp [usage] noinst_LIBRARIES = libfoo.a LIST_FILES = foo.list bar.list LIST_STAMP_FILES = foo.stamp bar.stamp LIST_GENERATED_FILES = foo.c bar.c foo.h bar.h PARSE_LIST_COMMAND = ... PARSE_LIST_FLAGS = ... include $(top_srcdir)/aux/list.make libfoo_a_SOURCES = ... $(LIST_FILES) nodist_libfoo_a_SOURCES = $(LIST_GENERATED_FILES) BUILT_SOURCES = $(LIST_STAMP_FILES) MOSTLYCLEANFILES = $(LIST_STAMP_FILES) $(LIST_GENERATED_FILES)
Re: builddir vs. srcdir
Stepan Kasal wrote: About the side topic of suffixes: [...] I finally decided that suffixes are good and ``are worth it'', mostly because I use generated files in 4 different directories (with two files in one of them), so rewriting the rule 5 times is kinda nasty. Based on your suggestions, study of GTK+ `Makefile.am' and own thoughts, I came up with this: [file `aux/list.make'] SUFFIXES = .list $(LIST_STAMP_FILES) : Makefile $(PARSE_LIST_COMMAND) # `PARSE_LIST_COMMAND' and `PARSE_LIST_FLAGS' should be set by the # includer. # PARSE_LIST = $(PARSE_LIST_COMMAND) $(PARSE_LIST_FLAGS) # We use `cmp' here to avoid unneeded recompilations of files that # depend on generated ones (only really useful for `.h' files.) # PARSE_LIST_BUILD_RULE = \ if $(PARSE_LIST) `test -f '$' || echo '$(srcdir)/'`$ \ $*.h.new $*.c.new; then\ if cmp -s $*.c.new $*.c; \ then rm -f $*.c.new; else mv -f $*.c.new $*.c; \ fi; \ if cmp -s $*.h.new $*.h; \ then rm -f $*.h.new; else mv -f $*.h.new $*.h; \ fi; \ echo timestamp $@; \ else\ (rm -f $*.c $*.c.new $*.h $*.h.new ; exit 1) \ fi .list.stamp: $(PARSE_LIST_BUILD_RULE) # Since $(LIST_GENERATED_FILES) defined by the includer don't (at # least shouldn't) have any dependencies, if this rule is being # executed, it probably means that one of the files was removed. # Then all we can do is to force rebuilding of corresponding stamp # file, which builds the required sources ``by side-effect.'' # $(LIST_GENERATED_FILES): rm -f $*.stamp; $(MAKE) $(AM_MAKEFLAGS) $*.stamp [usage] noinst_LIBRARIES = libfoo.a LIST_FILES = foo.list bar.list LIST_STAMP_FILES = foo.stamp bar.stamp LIST_GENERATED_FILES = foo.c bar.c foo.h bar.h PARSE_LIST_COMMAND = ... PARSE_LIST_FLAGS = ... include $(top_srcdir)/aux/list.make libfoo_a_SOURCES = ... $(LIST_FILES) nodist_libfoo_a_SOURCES = $(LIST_GENERATED_FILES) BUILT_SOURCES = $(LIST_STAMP_FILES) MOSTLYCLEANFILES = $(LIST_STAMP_FILES) $(LIST_GENERATED_FILES) Everything seems to work just fine and as expected, however I suspect that the ``rm -f $*.stamp; $(MAKE) $(AM_MAKEFLAGS) $*.stamp'' command will cause the problem with parallel make you mentioned. Is it so? Also, is there a way to automatically set `LIST_STAMP_FILES' and `LIST_GENERATED_FILES' from `LIST_FILES'? (I'm aware of GNU make's text manipulation functions, but they are GNU make specific...) Paul
Re: builddir vs. srcdir
Hello, On Thu, Mar 10, 2005 at 10:05:51PM +0200, Paul Pogonyshev wrote: I'm not sure which one comes first. [...] I can just do foo.c : foo.h foo.c foo.h : ... if $(BUILD_THEM_FILES) foo.list foo.h foo.c; then \ touch foo.c; \ else\ (rm -f foo.c foo.h; exit 1) \ fi right? Correct, of course. About the side topic of suffixes: Well, I also need something like `echo $* | sed 's/\.h$/.c/'` No, $* is the base without the suffix. in one directory I have two `.list' files, one of which is processed by my parser, while the other---by `glib-genmarshal' (I chose my suffix before I even started with GTK+ GUI.) One option is to change the suffix for your parser, of course. Or you could do something like: LIST_H_CMDS = if $(BUILD_THEM_FILES) $*.list $*.h $*.c; then\ touch $*.c; \ else\ (rm -f $*.c $*.h; exit 1) \ fi BUILD_THEM_FILES = case $* in \ *marshal*) $(GENMARSHAL_CMD) $*.list ... ;; \ *) $(PARSE) $*.list ... ;; \ esac .list.h: $(LIST_H_CMDS) .list.c: $(LIST_H_CMDS) All the variables in the commands, are expanded just before execution, so the usage of $* ``outside of a rule'' is correct here. But when I recall that all this is done just to be able to do xx_SOURCES = foo.list ... intead of xx_SOURCES = ... nodist_xx_SOURCES = foo.c EXTRA_DIST = foo.list then I agree it's not worth it. (Setting CLEANFILES = foo.c foo.h; BUILT_SOURCES = foo.h is necessary in both cases.) Have a nice day, Stepan
Re: builddir vs. srcdir
Stepan Kasal wrote: Hello, On Tue, Mar 08, 2005 at 11:56:56PM +0200, Paul Pogonyshev wrote: foo.c foo.h : $(srcdir)/foo.list $(PARSE_LIST) $(PARSE_LIST) $(srcdir)/foo.list foo.h foo.c\ || (rm -f foo.c foo.h ; exit 1) This rule can break with parallel make. You can solve the second issue by adding the dependency: foo.c: foo.h Whats about a more general case SOMEFILES=a b c d e $(SOMEFILES): srcfile buildsomehow srcfile $(SOMEFILES) || (rm -f $(SOMEFILES); false) ??? Regards Harri
Re: builddir vs. srcdir
* Harald Dunkel wrote on Thu, Mar 10, 2005 at 09:27:34AM CET: Stepan Kasal wrote: On Tue, Mar 08, 2005 at 11:56:56PM +0200, Paul Pogonyshev wrote: foo.c foo.h : $(srcdir)/foo.list $(PARSE_LIST) $(PARSE_LIST) $(srcdir)/foo.list foo.h foo.c\ || (rm -f foo.c foo.h ; exit 1) This rule can break with parallel make. You can solve the second issue by adding the dependency: foo.c: foo.h Whats about a more general case SOMEFILES=a b c d e $(SOMEFILES): srcfile buildsomehow srcfile $(SOMEFILES) || (rm -f $(SOMEFILES); false) Please read info '(automake.info)Multiple Outputs' which comes with a recent Automake and documents the whole issue very thoroughly. Regards, Ralf
Re: builddir vs. srcdir
Hi, On Wed, Mar 09, 2005 at 11:21:35PM +0200, Paul Pogonyshev wrote: And I'd like to suggest that you use SUFFIXES to handle the .list source. Please look at the following example: Well, my generator is even more non-standard, since I need to pass an additional command-line parameter sometimes. So, `SUFFIXES' are not an option, although I agree it would have been nicer to use them. well, in the example I cited, I had several .list sources, but only one in each directory. Thus I was able to have different rules in different directories. This is not usable for you, but you can make the .list.h rule general: $(PARSE) `case $* in *this.list) echo this options;; \ *that.list) echo that opt;; *) echo default opt;;esac` $ ($* is substituted by make) Or you can place the options at the first line of the .list source and grep for them. And if your generator creates *.c file after *.h file, you should define the dependency: foo.c: foo.h and vice versa if the generator creates .h after .c. That will make make happy and you won't observe repeated re-generation of the .c file. HTH, Stepan Kasal
Re: builddir vs. srcdir
Harald == Harald Dunkel [EMAIL PROTECTED] writes: Harald Whats about a more general case http://sources.redhat.com/automake/automake.html#Multiple-Outputs -- Alexandre Duret-Lutz
Re: builddir vs. srcdir
Alexandre Duret-Lutz wrote: Paul == Paul Pogonyshev [EMAIL PROTECTED] writes: Paul Stepan Kasal wrote: Hello, On Tue, Mar 08, 2005 at 11:56:56PM +0200, Paul Pogonyshev wrote: because the generated sources are placed into the build directory, while `make' looks for them in the source directory. generally, make should look for them in both places. Paul Yes, and it does look in both places, but the dependencies make `foo.o' Paul depend on `$(srcdir)/foo.c', This could happen if this location was the right one in the past, and building foo.c in the build directory is a recent change. In that case the dependency file still have the old info. The way to erase dependency info is distclean: make -k distclean; ./configure; make Yup, you are right, thanks. Should have guessed myself... :[ Paul
Re: builddir vs. srcdir
Hi, On Wed, Mar 09, 2005 at 11:21:35PM +0200, Paul Pogonyshev wrote: And I'd like to suggest that you use SUFFIXES to handle the .list source. Please look at the following example: Well, my generator is even more non-standard, since I need to pass an additional command-line parameter sometimes. So, `SUFFIXES' are not an option, although I agree it would have been nicer to use them. well, in the example I cited, I had several .list sources, but only one in each directory. Thus I was able to have different rules in different directories. This is not usable for you, but you can make the .list.h rule general: $(PARSE) `case $* in *this.list) echo this options;; \ *that.list) echo that opt;; *) echo default opt;;esac` $ ($* is substituted by make) Well, I also need something like `echo $* | sed 's/\.h$/.c/'` (not tested), so using suffix rules seems like too much trouble. Besides, in one directory I have two `.list' files, one of which is processed by my parser, while the other---by `glib-genmarshal' (I chose my suffix before I even started with GTK+ GUI.) Or you can place the options at the first line of the .list source and grep for them. Then I could just make the parser read them as well. That might be a good idea, actually, since these options are more like modes that are absolutely necessary to be set correctly to parse anything. And if your generator creates *.c file after *.h file, you should define the dependency: foo.c: foo.h and vice versa if the generator creates .h after .c. That will make make happy and you won't observe repeated re-generation of the .c file. I'm not sure which one comes first. They are generated on-the-fly in parallel (i.e. fprintf's to both files are mixed.) I assume in this case I can just do foo.c : foo.h foo.c foo.h : ... if $(BUILD_THEM_FILES) foo.list foo.h foo.c; then \ touch foo.c; \ else\ (rm -f foo.c foo.h; exit 1) \ fi right? BTW, I have just built everything in a separate directory for the first time :) I need to sort out that parallel building stuff, but otherwise all works perfectly. Thanks for your help. Paul
builddir vs. srcdir
Hi. I'm currently massaging my `Makefile.am's to allow building in a separate directory. However, I have a problem which I cannot find how to solve. I have a few source (`.c' and `.h') files which are generated at build time from another source using a custom utility. When the build directory is the same as the source directory, everything is fine. However, with separate build directory things go hairy, because the generated sources are placed into the build directory, while `make' looks for them in the source directory. Automake pseudo-code looks like this: noinst_LIBRARIES = libfoo.a BUILT_SOURCES = \ foo.c \ \ foo.h PARSE_LIST = ./parse-list$(EXEEXT) foo.c foo.h : $(srcdir)/foo.list $(PARSE_LIST) $(PARSE_LIST) $(srcdir)/foo.list foo.h foo.c\ || (rm -f foo.c foo.h ; exit 1) libfoo_a_SOURCES = ... nodist_libfoo_a_SOURCES = $(BUILT_SOURCES) plus the lines to build the parser utility and other non-important stuff. Paul
Re: builddir vs. srcdir
Hello, On Tue, Mar 08, 2005 at 11:56:56PM +0200, Paul Pogonyshev wrote: because the generated sources are placed into the build directory, while `make' looks for them in the source directory. generally, make should look for them in both places. Let me point out several problems: BUILT_SOURCES = \ foo.c \ foo.h You probably need only the foo.h file here. foo.c foo.h : $(srcdir)/foo.list $(PARSE_LIST) $(PARSE_LIST) $(srcdir)/foo.list foo.h foo.c\ || (rm -f foo.c foo.h ; exit 1) This rule can break with parallel make. For details about these two issues, see http://sources.redhat.com/automake/automake.html#Built-sources-example You can solve the second issue by adding the dependency: foo.c: foo.h Yet it might be more readable if you change your generator so that it would generate .c and .h in separate runs. And I'd like to suggest that you use SUFFIXES to handle the .list source. Please look at the following example: noinst_LTLIBRARIES = libgoffice-utils.la libgoffice_utils_la_SOURCES = \ go-marshalers.list \ go-font.c CLEANFILES =\ go-marshalers.h \ go-marshalers.c # A hint is needed to build the header first: BUILT_SOURCES = go-marshalers.h GENMARSHAL_COMMAND = $(GLIB_GENMARSHAL) --prefix=go_ SUFFIXES = .list .list.h: $(GLIB_GENMARSHAL) $(GENMARSHAL_COMMAND) --header $ $@ .list.c: $(GLIB_GENMARSHAL) (echo '/* This file has been automatically generated. Do not edit. */' \ echo '#include $*.h' \ $(GENMARSHAL_COMMAND) --body $ ) $@ This works with current Automake 1.9.5 (which I recommend), as well as an ancient 1.7.x version. HTH, Stepan Kasal
Re: builddir vs. srcdir
Stepan Kasal wrote: Hello, On Tue, Mar 08, 2005 at 11:56:56PM +0200, Paul Pogonyshev wrote: because the generated sources are placed into the build directory, while `make' looks for them in the source directory. generally, make should look for them in both places. Yes, and it does look in both places, but the dependencies make `foo.o' depend on `$(srcdir)/foo.c', so `make' tells it has no rule to build the latter. How do I make dependencies tell `foo.o' depends on `$(builddir)/foo.c' instead? Let me point out several problems: BUILT_SOURCES = \ foo.c \ foo.h You probably need only the foo.h file here. Right. foo.c foo.h : $(srcdir)/foo.list $(PARSE_LIST) $(PARSE_LIST) $(srcdir)/foo.list foo.h foo.c\ || (rm -f foo.c foo.h ; exit 1) This rule can break with parallel make. For details about these two issues, see http://sources.redhat.com/automake/automake.html#Built-sources-example You can solve the second issue by adding the dependency: foo.c: foo.h Yet it might be more readable if you change your generator so that it would generate .c and .h in separate runs. I'd hate to hack the generator for such a goal. I'll better stick with the `foo.c: foo.h' solution. Maybe generating two files at once is non- standard, but it seems natural, since they are so closely related and are built from one source file, and that's the way it works already. And I'd like to suggest that you use SUFFIXES to handle the .list source. Please look at the following example: Well, my generator is even more non-standard, since I need to pass an additional command-line parameter sometimes. So, `SUFFIXES' are not an option, although I agree it would have been nicer to use them. Paul
Re: builddir vs. srcdir
Paul == Paul Pogonyshev [EMAIL PROTECTED] writes: Paul Stepan Kasal wrote: Hello, On Tue, Mar 08, 2005 at 11:56:56PM +0200, Paul Pogonyshev wrote: because the generated sources are placed into the build directory, while `make' looks for them in the source directory. generally, make should look for them in both places. Paul Yes, and it does look in both places, but the dependencies make `foo.o' Paul depend on `$(srcdir)/foo.c', This could happen if this location was the right one in the past, and building foo.c in the build directory is a recent change. In that case the dependency file still have the old info. The way to erase dependency info is distclean: make -k distclean; ./configure; make -- Alexandre Duret-Lutz
