Re: Expressing dependencies

[Laurence Finston]

On Mon, 5 Jan 2004, Robert Collins wrote:

> On Mon, 2004-01-05 at 03:53, Laurence Finston wrote:
>
> > This is essentially what I tried to do by using the auxiliary program
> > `3DLDFcpl' in the rule for building the executable `3dldf' (roughly):
> >
> > 3dldf: $(3DLDF_CWEBS)
> >3DLDFcpl
>
> Thats not quite what I was suggesting.
>

It would have worked quite nicely, though, except for the rules for the .o
files that Automake generated automatically. If I remember correctly, it
also affected the `dist' target and the various `clean' targets.


> > It's reasonable behavior for Automake to assume that the sources for a C++
> > program are called .cc, .cxx, or .c++, but it is
> > very restrictive to assume files need to be rebuilt based merely on the
> > information that their prerequisites have a more recent timestamp.
>
> As I said, thats the /only/ information (cheaply) available to automake.
> Alternatives include generating a md5 of the file and comparing that to
> a calculated one, or other such has-content-changed tests. But make
> isn't language aware - it can't tell if a file change is 'meaningful'.
>

I don't know what an md5 is.

> > I don't think the problem lies with CWEB, or Bison and Flex, for that matter.
> > CWEB isn't GNU software, and I doubt whether the authors would appreciate
> > being asked to "fix" it. We can't do anything about make, either. I think the
> > problem should be solved within Automake.
>
> The problem lies in the concept of the header 'maybe changing'. If the
> header is dependent on the .web source, then it can't be considered
> correct if it's timestamp is older than the .web source - because make
> is meant to look at just modification dates. If you want to consider
> file content changes, make has to generate some database (of sorts) to
> track file content, and while we *could* do that in GNU Make, automake
> targets posix make - so that would not be portable. So the rules that
> will most likely work well for you are - no target to build the .h file
> at all, and magic in a script to replace it if needed. Note that if the
> header is replaced, you'll need to reinvoke make on the same dir, to get
> it to notice that reliably.
>
> Rob
>

There is a mechanism for
regenerating a header file, which is needed for the case that it doesn't
exist.

I have solved the immediate problem for 3DLDF.
That's not what I'm getting at. In my opinion, it would be worthwhile
to try to implement a general solution for this problem, which
affects users of other development tools, too. I believe that the solution
should be implemented within Automake.

These are the problems that remain:

1. If a .web file changes, but these changes affect neither the .cxx file
nor the .h file, it must be touched in such a way that its timestamp
is the same as the oldest of the two associated timer files. This
makes it necessary to revert the buffer, if the .web file is currently
being edited. This is harmless, but annoying. If it's not touched,
make will run ctangle on it the next time around.

2. I wasn't able to rewrite the rules in order to make it possible for
a single .web file to write multiple output files that are used in the
rules for building the executable `3dldf'. Maybe it's possible, but it
was just too complicated. This means that the GNU build tools are
limiting my ability to use the features of CWEB.

3. It was fairly difficult for me to get the rules to work right. I
think they do now, but problems may still crop up. Solving it for
3DLDF helped me, but it doesn't help anyone else.

I submitted my package to the GNU project after it was "finished", and
then I had to go back and learn how to use the GNU build tools for it. I
could have saved myself a lot of trouble by using them in the first
place. I suspect a lot of people will make the same mistake.

If I was writing low-level system software, I would just write in C
without using CWEB, and I wouldn't have these problems with
make. However, 3DLDF isn't that kind of software, and using CWEB has
significant advantages. I think authors of GNU packages should be
encouraged to use literate programming tools rather than
discouraged. If Automake doesn't provide support for the tools
people are using, of whatever kind, I think this will tend to
discourage them from submitting their packages to the GNU project.

This isn't just a theoretical problem for me. I'm considering
implementing interfaces to packages that supply 3D-modelling and
rendering functionality via libraries. I have to be careful about
this, because of the copyright and licensing issues involved. If
possible, I'd prefer to use GNU packages, but many packages are not
GNU. I'm not saying that lack of native support for CWEB in Automake
is the reason, I'm sure it's not. However, I think it would be
worthwhile trying to make it easier for people to conform to the GNU
Coding Standards while still using the development tools they like. I
am not suggesting changing the standards.

Re: Expressing dependencies

[Robert Collins]

On Mon, 2004-01-05 at 03:53, Laurence Finston wrote:

> This is essentially what I tried to do by using the auxiliary program
> `3DLDFcpl' in the rule for building the executable `3dldf' (roughly):
> 
> 3dldf: $(3DLDF_CWEBS)
>3DLDFcpl

Thats not quite what I was suggesting.

> Not changing the timestamp of the .cxx file isn't enough. In fact, I change
> the name of the ctangle output from .c to .cxx based on
> whether the file has changed since the last version, and use the .cxx file in
> the rule for building the object files. This preserves the timestamp of the
> .cxx file, if the .c file doesn't contain any significant changes.
> 
> I actually had a problem with Automake assuming that 3DLDF was a C program
> rather than a C++ program because of the .c extension. Comparing the .c files
> to the .cxx files, renaming them, if appropriate, and using the .cxx files in
> the rules for building the .o files solves this problem. I think there's at
> least one other way of solving it, perhaps by means of an Autoconf variable,
> but I don't remember off-hand. 

Right, thats orthogonal though: if we 3dldf.o is built from 3dldf.cxx
and 3dldf.h, and 3dldf.cxx and 3dldf.h are built from wdlfd.web; then we
can focus on the dependency issue - not the actual extensions.

> It's reasonable behavior for Automake to assume that the sources for a C++
> program are called .cc, .cxx, or .c++, but it is
> very restrictive to assume files need to be rebuilt based merely on the
> information that their prerequisites have a more recent timestamp. 

As I said, thats the /only/ information (cheaply) available to automake.
Alternatives include generating a md5 of the file and comparing that to
a calculated one, or other such has-content-changed tests. But make
isn't language aware - it can't tell if a file change is 'meaningful'.  

> As Andrew
> Suffield pointed out in his posting, this problem affects Bison and Flex, too,
> which are probably used much more often for GNU software than CWEB.  I suspect
> there are other tools affected by this problem as well.

Yes - and the same problem applies - make assumes that the commands it
runs are for a single purpose - with no side effects.

> I don't think the problem lies with CWEB, or Bison and Flex, for that matter. 
> CWEB isn't GNU software, and I doubt whether the authors would appreciate
> being asked to "fix" it. We can't do anything about make, either. I think the
> problem should be solved within Automake. 

The problem lies in the concept of the header 'maybe changing'. If the
header is dependent on the .web source, then it can't be considered
correct if it's timestamp is older than the .web source - because make
is meant to look at just modification dates. If you want to consider
file content changes, make has to generate some database (of sorts) to
track file content, and while we *could* do that in GNU Make, automake
targets posix make - so that would not be portable. So the rules that
will most likely work well for you are - no target to build the .h file
at all, and magic in a script to replace it if needed. Note that if the
header is replaced, you'll need to reinvoke make on the same dir, to get
it to notice that reliably.

Rob


-- 
GPG key available at: .


signature.asc
Description: This is a digitally signed message part

Re: Expressing dependencies

[Laurence Finston]

Robert Collins wrote:
---
> On Sun, 2004-01-04 at 08:17, Laurence Finston wrote:
> 
> > The problem is that make makes certain assumptions that don't apply when
CWEB
> > is used.
> 
> I think thats an incorrect statement. It would be more accurate to say
> that CWEB hasn't been built with any thought to the impact on make. Make
> has only the file system data available to it to determine 'has X
> changed more recently than Y.' config.status for example, when it
> regenerates config.h will only alter the file if the contents have
> changed - so that it preserves the timestamp. I think that most
> pre-preocessors in this sense could benefit from a wrapper of some sort
> that would equally not alter the file IF nothing had changed - and you
> could use that wrapper directly in make rules for ctangle.
> 
> Rob
> 

This is essentially what I tried to do by using the auxiliary program
`3DLDFcpl' in the rule for building the executable `3dldf' (roughly):

3dldf: $(3DLDF_CWEBS)
   3DLDFcpl

However, this didn't work, because some of the default rules automatically
generated by Automake expected rules involving the C++ files generated by
ctangle; ones for building the object files, I believe. 

Not changing the timestamp of the .cxx file isn't enough. In fact, I change
the name of the ctangle output from .c to .cxx based on
whether the file has changed since the last version, and use the .cxx file in
the rule for building the object files. This preserves the timestamp of the
.cxx file, if the .c file doesn't contain any significant changes.

I actually had a problem with Automake assuming that 3DLDF was a C program
rather than a C++ program because of the .c extension. Comparing the .c files
to the .cxx files, renaming them, if appropriate, and using the .cxx files in
the rules for building the .o files solves this problem. I think there's at
least one other way of solving it, perhaps by means of an Autoconf variable,
but I don't remember off-hand. 

It's reasonable behavior for Automake to assume that the sources for a C++
program are called .cc, .cxx, or .c++, but it is
very restrictive to assume files need to be rebuilt based merely on the
information that their prerequisites have a more recent timestamp. As Andrew
Suffield pointed out in his posting, this problem affects Bison and Flex, too,
which are probably used much more often for GNU software than CWEB.  I suspect
there are other tools affected by this problem as well.

I plan to start working on implementing an input routine using Bison and Flex.
Once 3DLDF has one, the executable won't have to be built over and over again.
However, this will continue to be necessary for users for a couple of months,
at least, and it will always be necessary for developers. I don't know yet how
I'm going to solve the problem of rewriting the build rules in Makefile.am to
account for the Flex and Bison files.

The original pre-GNU distribution of 3DLDF didn't use Autoconf and Automake
and I was able to solve the problem of expressing the dependencies among the
files. It's not even a difficult problem to solve, when you don't have to use
make. RMS wanted 3DLDF to have a configure script and Makefile.in files with
all of the required targets and I thought that was reasonable, and even
desirable. However, the only practical way to do this, in my opinion, is to
use Autoconf and Automake, which are great in other ways. However, I think the
problem of expressing dependencies is a serious one.

I don't think the problem lies with CWEB, or Bison and Flex, for that matter. 
CWEB isn't GNU software, and I doubt whether the authors would appreciate
being asked to "fix" it. We can't do anything about make, either. I think the
problem should be solved within Automake. 

I'm not just complaining, I'd be quite willing to contribute to a solution. I
haven't even looked at the Automake sources yet, so I don't know whether I
could help by doing any programming. However, I do at least have a project
that could be used for testing any facilities the Automake developers program.
This assumes that other people also think that it's worthwhile to try to solve
this problem. 

Laurence Finston
3DLDF maintainer
Website: http://wwwuser.gwdg.de/~lfinsto1
email: [EMAIL PROTECTED]

Re: Expressing dependencies

[Bruce Korb]

Laurence Finston wrote:

> The problem is that make makes certain assumptions that don't apply when CWEB
> is used. `ctangle .web' creates .c. Additional files can
> also be written. In 3DLDF, each .web also writes .h.
> [...] However, not all changes
> to a .web file cause the .c or .h file to differ from its previous version,
> while make assumes that a target file must be rebuilt if one of its
> prerequisites is newer.

This is the general problem where there is not a strict 1:1 mapping between
a primary source and the one and only build product.  Consider your .web file
to really be several source files tangled up into one source file (with some
pieces of it common to several derived files).  Make has no conceivable way
of disentangling effects without "ctangle" itself.  So, the only way to make
it work is to run ctangle, determine if the output has changed and, if so,
replace the product file(s) and touch a timestamp file.  Not especially hard:

  --stamp : 
 
   && touch $@
 for  ; do \
if  ; \
then rm  ; \
else mv   ; fi ; done

Of course, make itself does the out-of-date analysis before running any
commands.  Not much automake can do about that.  Perhaps a layered
approach where a special target first ensures that derived source files
are up to date, then it re-invokes make to ensure that final products
are all up to date.  Yummy.  That's not an automake issue, tho.

Re: Expressing dependencies

[Richard Dawe]

Hello.

Andrew Suffield wrote:
[snip]
It may be worth noting that bison and flex, as invoked by automake,
have comparable issues with their ancillary files (especially
bison-generated headers). I never managed to figure a way to have
those work sanely with make either, let alone automake.
From the perspective of automake, it gets even worse, if you consider 
what bison (and maybe flex) do to filenames on DOS. There's some hideous 
mangling of the filenames to fit into the 8+3 short filename convention, 
if only short filenames are available (Windows '95 and later support 
long filenames).

   foo.y -> foo.tab.c (Linux, Unices & DOS with LFNs)
   foo.y -> foo_tab.c (DOS with SFNs)
The nasty part of this is that bison does this mangling without being 
told to. The only sane solution I came up with was to always tell it the 
file prefixes of the output files. But that breaks automake-generated 
Makefiles, IIRC.

Bye, Rich =]

--
Richard Dawe [ http://homepages.nildram.co.uk/~phekda/richdawe/ ]
"You can't evaluate a man by logic alone."
  -- McCoy, "I, Mudd", Star Trek

Re: Expressing dependencies

[Andrew Suffield]

On Sun, Jan 04, 2004 at 09:39:28AM +1100, Robert Collins wrote:
> On Sun, 2004-01-04 at 08:17, Laurence Finston wrote:
> 
> > The problem is that make makes certain assumptions that don't apply when CWEB
> > is used.
> 
> I think thats an incorrect statement. It would be more accurate to say
> that CWEB hasn't been built with any thought to the impact on make. Make
> has only the file system data available to it to determine 'has X
> changed more recently than Y.' config.status for example, when it
> regenerates config.h will only alter the file if the contents have
> changed - so that it preserves the timestamp. I think that most
> pre-preocessors in this sense could benefit from a wrapper of some sort
> that would equally not alter the file IF nothing had changed - and you
> could use that wrapper directly in make rules for ctangle.

It may be worth noting that bison and flex, as invoked by automake,
have comparable issues with their ancillary files (especially
bison-generated headers). I never managed to figure a way to have
those work sanely with make either, let alone automake.

-- 
  .''`.  ** Debian GNU/Linux ** | Andrew Suffield
 : :' :  http://www.debian.org/ |
 `. `'  |
   `- -><-  |


signature.asc
Description: Digital signature

Re: Expressing dependencies

[Robert Collins]

On Sun, 2004-01-04 at 08:17, Laurence Finston wrote:

> The problem is that make makes certain assumptions that don't apply when CWEB
> is used.

I think thats an incorrect statement. It would be more accurate to say
that CWEB hasn't been built with any thought to the impact on make. Make
has only the file system data available to it to determine 'has X
changed more recently than Y.' config.status for example, when it
regenerates config.h will only alter the file if the contents have
changed - so that it preserves the timestamp. I think that most
pre-preocessors in this sense could benefit from a wrapper of some sort
that would equally not alter the file IF nothing had changed - and you
could use that wrapper directly in make rules for ctangle.

Rob

-- 
GPG key available at: .


signature.asc
Description: This is a digitally signed message part