Fix distcc/ccache interferences with the test suite
Hi all, From 02a3220b061f232d30baf1d1d3bc497d7e9417b8 Mon Sep 17 00:00:00 2001 From: Akim Demaille demai...@gostai.com Date: Mon, 5 Jan 2009 15:10:55 +0100 Subject: [PATCH] Fix distcc/ccache interferences with the test suite. * tests/localization.at (localized compiler messages): Ignore distcc log messages. --- tests/localization.at | 18 +- 1 files changed, 17 insertions(+), 1 deletions(-) diff --git a/tests/localization.at b/tests/localization.at index d489fef..a338bda 100644 --- a/tests/localization.at +++ b/tests/localization.at @@ -1,6 +1,6 @@ # localization.at -- libtool and locales-*- Autotest -*- # -# Copyright (C) 2008 Free Software Foundation, Inc. +# Copyright (C) 2008, 2009 Free Software Foundation, Inc. # Written by Ralf Wildenhues, 2008 # # This file is part of GNU Libtool. @@ -47,6 +47,22 @@ mv -f stdout expected-stdout mv -f stderr expected-stderr AT_CHECK([$LIBTOOL --mode=compile --tag=CC $CC $CPPFLAGS $CFLAGS -c a.c || exit 1], [1], [stdout], [stderr]) + +# Get rid of useless differences, such as distcc process numbers +# and ccache file location. +# +# distcc[54517] ERROR: compile /Users/akim/.ccache/a.tmp.montero.gostai.ensta.fr.54506.i on localhost failed +# a.c:1: error: size of array 'x' is negative +# distcc[54506] ERROR: compile a.c on localhost failed +# +# Just ignore the whole line which only wraps the genuine compiler +# error message. +for f in expected-stdout expected-stderr stdout stderr +do + sed -e '/^distcc\[[0-9]*\]/d' $f $f.tmp + mv -f $f.tmp $f +done + AT_CHECK([diff expected-stderr stderr]) LTBASE=`$ECHO $LIBTOOL | sed 's,^.*/,,'` AT_CHECK([grep -v ^$LTBASE: compile stdout | diff expected-stdout -]) -- 1.6.0.4.790.gaa14a
Re: [PATCH] [cygwin|mingw] fix dlpreopen with --disable-static take 2
Peter Rosin wrote:
Den 2009-01-05 06:24 skrev Charles Wilson:
Interesting! Meanwhile, I have done some experiments on my own, as I
don't like the dependence on anything that comes with MinGW when
dealing with libtool and MSVC.
I kind of suspected that. What about the attached? This version needs
to link against libbfd and its dependencies -- so has to be compiled
using mingw gcc. But if this executable was included as part of msys?
(Speaking of dependencies, I don't think the current MinGW code
in libtool requires 'file' to be present, and I don't think it is
part of a minimal MSYS install. It's not in my install anyway.)
Only because its been over two years since msys 1.11 will be ready
RSN. It is intended that the file package be included in the minimal
1.11 install.
I have found that for MSVC import libraries the simplest thing is
to list the archive members to get to the dll name. I have tried
with:
lib -nologo -list $implib | grep -v '\.obj$' | sort -u
or, in gnu speak:
ar t $implib | grep -v '\.obj$' | sort -u
I noticed that, but wasn't sure if self-compiled (using MSVC) import
libraries were the same.
This works for all troublesome implibs that you have listed above
(at least those that I have easy access to, but I have at least one
from each class of problems) and a few others. Except for MAPI.lib,
but my MS provided dumpbin.exe (VS 2005) says
MAPI.lib : warning LNK4048: Invalid format file; ignored
for that one so I too think it is a pathological case.
Ack.
Also, it will not fail for Vfw32.Lib, it will instead list the three
dlls it imports (AVICAP32.dll, AVIFIL32.dll, MSVFW32.dll).
Well, we probably want it to fail. dlpreopen is supposed to work like
dlopen -- and you'd need to dlopen each of the three DLLs separately.
But if you -dlpreopen Vfw32.Lib, you'd need to determine which symbols
IN Vfw32.Lib came from which DLL, and generate three different groups in
you LT_LTX_*[] structure, to register those symbols with their effective
dlopen source:
{ AVICAP32.dll, 0 }
{ symbols }
{ AVIFIL32.dll, 0 }
{ symbols }
{ MSVFW32.dll, 0 }
{ symbols }
I don't think this is going to work well.
Using MS tools (instead of file or objdump -f) to identify if a .lib
is an import lib or a static lib seems to be trickier. One thing
that appears to work is to look for an __IMPORT_DESCRIPTOR_* symbol,
but that can obviously be thwarted by a devious (or ignorant) user...
That's ok. Rule #486: don't deliberately try to undermine your tools,
and then expect them to work.
BTW, those symbols also identifies the imported dll (but that breaks
when that which is imported isn't named foo.dll). E.g.
dumpbin -symbols $lib | grep '| __IMPORT_DESCRIPTOR_'
(output for Vfw32.Lib
001 SECT2 notype External |
__IMPORT_DESCRIPTOR_MSVFW32
001 SECT2 notype External |
__IMPORT_DESCRIPTOR_AVIFIL32
001 SECT2 notype External |
__IMPORT_DESCRIPTOR_AVICAP32
)
or, in gnu speak:
nm $lib | grep 'I __IMPORT_DESCRIPTOR_'
(output for Vfw32.Lib
I __IMPORT_DESCRIPTOR_MSVFW32
I __IMPORT_DESCRIPTOR_AVIFIL32
I __IMPORT_DESCRIPTOR_AVICAP32
)
Yeah, I wanted to avoid assuming that non-libtool shared libraries
always in in *.dll, because many packages (especially ones that do
dlopen/dlpreopen) still name their modules foo.so even on
cygwin/mingw. Take ImageMagick, for instance.
But...I also dislike for fixes to existing bugs, in existing platforms,
to be held up by not-yet-in-master support for other compilers. So, can
we get back to discussing the original patch, under the predicates of
cygwin and mingw (not msvc) $hosts?
Hey, I'm not opposed to the patch, I didn't intend to make that impression
either, sorry if I did. I'm just trying to determine what needs to be
done in the MSVC branch to keep up. Just poking and asking questions,
so thank you very much for your valuable input!
Oh, ok. Thanks.
--
Chuck
/* dllname.c -- tool to extract the name of the DLL associated with
an import library.
Copyright 2008 Charles S. Wilson
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 51 Franklin Street - Fifth Floor, Boston, MA
02110-1301, USA. */
#include stdio.h
#include stdarg.h
#include bfd.h
#include ansidecl.h
#include unistd.h
Re: [PATCH] [cygwin|mingw] fix dlpreopen with --disable-static take 2
Den 2009-01-05 06:24 skrev Charles Wilson:
Charles Wilson wrote:
Charles Wilson wrote:
Peter Rosin wrote:
I'm primarily trying to determine what impact this has on my
MSVC branch...
Ran some experiments on the libraries shipped with the Windows SDK. The
attached script worked ok on most of them. After eliminating the static
libraries and the import libraries to '*.dll' and '*.DLL' (241
successes in all), I'm left with the following 13 odd ducks:
These are correct, but are a reminder that import libraries exist for
objects other than those named foo.dll. So that's 5 more successes:
KSProxy.Lib :x86 archive import FOR ksproxy.ax
bthprops.lib :x86 archive import FOR bthprops.cpl
irprops.lib :x86 archive import FOR irprops.cpl
NetSh.Lib :x86 archive import FOR NETSH.EXE
WinSpool.Lib :x86 archive import FOR WINSPOOL.DRV
These are the true failures:
For the following 6 libraries, it is the LAST archive member with a
.idata$6 section, not the first one, that specifies the DLL.
GdiPlus.lib :x86 archive import FOR u.GdiplusStartup
MSTask.Lib :x86 archive import FOR .._setnetscheduleaccountinformat...@12
WS2_32.Lib :x86 archive import FOR ..inet_pton
ksuser.lib :x86 archive import FOR ..KsCreateTopologyNode
shell32.lib :x86 archive import FOR =.WOWShellExecute
windowscodecs.lib :x86 archive import FOR q.WICSetEncoderFormat_Proxy
This one imports symbols from multiple DLLs. One of them happens to be
in the last archive member, but...
Vfw32.Lib :x86 archive import FOR -.StretchDIB
I have no idea what this one is. objdump can't grok it:
MAPI.Lib :MAPI.Lib: Microsoft Visual C library
$ objdump -f MAPI.Lib
objdump: MAPI.Lib: File format not recognized
So that's 246 PASS, 8 FAIL.
So, I've prepared a patch for dlltool which adds an '--identify-ms'
flag, which modifies the behavior of the '--identify implib' option.
It searches for .idata$6 instead of .idata$7, AND attempts to
disambiguate between the one that specifies the DLL name and all the
other ones that list the symbols by inspecting the flags.
In almost all cases, the one that specifies the DLL name has the flag
value 0x0123
SEC_ALLOC | SEC_LOAD | SEC_DATA | SEC_HAS_CONTENTS
The other ones have flag values 0x00204103
SEC_ALLOC | SEC_LOAD | SEC_HAS_CONTENTS | SEC_IN_MEMORY | SEC_KEEP
This version of dlltool was able to operate on all of the import
libraries in the Windows SDK, except for:
MAPI.Lib: MAPI.Lib: Microsoft Visual C library
=== again, because bfd has no idea how to parse this one
Vfw32.Lib: Import library `Vfw32.Lib' specifies two or more dlls:
`MSVFW32.dll' and `AVIFIL32.dll'
And the following:
WebPost.Lib: x86 archive import FOR WEBPOST.DLL
ddao35.lib: x86 archive import FOR ddao35.dll
ddao35d.lib: x86 archive import FOR ddao35d.dll
ddao35u.lib: x86 archive import FOR ddao35u.dll
ddao35ud.lib: x86 archive import FOR ddao35ud.dll
More on these, later.
Note that this dlltool /succeeded/ on
GdiPlus.lib
MSTask.Lib
WS2_32.Lib
ksuser.lib
shell32.lib
windowscodecs.lib
where the script in my previous post failed. Dlltool can handle the case
where the one that specifies the DLL name is not first.
The five new failures (where the script succeeded) are interesting. In
each case, the rule above (the one has flag value 0x0123, and
the others do not) was incorrect:
$ ~/dlltool.exe --identify WebPost.Lib --identify-ms
flags: 0x0123 datasize: 000c data: 'WEBPOST.DLL'
5745 4250 4f53 542e 444c 4c00 WEBPOST.DLL.
flags: 0x0123 datasize: 0010 data: ''
0400 5770 4269 6e64 546f 5369 7465 4100 ..WpBindToSiteA.
/home/cwilson/dlltool: Import library `WebPost.Lib' specifies two or
more dlls: `WEBPOST.DLL' and `'
The error message is a little confusing: `WEBPOST.DLL' and `'? The
empty name is because the data contains the unprintable character \004
followed by '\0'. Recall for symbols, the first two bytes are a
little-endian count. So this is symbol 0x0004. I *guess* I could check
that both offset 0 and offset 1 contain printable characters. But that's
still just a heuristic, because a really big DLL might have
01my_symbol
where the first two bytes are 0x30 0x31 ('01') representing symbol
number 0x3130 or 12352.
But this wierd case occurs only when the import library appears to NOT
follow the pattern most of them do. In fact, in these five import
libraries, ALL of the .idata$6 sections have flag 0x0123, not just
the one we want.
But, what are they? Do we care?
The ddao35 libraries are the Microsoft JET 3.5 DAO C++ libraries, for
DOS-Win16 (!). Microsoft shipped the Jet 4.0 libraries with WinME and
W2k, and recommends against using ones older than that. Do we care that
you won't be able to dlpreopen (or dlltool --identify) these ancient
import libraries?)
Webpost.Lib (-- webpost.dll) seems to be part of the Web Publishing
Wizard API. I
Re: [PATCH] [cygwin|mingw] fix dlpreopen with --disable-static take 2
Den 2009-01-05 15:08 skrev Charles Wilson:
Peter Rosin wrote:
Den 2009-01-05 06:24 skrev Charles Wilson:
Interesting! Meanwhile, I have done some experiments on my own, as I
don't like the dependence on anything that comes with MinGW when
dealing with libtool and MSVC.
I kind of suspected that. What about the attached? This version needs
to link against libbfd and its dependencies -- so has to be compiled
using mingw gcc. But if this executable was included as part of msys?
Works for me (also works for import libs produced with the msvc branch).
How likely is dllname to make it into msys 1.11? Or will that have to
wait until 1.12? (I'm asking what you think, I know that definitive
answers to such questions are in short supply...)
However, from where I'm sitting adding a tool to MSYS proper that really
only benefits MSVC users (MinGW users could just as well have it
distributed with MinGW) seems a bit far fetched. Or? There are ways to
dig out the info using the dumpbin and lib programs, so it's not a
showstopper if this is not added to MSYS, even if the code in libtool
will be a wee bit longer. When there is an alternative, it seems even
more far fetched to have dllname added to MSYS. But *I* wouldn't say no
to it of course...
BTW, my libiberty is probably old, I had to take out the expandargv
call and add this CONST_STRNEQ definition:
#define CONST_STRNEQ(STR1,STR2) (strncmp (STR1, STR2 , sizeof (STR2) - 1) ==
0)
(Speaking of dependencies, I don't think the current MinGW code
in libtool requires 'file' to be present, and I don't think it is
part of a minimal MSYS install. It's not in my install anyway.)
Only because its been over two years since msys 1.11 will be ready
RSN. It is intended that the file package be included in the minimal
1.11 install.
Oh, ok. Good enough for me.
I have found that for MSVC import libraries the simplest thing is
to list the archive members to get to the dll name. I have tried
with:
lib -nologo -list $implib | grep -v '\.obj$' | sort -u
or, in gnu speak:
ar t $implib | grep -v '\.obj$' | sort -u
I noticed that, but wasn't sure if self-compiled (using MSVC) import
libraries were the same.
They are (at least mine...)
Also, it will not fail for Vfw32.Lib, it will instead list the three
dlls it imports (AVICAP32.dll, AVIFIL32.dll, MSVFW32.dll).
Well, we probably want it to fail. dlpreopen is supposed to work like
dlopen -- and you'd need to dlopen each of the three DLLs separately.
But if you -dlpreopen Vfw32.Lib, you'd need to determine which symbols
IN Vfw32.Lib came from which DLL, and generate three different groups in
you LT_LTX_*[] structure, to register those symbols with their effective
dlopen source:
{ AVICAP32.dll, 0 }
{ symbols }
{ AVIFIL32.dll, 0 }
{ symbols }
{ MSVFW32.dll, 0 }
{ symbols }
I don't think this is going to work well.
I don't really see why this is not going to work well, but I'm not a
heavy libltdl user (yet...) so I'm pretty ignorant on the subject.
So, for the sake of argument, I agree that it should fail. But then I
think it should fail in libtool, not in the tool that digs out the
dll name(s) from the import library. But that's a minor point...
Cheers,
Peter
Re: Fix distcc/ccache interferences with the test suite
Hi Akim, * Akim Demaille wrote on Mon, Jan 05, 2009 at 03:22:24PM CET: Hi all, Are those the only distcc/ccache-induced failures? Thanks, Ralf From 02a3220b061f232d30baf1d1d3bc497d7e9417b8 Mon Sep 17 00:00:00 2001 From: Akim Demaille demai...@gostai.com Date: Mon, 5 Jan 2009 15:10:55 +0100 Subject: [PATCH] Fix distcc/ccache interferences with the test suite. * tests/localization.at (localized compiler messages): Ignore distcc log messages.
Re: [PATCH] [cygwin|mingw] fix dlpreopen with --disable-static take 2
Den 2009-01-04 03:35 skrev Charles Wilson: Peter Rosin wrote: I'm primarily trying to determine what impact this has on my MSVC branch... Den 2009-01-03 02:39 skrev Charles Wilson: *snip* +*cygwin* | *mingw* | *cegcc* ) We should strive to have fewer of these in ltmain.m4sh, not more. Yep. But the problem is, there are really two BIG categories of platforms: those that support ELF-semantics for shared libraries, and those that support PE-DLL semantics. The differences between, say, HP and Linux are in this regard much less significant than the differences between win32 (cygwin, mingw, msvc, even wince) and any *nixoid. And vice verse: cygwin and msvc are much more similar *with regards to the construction of shared libraries* than they are different (even though the implib/static lib formats are non-interchangeable). I'm not sure it would be an improvement, exactly, but we could have a libtool variable $LT_HOST_SUPPORTS_PE_DLL (or a function that takes $host), and replace many of these case $host in *cygwin* | *mingw* | *cegcc* ) ... ;; everything else ) ... ;; esac occurences with 'if host_supports_pe_dll ; then ... ; else ... ; fi' Still ugly, but it means you only have to fix the case $host pattern in one place. I think it should be like it is for everything else, a separate control- ling variable for stuff that seem orthogonal. Many of the case $host constructs should probably be if test $LT_HOST_SUPPORTS_PE_DLL = yes (with the variable in lower case to conform), but I'm sure there are examples where the controlling variable should be named something else. I think there is value in separating these things, it serves as documentation of what pieces of ltmain.m4sh are connected to each other. It also helps when something like the MSVC branch is added as many, but not all, things are equal between MSVC and MinGW. A central function would be a step back IMHO, as a MSVC exception (or whatever exception) for some specific snippet of code would probably get uglier. It's not as if we are talking hundreds of new variables, my guess would be ten or so. But that's without looking at the code for quite some time... That said, I'm still not objecting to this patch as is, one more case $host is not going to kill us. Cheers, Peter
Re: [PATCH] [cygwin|mingw] fix dlpreopen with --disable-static take 2
Den 2009-01-06 02:06 skrev Charles Wilson:
Maybe under that name. But a libbfd-ified version of impgen (as a
replacement for the IMO totally broken -- but part of mingw-utils-0.3 --
reimp program), that happens to also supply an --identify foo
--identify-ms functionality? Not so far-fetched.
Right, but it still seems as if this new fixed impgen tool is closer
to MinGW than to MSYS proper. However, as you say, better ask on a better
list...
Also, it will not fail for Vfw32.Lib, it will instead list the three
dlls it imports (AVICAP32.dll, AVIFIL32.dll, MSVFW32.dll).
Well, we probably want it to fail. dlpreopen is supposed to work like
dlopen -- and you'd need to dlopen each of the three DLLs separately.
But if you -dlpreopen Vfw32.Lib, you'd need to determine which symbols
IN Vfw32.Lib came from which DLL, and generate three different groups in
you LT_LTX_*[] structure, to register those symbols with their effective
dlopen source:
{ AVICAP32.dll, 0 }
{ symbols }
{ AVIFIL32.dll, 0 }
{ symbols }
{ MSVFW32.dll, 0 }
{ symbols }
I don't think this is going to work well.
I don't really see why this is not going to work well, but I'm not a
heavy libltdl user (yet...) so I'm pretty ignorant on the subject.
No, what I meant was: IF you constructed all that then it would work.
BUT there is no support currently in libtool for generating that kind of
thing from a single implib. It would be very ugly indeed. So, at
present if func_msvc_dll_for_implib returns a list of DLLs, what will
the caller actually do? How will the caller know which symbols should
go with which DLL? Does it need to be stateful?
But weigh the cost/benefit: LOADS of new code to write, test, and debug
-- so that a pathological import lib that specifies imports from more
than one DLL can be -dlpreopened. Is it worth it? What's the
opportunity cost -- what /other/ more useful things could that
developer-time be spent on?
Agreed.
So, for the sake of argument, I agree that it should fail. But then I
think it should fail in libtool, not in the tool that digs out the
dll name(s) from the import library. But that's a minor point...
Well, see this and the following thread:
http://sourceware.org/ml/binutils/2008-11/msg00078.html
I'm only saying that from the binutils p.o.v. it makes at least some
sence to report all imported dlls. At least optionally, but again, this
was just a minor point...
Cheers,
Peter
