Re: parallel make bug with subdir-objects (was: Library locations)

[Braden McDaniel]

On Mon, 2007-08-06 at 08:33 +0200, Ralf Wildenhues wrote:
> Hello Braden,
> 
> * Braden McDaniel wrote on Mon, Aug 06, 2007 at 12:27:48AM CEST:
> > 
> > Related to this, I have observed dependency problems when using
> > "make -j[n > 1]" with a nonrecursive make setup (and on a multiprocessor
> > system). I have not filed a bug report on this because I'm at somewhat
> > of a loss as to how to condense this failure into a useful bug report.
> 
> Post the Makefile.am and post a full 'make -jN' output that shows the
> failure, please.  If you use Automake < 1.10, please update to that and
> see if you can still reproduce the failure.

Sorry for the much-delayed follow-up on this; but I think I've found the
problem here. (And, yes, I'm using Automake 1.10.)

My Makefile.am includes targets for both libraries and programs; and the
latter depend on the former. If I refer to a library dependency relative
to top_builddir, make says it has no rule to build the target. If I
refer to the library relative to builddir, it works fine.

-- 
Braden McDaniel   e-mail: <[EMAIL PROTECTED]>
Jabber: <[EMAIL PROTECTED]>

Re: Library locations

[Ralf Wildenhues]

* Braden McDaniel wrote on Mon, Aug 06, 2007 at 03:40:09AM CEST:
> On Mon, 2007-08-06 at 10:53 +1000, Russell Shaw wrote:
> 
> > However, the shared object that dlopen requires is in a new .libs
> > directory:
[...]
> >modules/System/.libs/system.so
[...]
> > Dlopen fails because it is looking for modules/System/system.so
> > When installed, i'll have the install location at:
> > 
> >/usr/lib/$(pkglibdir)/System/system.so
> > 
> > Should i just have dlopen look for modules/System/.libs/system.so
> > and modules/System/system.so and use the first one that succeeds, or
> > is there a cleaner way for avoiding this?
> 
> You can use libltdl's ltdlopen instead. libltdl is included with
> libtool.

The answer isn't wrong, but the answer isn't hitting the nail fully on
the head either.

One way to dlopen uninstalled modules is to
  libtool --mode=execute -dlopen modules/System/system.la program [ARGS...]

which should set LD_LIBRARY_PATH (or the appropriate variable on the
system) to contain modules/System/.libs to find system.so.  You may want
to refer to `libtool' as `$(LIBTOOL)' in Makefile.am (or even add
`$(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS)' for user benefit).

Hope that helps.

Cheers,
Ralf

parallel make bug with subdir-objects (was: Library locations)

[Ralf Wildenhues]

Hello Braden,

* Braden McDaniel wrote on Mon, Aug 06, 2007 at 12:27:48AM CEST:
> 
> Related to this, I have observed dependency problems when using
> "make -j[n > 1]" with a nonrecursive make setup (and on a multiprocessor
> system). I have not filed a bug report on this because I'm at somewhat
> of a loss as to how to condense this failure into a useful bug report.

Post the Makefile.am and post a full 'make -jN' output that shows the
failure, please.  If you use Automake < 1.10, please update to that and
see if you can still reproduce the failure.

Thanks,
Ralf

Re: Library locations

[Braden McDaniel]

On Mon, 2007-08-06 at 10:53 +1000, Russell Shaw wrote:

[snip]

> However, the shared object that dlopen requires is in a new .libs
> directory:
> 
>modules/System/.libs/system.o
>modules/System/.libs/system.so
>modules/System/.libs/system.so.0
>modules/System/.libs/system.so.0.0.0
> 
> 
> Dlopen fails because it is looking for modules/System/system.so
> When installed, i'll have the install location at:
> 
>/usr/lib/$(pkglibdir)/System/system.so
> 
> 
> Should i just have dlopen look for modules/System/.libs/system.so
> and modules/System/system.so and use the first one that succeeds, or
> is there a cleaner way for avoiding this?

You can use libltdl's ltdlopen instead. libltdl is included with
libtool.

> Also, what's the best way to determine from within the program whether
> it is currently installed or not?

I generally provide environment variables that can be set to search
paths or fully-qualified executables, as appropriate. If such
environment variables aren't set, I default to using programs/libraries
in their installed locations.

-- 
Braden McDaniel   e-mail: <[EMAIL PROTECTED]>
Jabber: <[EMAIL PROTECTED]>

Re: Library locations

[Russell Shaw]

Ralf Wildenhues wrote:

Hello Braden, Russell,

* Braden McDaniel wrote on Sun, Aug 05, 2007 at 08:27:07PM CEST:

On Mon, 2007-08-06 at 03:49 +1000, Russell Shaw wrote:

I want it to be like:

   System_LTLIBRARIES = modules/System/system.la
   modules_System_system_la_SOURCES = modules/System/system.c
   modules_System_system_la_LDFLAGS = -module

That makes "modules/System/system.la" ok, but it still puts
system.o and system.lo in the top level src directory.


The latter feature is governed by the Automake option subdir-objects.
Note that the option has existed for a long time, but its functionality
has been improved lately, e.g., for non-C sources.  See the NEWS file
for details.


Hi,
I forgot about that option.


Look again. That's entirely inconsistent with my experience with
automake's (1.10) behavior. The object file generated from the above
should be "modules/System/modules_System_system_la-system.lo".


Then you have subdir-objects set.


That means the program won't work in the uninstalled state,
because dlopen is looking for system.o as modules/System/system.o.

That seems unlikely. dlopen should be looking for system.so. I don't
imagine it cares about the location of system.o.


I agree with Braden here, dlopen should not care about object positions
and things should work with subdir-objects and without.  If you have a
failure setup, please show us how to reproduce it.


I have:

  AUTOMAKE_OPTIONS = subdir-objects

  System_LTLIBRARIES = modules/System/system.la
  modules_System_system_la_SOURCES = modules/System/system.c
  modules_System_system_la_LDFLAGS = -module
  AM_LDFLAGS = -export-dynamic

and now it works better with:

  modules/System/system.la
  modules/System/system.lo
  modules/System/system.o

generated.


However, the shared object that dlopen requires is in a new .libs directory:

  modules/System/.libs/system.o
  modules/System/.libs/system.so
  modules/System/.libs/system.so.0
  modules/System/.libs/system.so.0.0.0


Dlopen fails because it is looking for modules/System/system.so
When installed, i'll have the install location at:

  /usr/lib/$(pkglibdir)/System/system.so


Should i just have dlopen look for modules/System/.libs/system.so
and modules/System/system.so and use the first one that succeeds, or
is there a cleaner way for avoiding this?

Also, what's the best way to determine from within the program whether
it is currently installed or not?

Re: Library locations

[Russell Shaw]

Ralf Wildenhues wrote:

Hello Braden, Russell,

* Braden McDaniel wrote on Sun, Aug 05, 2007 at 08:27:07PM CEST:

On Mon, 2007-08-06 at 03:49 +1000, Russell Shaw wrote:

I want it to be like:

   System_LTLIBRARIES = modules/System/system.la
   modules_System_system_la_SOURCES = modules/System/system.c
   modules_System_system_la_LDFLAGS = -module

That makes "modules/System/system.la" ok, but it still puts
system.o and system.lo in the top level src directory.


The latter feature is governed by the Automake option subdir-objects.
Note that the option has existed for a long time, but its functionality
has been improved lately, e.g., for non-C sources.  See the NEWS file
for details.


Hi,
I forgot about that option.


Look again. That's entirely inconsistent with my experience with
automake's (1.10) behavior. The object file generated from the above
should be "modules/System/modules_System_system_la-system.lo".


Then you have subdir-objects set.


That means the program won't work in the uninstalled state,
because dlopen is looking for system.o as modules/System/system.o.

That seems unlikely. dlopen should be looking for system.so. I don't
imagine it cares about the location of system.o.


I agree with Braden here, dlopen should not care about object positions
and things should work with subdir-objects and without.  If you have a
failure setup, please show us how to reproduce it.


I have:

  AUTOMAKE_OPTIONS = subdir-objects

  System_LTLIBRARIES = modules/System/system.la
  modules_System_system_la_SOURCES = modules/System/system.c
  modules_System_system_la_LDFLAGS = -module
  AM_LDFLAGS = -export-dynamic

and now it works better with:

  modules/System/system.la
  modules/System/system.lo
  modules/System/system.o

generated.


However, the shared object that dlopen requires is in a new .libs directory:

  modules/System/.libs/system.o
  modules/System/.libs/system.so
  modules/System/.libs/system.so.0
  modules/System/.libs/system.so.0.0.0


Dlopen fails because it is looking for modules/System/system.so
When installed, i'll have the install location at:

  /usr/lib/$(pkglibdir)/System/system.so


Should i just have dlopen look for modules/System/.libs/system.so
and modules/System/system.so and use the first one that succeeds, or
is there a cleaner way for avoiding this?

Re: Library locations

[Russell Shaw]

Braden McDaniel wrote:

On Mon, 2007-08-06 at 03:49 +1000, Russell Shaw wrote:

Hi,
In automake.am, i have:

   System_LTLIBRARIES = system.la
   system_la_SOURCES = modules/System/system.c
   system_la_LDFLAGS = -module

This compiles ok. However, it puts system.la in the top level src directory.


I want it to be like:

   System_LTLIBRARIES = modules/System/system.la
   modules_System_system_la_SOURCES = modules/System/system.c
   modules_System_system_la_LDFLAGS = -module


That makes "modules/System/system.la" ok, but it still puts
system.o and system.lo in the top level src directory.


Look again. That's entirely inconsistent with my experience with
automake's (1.10) behavior. The object file generated from the above
should be "modules/System/modules_System_system_la-system.lo".


That means the program won't work in the uninstalled state,
because dlopen is looking for system.o as modules/System/system.o.


That seems unlikely. dlopen should be looking for system.so. I don't
imagine it cares about the location of system.o.


Hi,
I have:

  AUTOMAKE_OPTIONS = subdir-objects

  System_LTLIBRARIES = modules/System/system.la
  modules_System_system_la_SOURCES = modules/System/system.c
  modules_System_system_la_LDFLAGS = -module

and now it works better with:

  modules/System/system.la
  modules/System/system.lo
  modules/System/system.o

generated.

Re: Library locations

[Braden McDaniel]

On Sun, 2007-08-05 at 23:33 +0200, Ralf Wildenhues wrote:

[snip]

> Then you have subdir-objects set.

I do; I forgot about that.

Related to this, I have observed dependency problems when using
"make -j[n > 1]" with a nonrecursive make setup (and on a multiprocessor
system). I have not filed a bug report on this because I'm at somewhat
of a loss as to how to condense this failure into a useful bug report.
If you're interested in knowing about such failures, any guidance you
could provide here would be appreciated.

I can certainly tar up my source tree and send it to you; though I
appreciate that could be a bit overwhelming.

-- 
Braden McDaniel   e-mail: <[EMAIL PROTECTED]>
Jabber: <[EMAIL PROTECTED]>

Re: Library locations

[Ralf Wildenhues]

Hello Braden, Russell,

* Braden McDaniel wrote on Sun, Aug 05, 2007 at 08:27:07PM CEST:
> On Mon, 2007-08-06 at 03:49 +1000, Russell Shaw wrote:
> > 
> > I want it to be like:
> > 
> >System_LTLIBRARIES = modules/System/system.la
> >modules_System_system_la_SOURCES = modules/System/system.c
> >modules_System_system_la_LDFLAGS = -module
> > 
> > That makes "modules/System/system.la" ok, but it still puts
> > system.o and system.lo in the top level src directory.

The latter feature is governed by the Automake option subdir-objects.
Note that the option has existed for a long time, but its functionality
has been improved lately, e.g., for non-C sources.  See the NEWS file
for details.

> Look again. That's entirely inconsistent with my experience with
> automake's (1.10) behavior. The object file generated from the above
> should be "modules/System/modules_System_system_la-system.lo".

Then you have subdir-objects set.

> > That means the program won't work in the uninstalled state,
> > because dlopen is looking for system.o as modules/System/system.o.
> 
> That seems unlikely. dlopen should be looking for system.so. I don't
> imagine it cares about the location of system.o.

I agree with Braden here, dlopen should not care about object positions
and things should work with subdir-objects and without.  If you have a
failure setup, please show us how to reproduce it.

Cheers,
Ralf

Re: Library locations

[Braden McDaniel]

On Mon, 2007-08-06 at 03:49 +1000, Russell Shaw wrote:
> Hi,
> In automake.am, i have:
> 
>System_LTLIBRARIES = system.la
>system_la_SOURCES = modules/System/system.c
>system_la_LDFLAGS = -module
> 
> This compiles ok. However, it puts system.la in the top level src directory.
> 
> 
> I want it to be like:
> 
>System_LTLIBRARIES = modules/System/system.la
>modules_System_system_la_SOURCES = modules/System/system.c
>modules_System_system_la_LDFLAGS = -module
> 
> 
> That makes "modules/System/system.la" ok, but it still puts
> system.o and system.lo in the top level src directory.

Look again. That's entirely inconsistent with my experience with
automake's (1.10) behavior. The object file generated from the above
should be "modules/System/modules_System_system_la-system.lo".

> That means the program won't work in the uninstalled state,
> because dlopen is looking for system.o as modules/System/system.o.

That seems unlikely. dlopen should be looking for system.so. I don't
imagine it cares about the location of system.o.

-- 
Braden McDaniel   e-mail: <[EMAIL PROTECTED]>
Jabber: <[EMAIL PROTECTED]>

Library locations

[Russell Shaw]

Hi,
In automake.am, i have:

  System_LTLIBRARIES = system.la
  system_la_SOURCES = modules/System/system.c
  system_la_LDFLAGS = -module

This compiles ok. However, it puts system.la in the top level src directory.


I want it to be like:

  System_LTLIBRARIES = modules/System/system.la
  modules_System_system_la_SOURCES = modules/System/system.c
  modules_System_system_la_LDFLAGS = -module


That makes "modules/System/system.la" ok, but it still puts
system.o and system.lo in the top level src directory.

That means the program won't work in the uninstalled state,
because dlopen is looking for system.o as modules/System/system.o.