Re: Patch to allow make to load plugins that add new functions.

[Tim Murphy]

Hi :-)

This is a response to a rather old email about loadable modules.

I'm keen to see the patch for the load operator if you have it :-).

On 5 April 2012 15:12, Paul Smith psm...@gnu.org wrote:

 Before going too much further note that I've got a semi-implemented
 load operator in my source already, which fulfills a similar function
 except in a less sophisticated way: it just calls a function in the
 loaded object after loading and that function can do whatever it wants.

 I can send along a patch if you're interested in checking it out.  I
 have some regression tests added, documentation, autoconf, etc.

Regards,

Tim

-- 
You could help some brave and decent people to have access to
uncensored news by making a donation at:

http://www.thezimbabwean.co.uk/friends/

___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: Patch to allow make to load plugins that add new functions.

[Samkit Jain]

I have been breaking my head over a simple thing of measuring the progress
of build i.e. I wanted to know how many files are supposed to be built and
how many are already built till now giving an idea of how much more time I
would have to wait.
The only way I could think of this was to have hook/callback support from
make internally to makefile scripts or a plugin for things like file added
as prerequisite, prerequisite is newer than target, prerequisite is not
newer than target.
Would the plugin approach be able to help in with this kind of requests? It
would be great if plugin can redirect calls to makefile rather than having
everything inside itself which might be c/c++ and sometimes too much to
write compared to precise/smaller make language.

With my above proposal:

1. A hook in makefile/plugin for file added as prerequisite will be
invoked during first phase of make.
2. A hook in makefile/plugin for prerequisite is newer than target will
be invoked during the second phase of make.
3. A hook in makefile/plugin for prerequisite is not newer than target
will be invoked during the second phase of make.

Now inside the hook, a simple linear equation can be written to provide the
measure of progress. In the example above, it can be 1 - 2 - 3 files are
remaining to be built.

I am not familiar with make code base so my requirement might be very
ambitious but nevertheless I want to put my point forward.

-- 
- Samkit
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: Patch to allow make to load plugins that add new functions.

[Paul Smith]

On Sat, 2012-05-12 at 00:04 +0530, Samkit Jain wrote:
 I have been breaking my head over a simple thing of measuring the
 progress of build

It's simple to say, but it's not simple to know.  The structure of make
and how it performs builds means that it's not possible to know this.

make simply starts work, and keeps going building out of date things one
at a time until there's nothing left to build.  Only once make discovers
that there's nothing left, does it know that it's done.

It's NOT the case that make figures out everything that has to be built,
then walks through that list building it all.

 Would the plugin approach be able to help in with this kind of
 requests?

No, I don't see how.  Only a massive change to the way make works will
allow this information to be available.

 With my above proposal:
 
 1. A hook in makefile/plugin for file added as prerequisite will be
 invoked during first phase of make.
 2. A hook in makefile/plugin for prerequisite is newer than target
 will be invoked during the second phase of make.
 3. A hook in makefile/plugin for prerequisite is not newer than
 target will be invoked during the second phase of make.
 
 Now inside the hook, a simple linear equation can be written to
 provide the measure of progress. In the example above, it can be 1 -
 2 - 3 files are remaining to be built.

That is not good enough.  Consider this makefile:

all: a b c
debug: x y z
a: e f g
b: h i j
c: k l m n o
x: s
y: t
z: u v w

The total number of targets known in the makefile is 24.  Given make
or make all, the maximum number of targets that we *might* want to
build is 15.

However, make doesn't know that until it's checked the last target: when
make starts to build all it sees the prerequisites a b c.  Here make
thinks there are 4 total targets to build.  Now make looks at a, and
now it sees there are 7 total targets.  Once it checks all those and
starts on b make will see there are really 10 total targets.  And once
it's done all those and looks at c it will realize there are 15 total
targets.  Once it's done with 'c' it will realize there's nothing left
to do and NOW make knows there were a total of 15 total targets to
build.  But, it's done at that point anyway.



___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: Patch to allow make to load plugins that add new functions.

[Tim Murphy]

Knowing make's current idea of where it is would still be nice even if
it keeps changing.

I also would like to hook to the start and end of the execution of
every recipe because there are many uses for this, one of which might
be progress but triggering retries (when unreliable network shares are
involved), logging what recipes failed with the return code so on are
other use cases.  You can do a lot of this if you write your makefiles
specially but that's not much use for building things written by other
people or other tools.

Regards,

Tim



On 12 May 2012 06:38, Samkit Jain samkit.fe...@gmail.com wrote:
 Yeah, I guess what I was thinking simple is more complex to code.
 But considering this kind of request is coming to GNU make since a couple
 years now(the first I saw was in 2007), there should be a way to handle it
 in generic manner.
 But obviously saying is easier than doing, so I will also start reading code
 to understand the complexity myself.

 --
 - Samkit

 ___
 Bug-make mailing list
 Bug-make@gnu.org
 https://lists.gnu.org/mailman/listinfo/bug-make




-- 
You could help some brave and decent people to have access to
uncensored news by making a donation at:

http://www.thezimbabwean.co.uk/friends/

___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: Patch to allow make to load plugins that add new functions.

[Eric Melski]

On 04/05/2012 11:59 PM, Tim Murphy wrote:


As for other tools like Electric Make, I want to be able to arm-twist
them to support plugins (politely of course, Eric :-) ) without them
being able to say it's impossible or unreliable or that it requires
them to simulate the volatile internals of make to some infeasible
degree.  I realise that this is just my interest but it makes the
difference between bothering to use a plugin mechanism and not
bothering at all since it couldn't offer me a uniform makefile that I
could use across systems - I'd solve all my worries on GNU make and
still be busted on emake and since GNU make doesn't offer that amazing
filesystem which tells you when you have parallelsim problems and
corrects them. In other words I can't give up on emake without
converting 100s of developers into experts in writing good parallel
makefiles and inventing a way to get make to work on clusters of
machines (without pvmgmake's horrendously complicated setup).   People
are so much more difficult than software.

So my vote, FWIW, is to define what one does expect make to offer
plugins (the nice bits of make) and define what it definitely
doesn't and won't in the near future.  Anyone who wants a lot more is
probably going to have to customise make itself anyhow.  I think the
plugin should say what environment it expects and make should decide
if it can offer that.  If make fails on something it claims to support
then it's a bug on make otherwise it's clearly a plugin problem etc.

As you say with expectations, one has to set them at some point.


Adding a plugin mechanism to ElectricMake has been on my wishlist for 
quite some time actually.  I'd love to collaborate with Paul, et al, on 
the design so that we come up with something that works for both gmake 
and emake.  In my wildest imaginings we could cook up something that 
would be binary-compatible, so we only have to compile a plugin once to 
use it with both tools, but I doubt that's really achievable.  The next 
best thing would be source-compatibility, so we at least don't have to 
tweak the plugin implementation to work with each tool (although I 
suppose that too could be manageable, in the same way that you see 
plugins for scripting languages with different 'bindings' for different 
languages -- still, we would need to agree on some fundamentals for that 
to work I think).


The single biggest issue in this regard is a difference in architecture 
between gmake and emake.  Fundamentally, gmake is one-make-per-process, 
and there are a lot of assumptions and shortcuts baked into the code 
because of that.  emake is many-makes-per-process.  In practical terms 
that means that we have to pass around additional context when doing 
expansions (eg, you have to know which make instance the expansion is 
part of, so that you use the right variable table).  I think to make 
something that would work for both gmake and emake we would have to 
abstract away this difference -- for example, rather than directly 
accessing data structures, we would define a set of API's for the 
various operations that are needed during expansion.


So there's definitely interest in collaboration on our end; is there any 
interest from the gmake devs?


Best regards,

Eric Melski
Architect
Electric Cloud, Inc.



___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: Patch to allow make to load plugins that add new functions.

[Eldar Abusalimov]

Hi,

2012/4/10 Eric Melski s...@melski.net

 So there's definitely interest in collaboration on our end; is there any
 interest from the gmake devs?


I'm currently working on improving performance of gmake variable expansion
engine. In particular, I have rewritten the way of handling 'call'
arguments and iteration variable of 'foreach' function, hereby achieving
O(1) complexity of variable look up when being inside such contexts
(instead of O(depth), where depth is number of nesting $(call ...) or
$(foreach ...)). Now the patched version shows more than 2x speed up on a
benchmark based on gmsl-tests and even more in case of deep call recursion,
e.g. it is 7x faster on reversing a list of 1000 x'es.

The patch also involves different memory management optimizations such as
allocating as mush as possible on the stack and avoiding unnecessary
memcpy'es, but the first thing that I have rewritten was variable_buffer.
Instead of using a single global pointer and swapping it when necessary, a
special struct vbuffer object was introduced, which is passed in every
'func_xxx' function and other variable expansion routines. Practically, it
is possible to create a local buffer, and perform an expansion into it,
without interfering with any global structure. So, I think, such interface
could be a part of gmake-/emake-independent API for plugins.

Now the patch is mostly done, but I didn't send it because some parts of
code are still poorly documented. I planned to complete in a few weeks, but
I have no time right now, so you could take a look at draft version here:
http://pastebin.com/UGs6aawV
I would appreciate any comments.

-- 
Best regards,
Eldar Sh. Abusalimov
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: Patch to allow make to load plugins that add new functions.

[Tim Murphy]

Hi :-)

On 6 April 2012 01:16, Paul Smith psm...@gnu.org wrote:
 On Thu, 2012-04-05 at 23:59 +0100, Tim Murphy wrote:
 I see the value in a plugin system as being that I don't have to
 recompile the plugins for every version of make. In a way it's tending
 towards why bother if you did have to do that.

 Well, this kind of combines with my other issue regarding automatic
 remaking of objects :-)

 The idea is that you wouldn't have a predefined set of make plugin
 OBJECTS that you distributed.  Instead you'd keep the SOURCE CODE for
 your plugins for your build environment right there in your source tree,
 and build them as part of your build process.  If they are automatically
 recompiled when out of date, this would be seamless.

I get it - a package can have it's own little collection of plugins
and keep them in source control with everything else.

 I have a hard time envisioning massive, complex plugins: most likely
 they'll be small, straightforward functions that perform targeted
 operations more efficiently than make (or Guile) can do them.  Such
 things would only take a couple of seconds to build and would be
 unnoticeable in any serious build environment.

I don't see any problem with performance at all.  I can see that with
some build systems that generate makefile it's not going to be easy to
have a little pre-build step where you build such plugins and thus
you're going to want to be able to trigger them from the main build
because that's all there is.  So this makes more sense to me now.
Since I don't personally need it, I'd be  thinking of that as the
feature to put in after an environment variable/commandline option
since this lets the feature get out there and get tested now by some
of us and it doesn't preclude the addition of new syntax.

 You also have to think about where in makes startup these things
 happen. Some stuff needs to be done early, some later - if your plugin
 loader needs a fully initialised make process (because it has to read
 a makefile to do load) then although you might think it is infinitely
 flexible, it does come with limits.

 Heh, I didn't say infinitely flexible; that's too much to ask.  We
 have the --eval command line option already.  I'm open to adding an
 environment variable if someone can describe a powerful use-case.

 I have to admit to a secondary agenda in that I'm thinking about tools
 like Electric Accelerator which mimic gnu make - I'd like them to be
 able to implement plugins too so that my build system can be
 accelerated ($) or free :-) depending on the users choice.  This
 is why I was thinking of it being quite defined and with restricted
 access and using versions so that it's clear what needs to be
 supported and what's offered.

 Hm.  I'd need to understand better what exactly would be involved here.
 What kind of plugins would be used for this and what would they do?  I'm
 not that jazzed about designing in a vacuum, especially just to enable
 proprietary software to work better with GNU make :-).

 What I really DON'T want to do is provide a locked down interface, which
 then turns out to be too restrictive to be useful and we're constantly
 fielding requests to open this or that, and having to worry about
 backward compatibility, etc.  I'd prefer to leave things open, at least
 for a while until we get some feedback/experience.  Yes, this does shift
 the burden away from me and onto the plugin writers (where it belongs of
 course!! ;-)), but it's not like there are so many versions of make
 floating around that this is that onerous.

Most programs which are not designed with the idea that they will be
used as libraries or that they will have lots of plugins are not
really structured to make it easy.  Make seems to be a mixture like
anything - the stuff about adding functions is very nice but, for
example, there is no clear way to use plugins to change the
uptodateness of files to use md5 as discussed in another thread. So
there is a nice portion and a not so nice portion. My major interest
is in having an easy life and since I've been a makefile writer most
of the time, I am looking for whatever makes that easier. If I have to
go to the extra trouble of writing plugins then I want them to not be
a constant source of change and I want them not to fail mysteriously.
Basically I want to use the nice portion of Make's API and know that
make developers have some degree of commitment to keeping it working
as is.  I don't want to dip into the nasty bit of make because that
can hardly ever be made to do what I want except by extreme trickery
which might work now but become unworkable in the next release.

As for other tools like Electric Make, I want to be able to arm-twist
them to support plugins (politely of course, Eric :-) ) without them
being able to say it's impossible or unreliable or that it requires
them to simulate the volatile internals of make to some infeasible
degree.  I realise that this is just my interest but it makes the

Re: Patch to allow make to load plugins that add new functions.

[David Boyce]

On Fri, Apr 6, 2012 at 2:59 AM, Tim Murphy tnmur...@gmail.com wrote:
 I was thinking of marking this feature as experimental in the first
 release (in the documentation), just to be more clear on expectations.

 Very very much so - there are many platforms to support anyhow and
 when someone eventually tries plugins on them we might see new
 problems so better to let it take time to mature.

Apropos of this, the Apache Portable Runtime has a nice abstraction
over dlopen and LoadLibrary. Assuming they're license-compatible, it
might be worth looking into borrowing that logic for this feature.

-David Boyce

___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: Patch to allow make to load plugins that add new functions.

[Eli Zaretskii]

 Date: Fri, 6 Apr 2012 12:08:33 -0400
 From: David Boyce david.s.bo...@gmail.com
 Cc: bug-make@gnu.org bug-make@gnu.org
 
 Apropos of this, the Apache Portable Runtime has a nice abstraction
 over dlopen and LoadLibrary.

Is it significantly better than what libltdl provides?  The advantage
of the latter is that it's readily available for both Posix and MinGW
builds.


___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: Patch to allow make to load plugins that add new functions.

[David Boyce]

On Fri, Apr 6, 2012 at 12:40 PM, Eli Zaretskii e...@gnu.org wrote:
 Date: Fri, 6 Apr 2012 12:08:33 -0400
 From: David Boyce david.s.bo...@gmail.com
 Cc: bug-make@gnu.org bug-make@gnu.org

 Apropos of this, the Apache Portable Runtime has a nice abstraction
 over dlopen and LoadLibrary.

 Is it significantly better than what libltdl provides?  The advantage
 of the latter is that it's readily available for both Posix and MinGW
 builds.

Sorry, I've never used libltdl. Maybe it would have been better just
to say libraries exist to paper over the differences between various
platforms dynamic linking APIs; consider not reinventing the wheel.

David

___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: Patch to allow make to load plugins that add new functions.

[Daniel Herring]

On Fri, 6 Apr 2012, David Boyce wrote:


On Fri, Apr 6, 2012 at 2:59 AM, Tim Murphy tnmur...@gmail.com wrote:

I was thinking of marking this feature as experimental in the first
release (in the documentation), just to be more clear on expectations.


Very very much so - there are many platforms to support anyhow and
when someone eventually tries plugins on them we might see new
problems so better to let it take time to mature.


Apropos of this, the Apache Portable Runtime has a nice abstraction
over dlopen and LoadLibrary. Assuming they're license-compatible, it
might be worth looking into borrowing that logic for this feature.


Along those lines, libtool also has libltdl.  I've never really used 
either system.  (Guilty of rolling yet another custom one.)


- Daniel

___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: Patch to allow make to load plugins that add new functions.

[Eli Zaretskii]

 Date: Fri, 6 Apr 2012 13:30:31 -0400
 From: David Boyce david.s.bo...@gmail.com
 Cc: tnmur...@gmail.com, bug-make@gnu.org
 
 Sorry, I've never used libltdl. Maybe it would have been better just
 to say libraries exist to paper over the differences between various
 platforms dynamic linking APIs; consider not reinventing the wheel.

Well, FWIW, I've built Guile, which uses libltdl, with MinGW, and
didn't have any problems.

___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: Patch to allow make to load plugins that add new functions.

[Paul Smith]

On Fri, 2012-04-06 at 22:35 +0300, Eli Zaretskii wrote:
  Date: Fri, 6 Apr 2012 13:30:31 -0400
  From: David Boyce david.s.bo...@gmail.com
  Cc: tnmur...@gmail.com, bug-make@gnu.org
  
  Sorry, I've never used libltdl. Maybe it would have been better just
  to say libraries exist to paper over the differences between various
  platforms dynamic linking APIs; consider not reinventing the wheel.
 
 Well, FWIW, I've built Guile, which uses libltdl, with MinGW, and
 didn't have any problems.

Maybe this is just irrational prejudice but I've never had a good
experience using libtool and I'm SO uninterested in fighting with it in
GNU make.

I will admit that my distaste is so extreme that I've not even gone near
it for the last 5 years or more.  Maybe (hopefully) it's much improved.



___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: Patch to allow make to load plugins that add new functions.

[Eli Zaretskii]

 From: Paul Smith psm...@gnu.org
 CC: David Boyce david.s.bo...@gmail.com, bug-make@gnu.org
 Date: Fri, 6 Apr 2012 16:13:47 -0400
 
 Maybe this is just irrational prejudice but I've never had a good
 experience using libtool and I'm SO uninterested in fighting with it in
 GNU make.
 
 I will admit that my distaste is so extreme that I've not even gone near
 it for the last 5 years or more.  Maybe (hopefully) it's much improved.

libltdl has nothing to do with libtool.  It is just a library for
dynamically loading shared libraries, presenting the same API on all
supported platforms.  You can use that library but not libtool itself,
they have nothing in common except the team that is developing them.


___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: Patch to allow make to load plugins that add new functions.

[Tim Murphy]

On 6 April 2012 21:55, Eli Zaretskii e...@gnu.org wrote:
 From: Paul Smith psm...@gnu.org
 CC: David Boyce david.s.bo...@gmail.com, bug-make@gnu.org
 Date: Fri, 6 Apr 2012 16:13:47 -0400

 Maybe this is just irrational prejudice but I've never had a good
 experience using libtool and I'm SO uninterested in fighting with it in
 GNU make.

 I will admit that my distaste is so extreme that I've not even gone near
 it for the last 5 years or more.  Maybe (hopefully) it's much improved.

 libltdl has nothing to do with libtool.  It is just a library for
 dynamically loading shared libraries, presenting the same API on all
 supported platforms.  You can use that library but not libtool itself,
 they have nothing in common except the team that is developing them.



I had a little look at libtdl.   To be brutal I thought that using
dlopen/LoadLibrary directly was *much* easier. There isn't really
anything madly complicated about what's being done.

I realise that this might mean problems on architectures that don't
support these functions but I can't think up examples.

Either way I think it's again not something one should get stuck on -
the requirements of a plugin system should be quite simple and don't
really need many lines of code so I think that doing whatever's
quickest and easiest right now is good.  i.e in the end you'll have a
function like load_plugin() which will use whatever method is
required and the plugin initialiser will get called with some
information and that's about it.  If one suddenly decides to use a
library midway through for some reason then that's great - shouldn't
matter.  If using a new library causes the whole thing to stop working
properly for some reason then the design is probably too complicated
or the library is not suitable.

Regards,

Tim
-- 
You could help some brave and decent people to have access to
uncensored news by making a donation at:

http://www.thezimbabwean.co.uk/friends/

___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: Patch to allow make to load plugins that add new functions.

[David Boyce]

On Fri, Apr 6, 2012 at 5:36 PM, Tim Murphy tnmur...@gmail.com wrote:
 I had a little look at libtdl.   To be brutal I thought that using
 dlopen/LoadLibrary directly was *much* easier. There isn't really
 anything madly complicated about what's being done.

To clarify: when I originally spoke about APR, I was not suggesting
that GNU make actually incorporate and use APR but just that useful
bits of library-loading code could be stolen from it. And I assume the
same could go for libltdl. It might be worth at least looking at these
just to make sure you don't paint yourself into a corner API-wise
vis-a-vis HPUX or OS X or whatever.

David B

___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

RE: Patch to allow make to load plugins that add new functions.

[Lawrence Ibarria]

I like this idea quite a bit. 

I see this as still work in progress to define what type of functions the 
plugins can have. 
Maybe they can even create or change make variables. 

In the case of variable_buffer_output, I'd suggest that each plugin has an 
'initialization function'. This is a handshake where make provides the plugin 
with function pointers (avoids the extern function) and the plugin can provide 
its name, versions, and the function table. That simplifies the code that right 
now tries to get every single variable from the dl_handle. 

-- Lawrence


-Original Message-
From: bug-make-bounces+libarria=nvidia@gnu.org 
[mailto:bug-make-bounces+libarria=nvidia@gnu.org] On Behalf Of Tim Murphy
Sent: Thursday, April 05, 2012 2:51 AM
To: bug-make@gnu.org
Subject: Patch to allow make to load plugins that add new functions.

Hi,

I am between jobs which made me realise that I am absolutely free to
contribute to make for about 10 days :-)

The one thing I have wanted the most and the longest is a way to add
new functions without having to rebuild and look after a custom
version of make.  Essentially this should allow people to extend make
without necessarily having to maintain a custom version of it.

So here is an attempt at a plugin system for your interest. Currently
it ought to work on systems with dlopen.  LoadLibrary on windows
should do the job admirably but I haven't had time to have a go at
that yet.  So the feature is optional (configure --with-plugins).  I
am looking for feedback indicating interest or disagreement or
whatever and mostly I just want to put some code into the open so that
it's there.

My test plugin adds the $(equal ($X),$(Y)) function which returns $(X)
if the variables X and Y match as strings and returns the empty string
if not. I have often wanted this for use with $(if) and have a
horrible and inefficient macro for doing it now but it's something I
feel really ought to be built in.

Here's how you tell make to use a plugin:

./make --plugin=tests/plugins/test-plugin.so -f tests/plugins/plugintest1.mk


Here's the code for this simple test plugin:

#include plugin.h
#include string.h

/* The next line is a cheat  to save from having to include all the
make headers and possibly end up linking to all sorts of make object
files which defeats the purpose.
   Don't like it and  will have to think of something better: */
extern char *variable_buffer_output (char *ptr, const char *string,
unsigned int length);

int gnumake_plugin_api_version = 1; /* what api the plugin expects
make to provide - just a way for make to know if it's loading a plugin
that should work  */

int gnumake_plugin_major_version = 1; /* The version of this plugin -
might be useful for problem reports */
char gnumake_plugin_name[] = test_plugin;  /* also the name of the
feature added to .FEATURES */

/* $(equal astring1,astring2) returns blank, $(equal
astring1,astring1) returns astring1
 * The purpose of this function is to allow equality tests in $(if)
functions e.g. $(if $(equal $(X),$(Y)),something,else)
 * */
static char *function_equal(char *o, char **argv, const char *funcname)
{
  if (argv[0]  argv[1]  strcmp(argv[0],argv[1]) == 0)
{
  o = variable_buffer_output (o, argv[0], strlen (argv[0]));
}
  else
{
  o = variable_buffer_output (o, , 0);
}

return o;
}

padded_plugin_entry gnumake_plugin_function_table[] = {
{equal, 2, 2, 1, function_equal}, /* 2 args, expand them first */
 {, 0, 0, 0, (void *)0}
};

/* end -- */



So a plugin is fairly easy to write and it's not hard to build either.

I have tried to set it all up with autoconf and so on but I'm not an
expert at that.  There is also nothing stopping you from loading the
same plugin twice at the moment and it needs more tests and some
documentation so I realise that this patch is not right yet.

There is one set of modifications to main.c to add calls to
load_plugin() which is in a new file, manage-plugins.c, and there is a
new commandline option (--plugin=) which can be used multiple times.

That's it!

Regards,

Tim

-- 
You could help some brave and decent people to have access to
uncensored news by making a donation at:

http://www.thezimbabwean.co.uk/friends/

___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: Patch to allow make to load plugins that add new functions.

[Tim Murphy]

Hi,

On 5 April 2012 20:52, Lawrence Ibarria libar...@nvidia.com wrote:
 I like this idea quite a bit.

 I see this as still work in progress to define what type of functions the 
 plugins can have.
 Maybe they can even create or change make variables.

At this point, plugins are naughty and are kind of able to access
anything in make that they have the prototype for (using -rdynamic to
make all the symbols available).  This is definitely dodgy since
there's nothing stopping a plugin from trying to access something that
might be intended as fairly volatile within make.

 In the case of variable_buffer_output, I'd suggest that each plugin has an 
 'initialization function'. This is a handshake where make provides the plugin 
 with function pointers (avoids the extern function) and the plugin can 
 provide its name, versions, and the function table. That simplifies the code 
 that right now tries to get every single variable from the dl_handle.

 -- Lawrence

Ok, good idea - I'll add something to look for an initialiser and call
it and as you say, pass it a table. Returning values is critical and I
think that looking up variable values at the least is also probably
important.  Another thing I've wanted for ages is to be able to know
if some target was declared already - I want plugins to have the
ability to look at make's target database if possible although not
necessarily modify it.

Regards,

Tim


 -Original Message-
 From: bug-make-bounces+libarria=nvidia@gnu.org 
 [mailto:bug-make-bounces+libarria=nvidia@gnu.org] On Behalf Of Tim Murphy
 Sent: Thursday, April 05, 2012 2:51 AM
 To: bug-make@gnu.org
 Subject: Patch to allow make to load plugins that add new functions.

 Hi,

 I am between jobs which made me realise that I am absolutely free to
 contribute to make for about 10 days :-)

 The one thing I have wanted the most and the longest is a way to add
 new functions without having to rebuild and look after a custom
 version of make.  Essentially this should allow people to extend make
 without necessarily having to maintain a custom version of it.

 So here is an attempt at a plugin system for your interest. Currently
 it ought to work on systems with dlopen.  LoadLibrary on windows
 should do the job admirably but I haven't had time to have a go at
 that yet.  So the feature is optional (configure --with-plugins).  I
 am looking for feedback indicating interest or disagreement or
 whatever and mostly I just want to put some code into the open so that
 it's there.

 My test plugin adds the $(equal ($X),$(Y)) function which returns $(X)
 if the variables X and Y match as strings and returns the empty string
 if not. I have often wanted this for use with $(if) and have a
 horrible and inefficient macro for doing it now but it's something I
 feel really ought to be built in.

 Here's how you tell make to use a plugin:

 ./make --plugin=tests/plugins/test-plugin.so -f tests/plugins/plugintest1.mk


 Here's the code for this simple test plugin:

 #include plugin.h
 #include string.h

 /* The next line is a cheat  to save from having to include all the
 make headers and possibly end up linking to all sorts of make object
 files which defeats the purpose.
   Don't like it and  will have to think of something better: */
 extern char *variable_buffer_output (char *ptr, const char *string,
 unsigned int length);

 int gnumake_plugin_api_version = 1; /* what api the plugin expects
 make to provide - just a way for make to know if it's loading a plugin
 that should work  */

 int gnumake_plugin_major_version = 1; /* The version of this plugin -
 might be useful for problem reports */
 char gnumake_plugin_name[] = test_plugin;  /* also the name of the
 feature added to .FEATURES */

 /* $(equal astring1,astring2) returns blank, $(equal
 astring1,astring1) returns astring1
  * The purpose of this function is to allow equality tests in $(if)
 functions e.g. $(if $(equal $(X),$(Y)),something,else)
  * */
 static char *function_equal(char *o, char **argv, const char *funcname)
 {
  if (argv[0]  argv[1]  strcmp(argv[0],argv[1]) == 0)
    {
          o = variable_buffer_output (o, argv[0], strlen (argv[0]));
    }
  else
    {
          o = variable_buffer_output (o, , 0);
    }

 return o;
 }

 padded_plugin_entry gnumake_plugin_function_table[] = {
        {equal, 2, 2, 1, function_equal}, /* 2 args, expand them first */
             {, 0, 0, 0, (void *)0}
 };

 /* end -- */



 So a plugin is fairly easy to write and it's not hard to build either.

 I have tried to set it all up with autoconf and so on but I'm not an
 expert at that.  There is also nothing stopping you from loading the
 same plugin twice at the moment and it needs more tests and some
 documentation so I realise that this patch is not right yet.

 There is one set of modifications to main.c to add calls to
 load_plugin() which is in a new file, manage-plugins.c, and there is a
 new commandline option (--plugin=) 

Re: Patch to allow make to load plugins that add new functions.

[David Boyce]

A few years ago I suggested a plugin architecture much like this (but
I didn't supply a patch - crucial difference), to allow a plugin to
make the up-to-date determination, replacing the hardwired timestamp
system. That could in theory be quite useful as it would allow an MD5
or similar signature to be used without major mods to make internals
which is what's defeated such attempts in the past. Any idea how hard
it would be to handle this with a plugin?

On an unrelated note, how would you anticipate plugins being loaded in
real life? I think expecting  people tp type make
--plugin=tests/plugins/test-plugin.so ... is too unwieldy for daily
use. Environment variable? Auto-discovery of plugins in a conventional
location? Is it possible a .PLUGIN special target could be embedded in
a makefile that requires a given plugin, or would that be too late?
Maybe a MAKE_PLUGIN_PATH env var?

-David Boyce

On Thu, Apr 5, 2012 at 1:12 PM, Tim Murphy tnmur...@gmail.com wrote:
 Hi,

 On 5 April 2012 20:52, Lawrence Ibarria libar...@nvidia.com wrote:
 I like this idea quite a bit.

 I see this as still work in progress to define what type of functions the 
 plugins can have.
 Maybe they can even create or change make variables.

 At this point, plugins are naughty and are kind of able to access
 anything in make that they have the prototype for (using -rdynamic to
 make all the symbols available).  This is definitely dodgy since
 there's nothing stopping a plugin from trying to access something that
 might be intended as fairly volatile within make.

 In the case of variable_buffer_output, I'd suggest that each plugin has an 
 'initialization function'. This is a handshake where make provides the 
 plugin with function pointers (avoids the extern function) and the plugin 
 can provide its name, versions, and the function table. That simplifies the 
 code that right now tries to get every single variable from the dl_handle.

 -- Lawrence

 Ok, good idea - I'll add something to look for an initialiser and call
 it and as you say, pass it a table. Returning values is critical and I
 think that looking up variable values at the least is also probably
 important.  Another thing I've wanted for ages is to be able to know
 if some target was declared already - I want plugins to have the
 ability to look at make's target database if possible although not
 necessarily modify it.

 Regards,

 Tim


 -Original Message-
 From: bug-make-bounces+libarria=nvidia@gnu.org 
 [mailto:bug-make-bounces+libarria=nvidia@gnu.org] On Behalf Of Tim Murphy
 Sent: Thursday, April 05, 2012 2:51 AM
 To: bug-make@gnu.org
 Subject: Patch to allow make to load plugins that add new functions.

 Hi,

 I am between jobs which made me realise that I am absolutely free to
 contribute to make for about 10 days :-)

 The one thing I have wanted the most and the longest is a way to add
 new functions without having to rebuild and look after a custom
 version of make.  Essentially this should allow people to extend make
 without necessarily having to maintain a custom version of it.

 So here is an attempt at a plugin system for your interest. Currently
 it ought to work on systems with dlopen.  LoadLibrary on windows
 should do the job admirably but I haven't had time to have a go at
 that yet.  So the feature is optional (configure --with-plugins).  I
 am looking for feedback indicating interest or disagreement or
 whatever and mostly I just want to put some code into the open so that
 it's there.

 My test plugin adds the $(equal ($X),$(Y)) function which returns $(X)
 if the variables X and Y match as strings and returns the empty string
 if not. I have often wanted this for use with $(if) and have a
 horrible and inefficient macro for doing it now but it's something I
 feel really ought to be built in.

 Here's how you tell make to use a plugin:

 ./make --plugin=tests/plugins/test-plugin.so -f tests/plugins/plugintest1.mk


 Here's the code for this simple test plugin:

 #include plugin.h
 #include string.h

 /* The next line is a cheat  to save from having to include all the
 make headers and possibly end up linking to all sorts of make object
 files which defeats the purpose.
   Don't like it and  will have to think of something better: */
 extern char *variable_buffer_output (char *ptr, const char *string,
 unsigned int length);

 int gnumake_plugin_api_version = 1; /* what api the plugin expects
 make to provide - just a way for make to know if it's loading a plugin
 that should work  */

 int gnumake_plugin_major_version = 1; /* The version of this plugin -
 might be useful for problem reports */
 char gnumake_plugin_name[] = test_plugin;  /* also the name of the
 feature added to .FEATURES */

 /* $(equal astring1,astring2) returns blank, $(equal
 astring1,astring1) returns astring1
  * The purpose of this function is to allow equality tests in $(if)
 functions e.g. $(if $(equal $(X),$(Y)),something,else)
  * */
 static char 

Re: Patch to allow make to load plugins that add new functions.

[Tim Murphy]

Hi,

On 5 April 2012 23:12, Paul Smith psm...@gnu.org wrote:
 Hi Tim;

 Before going too much further note that I've got a semi-implemented
 load operator in my source already, which fulfills a similar function
 except in a less sophisticated way: it just calls a function in the
 loaded object after loading and that function can do whatever it wants.

 I can send along a patch if you're interested in checking it out.  I
 have some regression tests added, documentation, autoconf, etc.

 Boris and I discussed this on the alpha list a number of months ago and
 his position was that we should not try to restrict things.  His
 suggestion was to follow the GCC model and just let people do what they
 wanted to.  The downside of this is that we wouldn't be defining a big
 stable API: we would reserve the right to change things internally with
 every release, which would potentially break loadable objects.

I see the value in a plugin system as being that I don't have to
recompile the plugins for every version of make. In a way it's tending
towards why bother if you did have to do that.  The idea that you're
describing probably wouldn't cause disruptions to most plugins but
there's no way of knowing except to test all plugins with all versions
of make anyhow to find out.  It would be a maintenance burden that's
quite similar to maintaining a custom make.

You also have to think about where in makes startup these things
happen. Some stuff needs to be done early, some later - if your plugin
loader needs a fully initialised make process (because it has to read
a makefile to do load) then although you might think it is infinitely
flexible, it does come with limits.

I have to admit to a secondary agenda in that I'm thinking about tools
like Electric Accelerator which mimic gnu make - I'd like them to be
able to implement plugins too so that my build system can be
accelerated ($) or free :-) depending on the users choice.  This
is why I was thinking of it being quite defined and with restricted
access and using versions so that it's clear what needs to be
supported and what's offered.

 There are some outstanding issues: the first one is that I'd like the
 load operation to behave like the include operation, in that make
 would try to rebuild the target of the load and, if it were rebuilt,
 re-exec itself to reload it.  That's not implemented yet.

I think this is cool in a way but sort of overkill - I would imagine
building plugins as little as possible.  You see, if this is the first
time you've built the plugin then you can't have tested it - so why
are you running a build with an untested plugin where you don't know
if it meets the contract?  I know that in practice it will just be
getting built because building from source is a common thing even for
code that has been built and tested in exactly the same form 100x
before, but I see this as being a capability that is beside the point
for my use cases.  I will find the discussion if it's archived
somewhere and try to understand the case for it - my perspective might
be quite limited.

 Second, the way in which the objects are found for loading.  Do we just
 use the default dlopen() process?  Or do something more (or less!)
 sophisticated?

I don't know but dlopen and windows LoadLibrary are similar enough
that the same general methodology will work.  I'm not up enough on
alternatives to know if there's anything better.

 Finally, as you've noted in order for this to be useful generally we
 need to start publishing some header files.  What goes into them and how
 they get published is something of an open question.  It's possible that
 this will require a significant reworking of the current make header
 files (which would be OK with me since they're pretty messy currently).
 We could choose the public interface through what structures and
 functions we put into the publicly-facing header.

I have thought about this and it's harder if everything's to become
available for use and easier if one is defining some subset of
exported stuff.

A totally different integration mechanism might involve being able to
embed GNU make into other programs and talk to it and I see that as a
very interesting possibility which would become much easier with the
same kind of header reorg that a plugin mechanism with full access
might need.

 The most obvious interface is define a new function, but others could
 also be useful: eval, set variables, etc.

Yup.  I wondered about how make macros are (ha ha mostly) functional
so I was thinking about starting with reading variables, targets,
etc because it's completely uncontroversial and was going to leave
setting or any other side effect for a while longer since it's the
kind of area where I would be most likely to make a mess.

Regards,

Tim


-- 
You could help some brave and decent people to have access to
uncensored news by making a donation at:

http://www.thezimbabwean.co.uk/friends/

___

Re: Patch to allow make to load plugins that add new functions.

[Tim Murphy]

Hi,

On 5 April 2012 23:27, David Boyce david.s.bo...@gmail.com wrote:
 A few years ago I suggested a plugin architecture much like this (but
 I didn't supply a patch - crucial difference), to allow a plugin to
 make the up-to-date determination, replacing the hardwired timestamp
 system. That could in theory be quite useful as it would allow an MD5
 or similar signature to be used without major mods to make internals
 which is what's defeated such attempts in the past. Any idea how hard
 it would be to handle this with a plugin?

This is a serious issue indeed - I've wanted md5 timestamps for ages.
I think this is much more complicated than adding functions though.

I don't know how make evaluates prerequisites in detail and if I
naively stuck an md5 check in place of a call to stat() for a check
then the file might get read and stamped many times repeatedly which
could be very slow.

It also might be great until I was doing a check on a 650MB iso or
something like that.

So for a feature like this I'd be looking at heuristics and
complicated tricks to avoid the pitfalls and I'm actually fairly sure
it would be easier to do this without plugins first and then
generalise it afterwards to allow people to choose different checksum
algorithms and to set the tradeoffs about what kinds of files to use
each method on etc - but I'd do that after I had written a hardcoded
version and understood all the problems.

 On an unrelated note, how would you anticipate plugins being loaded in
 real life? I think expecting  people tp type make
 --plugin=tests/plugins/test-plugin.so ... is too unwieldy for daily
 use. Environment variable? Auto-discovery of plugins in a conventional
 location? Is it possible a .PLUGIN special target could be embedded in
 a makefile that requires a given plugin, or would that be too late?
 Maybe a MAKE_PLUGIN_PATH env var?

:-) that's a good point. I tend to use it from a script and I hadn't
thought about the convenience of it. Good suggestion for an
environment variable.

I am initialising the plugins quite early but I don't think that it's
critical to do so - hence adding syntax is quite possible.  I was
trying to avoid doing that because it's always so controversial but
since Paul already has a plan for syntax...

Regards,

Tim

 -David Boyce

 On Thu, Apr 5, 2012 at 1:12 PM, Tim Murphy tnmur...@gmail.com wrote:
 Hi,

 On 5 April 2012 20:52, Lawrence Ibarria libar...@nvidia.com wrote:
 I like this idea quite a bit.

 I see this as still work in progress to define what type of functions the 
 plugins can have.
 Maybe they can even create or change make variables.

 At this point, plugins are naughty and are kind of able to access
 anything in make that they have the prototype for (using -rdynamic to
 make all the symbols available).  This is definitely dodgy since
 there's nothing stopping a plugin from trying to access something that
 might be intended as fairly volatile within make.

 In the case of variable_buffer_output, I'd suggest that each plugin has an 
 'initialization function'. This is a handshake where make provides the 
 plugin with function pointers (avoids the extern function) and the plugin 
 can provide its name, versions, and the function table. That simplifies the 
 code that right now tries to get every single variable from the dl_handle.

 -- Lawrence

 Ok, good idea - I'll add something to look for an initialiser and call
 it and as you say, pass it a table. Returning values is critical and I
 think that looking up variable values at the least is also probably
 important.  Another thing I've wanted for ages is to be able to know
 if some target was declared already - I want plugins to have the
 ability to look at make's target database if possible although not
 necessarily modify it.

 Regards,

 Tim


 -Original Message-
 From: bug-make-bounces+libarria=nvidia@gnu.org 
 [mailto:bug-make-bounces+libarria=nvidia@gnu.org] On Behalf Of Tim 
 Murphy
 Sent: Thursday, April 05, 2012 2:51 AM
 To: bug-make@gnu.org
 Subject: Patch to allow make to load plugins that add new functions.

 Hi,

 I am between jobs which made me realise that I am absolutely free to
 contribute to make for about 10 days :-)

 The one thing I have wanted the most and the longest is a way to add
 new functions without having to rebuild and look after a custom
 version of make.  Essentially this should allow people to extend make
 without necessarily having to maintain a custom version of it.

 So here is an attempt at a plugin system for your interest. Currently
 it ought to work on systems with dlopen.  LoadLibrary on windows
 should do the job admirably but I haven't had time to have a go at
 that yet.  So the feature is optional (configure --with-plugins).  I
 am looking for feedback indicating interest or disagreement or
 whatever and mostly I just want to put some code into the open so that
 it's there.

 My test plugin adds the $(equal ($X),$(Y)) function which returns $(X)
 if 

Re: Patch to allow make to load plugins that add new functions.

[Paul Smith]

On Thu, 2012-04-05 at 18:27 -0400, David Boyce wrote:
 A few years ago I suggested a plugin architecture much like this (but
 I didn't supply a patch - crucial difference), to allow a plugin to
 make the up-to-date determination, replacing the hardwired timestamp
 system. [...] Any idea how hard it would be to handle this with a plugin?

As make is currently implemented it's not really possible.  That's not
to say it COULDN'T be and I do think this is a very useful goal, and I
certainly want to be sure whatever model we come up with can support
that, once it's possible.  But before we can achieve it there will need
to be some cleanup in the algorithms make uses: we need to abstract
these things.  Right now they're just kind of scattered inside other
functions (there is one main function but that's not the only place they
are handled).

 On an unrelated note, how would you anticipate plugins being loaded in
 real life?

The model I have today is a new processor definition load, which is
similar to include except that it loads a dynamic object.  The object
name can optionally have a function associated with it (currently in
parens, like:

load myplugin.so(init_myplugin)

) which, if given, will be invoked when the object is loaded.  If no
name is given, there's a default symbol name which is invoked.

Multiple objects can be loaded in one statement.




___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: Patch to allow make to load plugins that add new functions.

[Paul Smith]

On Thu, 2012-04-05 at 23:59 +0100, Tim Murphy wrote:
 I see the value in a plugin system as being that I don't have to
 recompile the plugins for every version of make. In a way it's tending
 towards why bother if you did have to do that.

Well, this kind of combines with my other issue regarding automatic
remaking of objects :-)

The idea is that you wouldn't have a predefined set of make plugin
OBJECTS that you distributed.  Instead you'd keep the SOURCE CODE for
your plugins for your build environment right there in your source tree,
and build them as part of your build process.  If they are automatically
recompiled when out of date, this would be seamless.

I have a hard time envisioning massive, complex plugins: most likely
they'll be small, straightforward functions that perform targeted
operations more efficiently than make (or Guile) can do them.  Such
things would only take a couple of seconds to build and would be
unnoticeable in any serious build environment.

 You also have to think about where in makes startup these things
 happen. Some stuff needs to be done early, some later - if your plugin
 loader needs a fully initialised make process (because it has to read
 a makefile to do load) then although you might think it is infinitely
 flexible, it does come with limits.

Heh, I didn't say infinitely flexible; that's too much to ask.  We
have the --eval command line option already.  I'm open to adding an
environment variable if someone can describe a powerful use-case.

 I have to admit to a secondary agenda in that I'm thinking about tools
 like Electric Accelerator which mimic gnu make - I'd like them to be
 able to implement plugins too so that my build system can be
 accelerated ($) or free :-) depending on the users choice.  This
 is why I was thinking of it being quite defined and with restricted
 access and using versions so that it's clear what needs to be
 supported and what's offered.

Hm.  I'd need to understand better what exactly would be involved here.
What kind of plugins would be used for this and what would they do?  I'm
not that jazzed about designing in a vacuum, especially just to enable
proprietary software to work better with GNU make :-).

What I really DON'T want to do is provide a locked down interface, which
then turns out to be too restrictive to be useful and we're constantly
fielding requests to open this or that, and having to worry about
backward compatibility, etc.  I'd prefer to leave things open, at least
for a while until we get some feedback/experience.  Yes, this does shift
the burden away from me and onto the plugin writers (where it belongs of
course!! ;-)), but it's not like there are so many versions of make
floating around that this is that onerous.

I was thinking of marking this feature as experimental in the first
release (in the documentation), just to be more clear on expectations.



___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make