Re: Dynamic adjustments of build dependencies for the number of available processors

2015-01-05 Thread Paul Smith
On Tue, 2015-01-06 at 07:05 +0100, SF Markus Elfring wrote: I wrote some blog posts about eval and other metaprogramming techniques in make that you might find interesting: http://make.mad-scientist.net/category/metaprogramming/ I find this article also useful and helpful for my

[bug #43936] Completion of error handling

2015-01-05 Thread Markus Elfring
URL: http://savannah.gnu.org/bugs/?43936 Summary: Completion of error handling Project: make Submitted by: elfring Submitted on: Mo 05 Jan 2015 10:10:58 CET Severity: 3 - Normal Item Group: Bug

Aborting script evaluation for targets without parallel build support

2015-01-05 Thread SF Markus Elfring
Hello, I would like to abort the evaluation of a make script if data processing was requested for a target which does not support a parallel software build process so far. I do not want to use the .NOTPARALLEL pseudo-target for some of my applications when I can call a make function like error on

Re: Cooperation with distributed job processing systems

2015-01-05 Thread Paul Smith
On Sat, 2015-01-03 at 23:30 +0100, SF Markus Elfring wrote: There are programming interfaces available which provide support for submitting jobs to bigger and more powerful computer systems. I imagine that the software jobserver could be extended for the convenient reuse of such APIs.

Re: Dynamic adjustments of build dependencies for the number of available processors

2015-01-05 Thread Paul Smith
On Sat, 2015-01-03 at 20:23 +0100, SF Markus Elfring wrote: I would like to use all processor cores for a software build. So I try to reuse a corresponding system setting by a command like getconf _NPROCESSORS_ONLN in recipes for a make file. A bit of build preparation needs to be performed

Re: Aborting script evaluation for targets without parallel build support

2015-01-05 Thread Paul Smith
On Mon, 2015-01-05 at 12:42 +0100, SF Markus Elfring wrote: 1. The list of targets which can be specified as command line parameters is not provided by the tool make 4.1-2.1 in the variable MAKEFLAGS. How should the target name be checked then? The goals provided on the command line are

Fwd: Dynamic adjustments of build dependencies for the number of available processors

2015-01-05 Thread Tim Murphy
-- Forwarded message -- From: Tim Murphy tnmur...@gmail.com Date: 5 January 2015 at 17:20 Subject: Re: Dynamic adjustments of build dependencies for the number of available processors To: SF Markus Elfring elfr...@users.sourceforge.net Hi On 5 January 2015 at 17:03, SF Markus

Re: Dynamic adjustments of build dependencies for the number of available processors

2015-01-05 Thread SF Markus Elfring
So I need to be flexible with the specification of input and output parameters for these programs. But I see some software development challenges for this use case. I'm sorry but I don't really understand what you're trying to do. How do you think about to share any more software

Re: Cooperation with distributed job processing systems

2015-01-05 Thread SF Markus Elfring
Jobserver is a method of communicating how many jobs make thinks are running between different instances (parent/child) of the make program itself, so it knows that no more than N jobs are invoked between all instances. How do you think about to delegate such an implementation detail to a

Re: Dynamic adjustments of build dependencies for the number of available processors

2015-01-05 Thread SF Markus Elfring
I wrote some blog posts about eval and other metaprogramming techniques in make that you might find interesting: http://make.mad-scientist.net/category/metaprogramming/ I find this article also useful and helpful for my software development tasks. You mention a construct define … endef

Re: Dynamic adjustments of build dependencies for the number of available processors

2015-01-05 Thread SF Markus Elfring
I assume that you want something different from this but you need to explain better, sorry :-) I hope that an other wording will be clearer. Can make rules be extended on demand while a build script is evaluated? How much can a command like getconf _NPROCESSORS_ONLN influence rule evaluation

Re: Dynamic adjustments of build dependencies for the number of available processors

2015-01-05 Thread Paul Smith
On Mon, 2015-01-05 at 18:03 +0100, SF Markus Elfring wrote: I'm sorry but I don't really understand what you're trying to do. How do you think about to share any more software development experiences for the application of the command getconf _NPROCESSORS_ONLN together with make tools?

Re: Dynamic adjustments of build dependencies for the number of available processors

2015-01-05 Thread Tim Murphy
On 5 January 2015 at 18:13, SF Markus Elfring elfr...@users.sourceforge.net wrote: I assume that you want something different from this but you need to explain better, sorry :-) I hope that an other wording will be clearer. Can make rules be extended on demand while a build script is

Re: Dynamic adjustments of build dependencies for the number of available processors

2015-01-05 Thread SF Markus Elfring
It seems like you want to automatically expand or contract the number of jobs that make will run in parallel, Yes, exactly. I can not use the parameter -j directly for the start of the main make process in my case so far. based on some processing of the recipe. One of the variables (or

Re: Cooperation with distributed job processing systems

2015-01-05 Thread Paul Smith
On Mon, 2015-01-05 at 18:24 +0100, SF Markus Elfring wrote: Jobserver is a method of communicating how many jobs make thinks are running between different instances (parent/child) of the make program itself, so it knows that no more than N jobs are invoked between all instances. How do

Re: Dynamic adjustments of build dependencies for the number of available processors

2015-01-05 Thread SF Markus Elfring
So you have in your toolbox $(shell) and $(eval). I am not familiar enough with the second make function. http://www.gnu.org/software/make/manual/html_node/Eval-Function.html $(eval) lets you generate rules dynamically. Does it really support the generation of completely new build rules

Re: Dynamic adjustments of build dependencies for the number of available processors

2015-01-05 Thread Howard Chu
SF Markus Elfring wrote: It seems like you want to automatically expand or contract the number of jobs that make will run in parallel, Yes, exactly. I can not use the parameter -j directly for the start of the main make process in my case so far. based on some processing of the recipe.

Re: Cooperation with distributed job processing systems

2015-01-05 Thread SF Markus Elfring
There are two completely separate things: one thing is make deciding how many jobs can be run in parallel and when more jobs can be started. Thanks for your explanation. That's controlled by the jobserver and currently there's no way to replace the built-in jobserver implementation with

Re: Dynamic adjustments of build dependencies for the number of available processors

2015-01-05 Thread Paul Smith
On Mon, 2015-01-05 at 20:06 +0100, SF Markus Elfring wrote: So you have in your toolbox $(shell) and $(eval). I am not familiar enough with the second make function. http://www.gnu.org/software/make/manual/html_node/Eval-Function.html I wrote some blog posts about eval and other

Re: Dynamic adjustments of build dependencies for the number of available processors

2015-01-05 Thread SF Markus Elfring
I guess I don't really see what the purpose of this is. If you have 2 CPUs you get targets MYTARGET_0 and MYTARGET_1. If you have 4 CPUs you get targets MYTARGET_0, MYTARGET_1, MYTARGET_2, MYTARGET_3. So... then what? I need to find some source files before I can start further processes

Re: Dynamic adjustments of build dependencies for the number of available processors

2015-01-05 Thread SF Markus Elfring
The corresponding processes should be efficiently started for parallel background execution after a serial data preparation was finished. This makes very little sense. This issue depends on work flow ordering. In a properly written Makefile, your serial preparation step will remain