(can't reply to thread from yesterday as i'm doing this from webmail and have no access to what's already been downloaded to my home system.)
asked about weird parallelization issues in make structure in the code base here: https://github.com/Juniper/JP4Agent and I wanted to demo a really trivial example to show when this issue would pop up -- if people want to improve this example, that would be ducky; I will probably write it up and post it on my wiki as a lesson to others. mk1: ==== first, a simple example of targets that can be processed in parallel: COUNTERS = count1 count2 .PHONY: top $(COUNTERS) top: $(COUNTERS) count1: for i in $$(seq 5) ; do \ echo "$@ $$i" ; \ sleep 1 ; \ done count2: for i in $$(seq 5) ; do \ echo "$@ $$i" ; \ sleep 1 ; \ done now in the above, there is no dependency relation between count1 and count2, so they *can* run in parallel. if one does not ask for that, they won't be: $ make -f mk1 for i in $(seq 5) ; do \ echo "count1 $i" ; \ sleep 1 ; \ done count1 1 count1 2 count1 3 count1 4 count1 5 for i in $(seq 5) ; do \ echo "count2 $i" ; \ sleep 1 ; \ done count2 1 count2 2 count2 3 count2 4 count2 5 $ but if one *does* ask for parallelism, one gets it: t$ make -f mk1 -j 2 for i in $(seq 5) ; do \ echo "count1 $i" ; \ sleep 1 ; \ done for i in $(seq 5) ; do \ echo "count2 $i" ; \ sleep 1 ; \ done count1 1 count2 1 count1 2 count2 2 count1 3 count2 3 count1 4 count2 4 count1 5 count2 5 $ so far, so good, yes? and this is because there is no relationship between targets count1 and count2 in this case. moving on … mk2: ==== COUNTERS = count1 count2 .PHONY: top $(COUNTERS) count top: $(COUNTERS) count1 count2: count count: for i in $$(seq 5) ; do \ echo "$$i" ; \ sleep 1 ; \ done this situation is clearly different in that count1 and count2 are now "related" via their mutual dependency on "count" so that (AIUI) that target should be processed only once, no matter what: $ make -f mk2 for i in $(seq 5) ; do \ echo "$i" ; \ sleep 1 ; \ done 1 2 3 4 5 $ t$ make -f mk2 -j 2 <-- even with parallelism, only once for i in $(seq 5) ; do \ echo "$i" ; \ sleep 1 ; \ done 1 2 3 4 5 $ so this seems fine, but that's not what appears to be happening in that JP4agent Makefile. rather than setting up proper dependencies, the later rules simply call make manually for subdirectories, as in: INSTALL_COMPONENTS = $(COMPONENTS:%=install-%) .PHONY: install $(INSTALL_COMPONENTS) install: $(INSTALL_COMPONENTS) $(INSTALL_COMPONENTS): echo $(DESTDIR) $(prefix) $(sysconfdir) make install -C $(@:install-%=%) which means any protection for parallelism is totally lost, correct? I believe I can demonstrate that with … mk3: ==== COUNTERS = count1 count2 .PHONY: top $(COUNTERS) count top: $(COUNTERS) count1 count2: $(MAKE) -f mk3 count <--- the important bit count: for i in $$(seq 5) ; do \ echo "$$i" ; \ sleep 1 ; \ done so, in the above, rather than defining proper dependencies, by manually calling the make command in a rule (is that the right way to do that?), first, if i invoke this makefile normally, it runs targets serially: $ make -f mk3 make -f mk3 count make[1]: Entering directory '/home/rday/t' for i in $(seq 5) ; do \ echo "$i" ; \ sleep 1 ; \ done 1 2 3 4 5 make[1]: Leaving directory '/home/rday/t' make -f mk3 count make[1]: Entering directory '/home/rday/t' for i in $(seq 5) ; do \ echo "$i" ; \ sleep 1 ; \ done 1 2 3 4 5 make[1]: Leaving directory '/home/rday/t' but if ask for parallelism, then bad things happen: $ make -f mk3 -j 2 make -f mk3 count make -f mk3 count make[1]: Entering directory '/home/rday/t' for i in $(seq 5) ; do \ echo "$i" ; \ sleep 1 ; \ done make[1]: Entering directory '/home/rday/t' for i in $(seq 5) ; do \ echo "$i" ; \ sleep 1 ; \ done 1 1 2 2 3 3 4 4 5 5 make[1]: Leaving directory '/home/rday/t' make[1]: Leaving directory '/home/rday/t' i submit that this is what is happening based on that top-level Makefile … any thoughts on whether my explanation above would seem to match what is happening there? is there a better/cleaner way to describe this? rday
