Re: Makefile help
Charles Curley writes: > It probably is, but I would not use make. A script which used find > would do as well without make's idiosyncrasies. Pretty much any > scripting language should do the job, so use one you are familiar with. Since he said there are a lot of files to convert I thought make might be a good choice for this after all. It would allow stopping the conversion and starting again, skipping already converted files with little overhead. Then again, any program could easily check if the target file exists too.
Re: Makefile help
On Fri, 23 Jul 2021 16:10:10 +0200 Grzesiek wrote: > Is it possible to do it using make in a fully automatic way? What I > mean is that make should find all sub-directories and files in > source_dir by itself. Note that the sub-directories level may be > grater than 1, no symlinks allowed. It probably is, but I would not use make. A script which used find would do as well without make's idiosyncrasies. Pretty much any scripting language should do the job, so use one you are familiar with. > > I do not have experience in writing Makefiles. Appropriate Makefile > examples appreciated. Another reason for you to avoid make. -- Does anybody read signatures any more? https://charlescurley.com https://charlescurley.com/blog/
Re: Makefile help
* 2021-07-23 16:10:10+0200, Grzesiek wrote: > Is it possible to do it using make in a fully automatic way? What I > mean is that make should find all sub-directories and files in > source_dir by itself. "make" doesn't find files. You can use "find" in a Makefile to find all necessary files and then a Makefile rule to do conversion if the source is newer than the destination. If you copy the following example remember to use tab character for indentation in the rules. src = $(shell find source_dir -type f -name '*.src' -print) dst = $(patsubst source_dir/%.src,dest_dir/%.dst,$(src)) all: $(dst) .PHONY: all dest_dir/%.dst: source_dir/%.src @mkdir -p $$(dirname $@) CMD $< $@ -- /// Teemu Likonen - .-.. https://www.iki.fi/tlikonen/ // OpenPGP: 4E1055DC84E9DFF613D78557719D69D324539450 signature.asc Description: PGP signature
Re: Makefile help
On Fri, Jul 23, 2021 at 04:10:10PM +0200, Grzesiek wrote:
> I have two directories: source_dir and dest_dir. In the directory source_dir
> I have number of sub-directories and files. My goal is:
>
> 1.Recreate the sub-directory structure of the source_dir inside of the
> dest_dir
>
> 2. Each file found in source_dir should be converted to a new format using
> some command CMD in the following way:
> CMD source_dir//.src dest_dir// name>.dst
> Is it possible to do it using make
Not really. It's a job for a shell script or some other program written
in a scripting language.
On Fri, Jul 23, 2021 at 04:29:24PM +0200, Grzesiek wrote:
> The number of files is rather large so make -j8 could save lot of time.
Ah, the good old X-Y strikes again. OK, at least now we know the real
goal -- you want to parallelize the conversions.
The obvious way to do that is with GNU xargs -0 -P. So, we're going to
generate a NUL-delimited stream of filenames to be processed, and feed
that to xargs -0 -P 8 -n 1 something.
Now that we know the middle piece, we only have to write the beginning
piece (the thing that generates the stream of NUL-delimited filenames),
and the ending piece (the "something" that xargs runs to perform each
conversion).
The beginning is simple: we'll be using find -print0 to generate the stream
of filenames. find source_dir -type f -print0
You can add additional restrictions to this if needed (e.g. -iname '*.wav').
The ending is basically what you specified above, except that we need to
wrap it in a miniature shell script to generate the destination pathname
from the source pathname. We also need to create the directory part of
the destination pathname, because presumably CMD doesn't do that for us.
So,
#!/bin/sh
find source_dir -type f -print0 |
xargs -0 -P 8 -n 1 sh -c '
dest=dest_dir/${1#*/}
dest=${dest%.src}.dst
dir=${dest%/*}
mkdir -p "$dir"
CMD "$1" "$dest"
' x
And, testing it:
unicorn:~$ printf %s\\0 {1..12} | xargs -0 -P 8 -n 1 sh -c 'echo "$$ processing
$1 at $(date +%H:%M:%S)"; sleep 3' x
56062 processing 1 at 10:56:52
56063 processing 2 at 10:56:52
56065 processing 3 at 10:56:52
56067 processing 4 at 10:56:52
56074 processing 6 at 10:56:52
56072 processing 5 at 10:56:52
56076 processing 7 at 10:56:52
56079 processing 8 at 10:56:52
56086 processing 9 at 10:56:55
56089 processing 11 at 10:56:55
56091 processing 12 at 10:56:55
56087 processing 10 at 10:56:55
The first 8 ran simultaneously, then after a 3 second delay, the last 4
ran simultaneously. Looks good to me. Obviously I didn't test the string
manipulations that generate the destination filename and subdirectory,
but I'm sure you can handle it from here.
Re: Makefile help
Grzesiek wrote: > > 1.Recreate the sub-directory structure of the source_dir inside of the > dest_dir > > 2. Each file found in source_dir should be converted to a new format using > some command CMD in the following way: > CMD source_dir//.src dest_dir// name>.dst It's not common practice to do so, and think you might have missed something. But anyway, lookout for function: "mkshadow" in package/tool: shtool Maybe it's doing what you want. Best regards, Klaus. -- Klaus Singvogel GnuPG-Key-ID: 1024R/5068792D 1994-06-27
Re: Makefile help
On 7/23/21 4:25 PM, john doe wrote: On 7/23/2021 4:10 PM, Grzesiek wrote: I do not have experience in writing Makefiles. Appropriate Makefile examples appreciated. Why do you need to use make? As far as I understand your question, you are not concernde about the 'mtime'. The number of files is rather large so make -j8 could save lot of time.
Re: Makefile help
On 7/23/2021 4:10 PM, Grzesiek wrote: I do not have experience in writing Makefiles. Appropriate Makefile examples appreciated. Why do you need to use make? As far as I understand your question, you are not concernde about the 'mtime'. -- John Doe
Makefile help
Hi there! I have two directories: source_dir and dest_dir. In the directory source_dir I have number of sub-directories and files. My goal is: 1.Recreate the sub-directory structure of the source_dir inside of the dest_dir 2. Each file found in source_dir should be converted to a new format using some command CMD in the following way: CMD source_dir//.src dest_dir/file>/.dst So, if source_dir contains: source_dir +sub1 | +file1.src | +file2.src | +file3.src +file4.src In the dest_dir I would like to obtain: dest_dir +sub1 | +file1.dst | +file2.dst | +file3.dst +file4.dst Is it possible to do it using make in a fully automatic way? What I mean is that make should find all sub-directories and files in source_dir by itself. Note that the sub-directories level may be grater than 1, no symlinks allowed. I do not have experience in writing Makefiles. Appropriate Makefile examples appreciated. -- Thanks in advance for any help. Greg
