Posted on stackoverflow and did not get a useful answer. I seem unable to fix my space in filenames issue using switches like -print0 for gnu-find and -0 for gnu-parallel, gnu-xargs in this scenario as is usually recommended.
I succeeded in combining find, parallel in pipe mode and xargs to run commands in parallel in "blocks" for 100k+ files. I use echo and ls in the examples below but I plan to use my own python command. Note that I want to run each command instance on more than one file due to overhead in starting my program hence the use of parallel in --pipe mode and --block etc. The command find ./dirNames/ -type f | parallel --pipe --block 100 -j4 --round-robin \ "echo \"Start *****\"; cat ; echo \"Done *****\"" results in Start ***** ./dirNames/bbbbbbbbbbbbbbbb ./dirNames/dddddddddddddddddddd ./dirNames/aaaaaaaaaaaaaaaa Done ***** Start ***** ./dirNames/cccccccc cccccccc ./dirNames/eeeeeeeeeeeeeeeeeeee Done ***** *as desired.* gnu-echo is run twice, in one instance it is run with 3 files and in the other instance with 2 files. If I try this with xargs and ls I run into the classic space in filename problem ... find dirNames/ -type f | parallel --pipe --block 40 -j4 --round-robin \ "echo \"Start *****\"; xargs ls -l ; echo \"Done *****\"" Resulting in this Start ***** -rw-rw-r-- 1 robert robert 0 Jun 24 10:10 dirNames/bbbbbbbbbbbbbbbb -rw-rw-r-- 1 robert robert 0 Jun 25 16:11 dirNames/eeeeeeeeeeeeeeeeeeee Done ***** Start ***** -rw-rw-r-- 1 robert robert 0 Jun 24 10:10 dirNames/aaaaaaaaaaaaaaaa Done ***** Start ***** -rw-rw-r-- 1 robert robert 0 Jun 25 16:11 dirNames/dddddddddddddddddddd Done ***** Start ***** Done ***** ls: cannot access 'dirNames/cccccccc': No such file or directory ls: cannot access 'cccccccc': No such file or directory which in this scenario I seem unable to fix using switches like -print0 for find and -0 for parallel and xargs as is usually recommended for this problem. parallel seems confused by the output of find with -print0. Please advise as I have truly run out of ideas :(
