On Sat, Jan 30, 2010 at 10:36:56AM +0100, Sylvain Pointeau wrote:
> echo "begin transaction" >> update.sql
>
> sqlite3 -separator $'\t' sample.db 'select rowid, item from foo;' |
> while read rowid item ; do
> status=$(long_running_process "${item}" )
> echo "update foo set status=${status} where rowid=${rowid} ;" >> update.sql
> done
>
> echo "commit transaction" >> update.sql
>
> sqlite3 sample.db < update.sql
More aesthetic (since it seems to matter :)
(
echo "begin transaction"
sqlite3 -separator $'\t' sample.db 'select rowid, item from foo;' |
while read rowid item ; do
status=$(long_running_process "${item}" )
echo "update foo set status=${status} where rowid=${rowid} ;"
done
echo "commit transaction"
) | sqlite3 sample.db
Now to parallelize this:
function par_updates {
typeset -i n
n=$1
shift
(
trap "((n++))" CHLD
echo "begin transaction"
sqlite3 -separator $'\t' sample.db 'select rowid, item from foo;' |
while read rowid item
do
while ((n == 0))
do
sleep 1
done
(echo "update foo set status=$("$@") where rowid=$rowid;") &
((n--))
done
echo "commit transaction"
) | sqlite3 sample.db
}
You should run this like so:
par_updates 20 long_running_program [arguments]
It should run up to 20 processes to generate the updates. And it should
batch all updates into one transaction.
I've not tested this, but it should work with recent versions of ksh93
and bash. (Older versions of ksh93 might not handle the SIGCHLD trap
correctly, IIRC.)
Nico
--
_______________________________________________
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
