Your code needs to explicitly captures all variables you want available in the
parallel for loop:
# Original
proc parallelCoverURL(db, dbc: DbConn) =
init(Weave)
var ids: seq[uint32] = db.getAllRows(sql"SELECT tconst FROM title_basics
WHERE tconst NOT IN (SELECT id FROM
covers.lolz)").mapIt(parseuint(it[0]).uint32)
dbc.exec(sql"CREATE TABLE IF NOT EXISTS lolz (id INT PRIMARY KEY, url
VARCHAR)")
let idsBuf = cast[ptr UncheckedArray[uint32]](ids[0].unsafeAddr)
parallelFor i in ids.low .. ids.high:
echo $idsBuf[i] & " " & myGetIMDBCoverURL(dbc,idsBuf,i,imdbURLs)
exit(Weave)
Run
Instead you should use:
# Fixed
proc parallelCoverURL(db, dbc: DbConn) =
init(Weave)
var ids: seq[uint32] = db.getAllRows(sql"SELECT tconst FROM title_basics
WHERE tconst NOT IN (SELECT id FROM
covers.lolz)").mapIt(parseuint(it[0]).uint32)
dbc.exec(sql"CREATE TABLE IF NOT EXISTS lolz (id INT PRIMARY KEY, url
VARCHAR)")
let idsBuf = cast[ptr UncheckedArray[uint32]](ids[0].unsafeAddr)
parallelFor i in ids.low .. ids.high:
captures: {dbc, idsBuf, imdbURLs}
echo $idsBuf[i] & " " & myGetIMDBCoverURL(dbc,idsBuf,i,imdbURLs)
exit(Weave)
Run
That said, Weave is to optimize compute, getting data from a database is
IO-bound, you'll only hammer your database and stress it with context switches:
* it's better if you retrieve what you need in memory and only then start
parallel processing. parallelFor makes sense for workloads that finishes at the
same time, like processing an image, the result from text queries will have
different lengths in particular and require different processing.
* or if your work is simple, just use plain async
* alternatively if work is sufficiently high, you can use spawn/sync
Also from your proc names, if you only want to download data, be sure to read
this:
<https://ep2019.europython.eu/media/conference/slides/KNhQYeQ-downloading-a-billion-files-in-python.pdf>
Lastly, if you are dealing with text, parallelFor is likely the wrong
architecture/construct. And make sure to compile with `--gc:arc` or `--gc:orc`
