On Friday, September 2, 2016 at 1:19:02 PM UTC-5, Jeremy Evans wrote:
>
> To work around this issue, you can use sharding (Dataset option like
> :servers=>{:foo=>{}}), and do dataset.server(:foo).paged_each.
> Alternatively, you can issue queries in a separate thread inside the
> Dataset#each block. Even something simple like
> dataset.each{Thread.new{...}.join} should work. Please try one of those
> options and report back.
>
Ah, your suggestion to use threads seems nice. I tested running this
simultaneously and it seems to work without a hitch:
Person.dataset.limit(15).paged_each { |p| Thread.new { sleep(1) ;
p.update_all(updated_at: Time.now.utc) ; print '.' }.join }
Is there any reason to prefer streaming support via sequel_pg if I can get
that working, or would you think this is a reasonable way to proceed? My
backfill process would look something like this, and would be iterating
over ~100 million rows:
Person.where(updated_at: nil).paged_each { |p| Thread.new {
p.update_all(updated_at: Time.now.utc) }.join }
Thanks again,
- Trevor
--
You received this message because you are subscribed to the Google Groups
"sequel-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/sequel-talk.
For more options, visit https://groups.google.com/d/optout.
