On Wednesday, May 8, 2019 at 5:04:34 PM UTC-7, Jonathan Simpson wrote:
>
> Thanks, Jeremy!
>
> No, we are not using the same database object. Each source or destination
> in these ETL jobs is it's own database object.
>
> What follows is the backtrace and I've cobbled together a quick/contrived
> example that I hope better demonstrates what we're doing specifically. If
> that doesn't help, I could work on putting together an actual example.
> Thanks again for your help!
>
> Backtrace:
>
> /home/deploy/redacted/shared/bundle/ruby/2.6.0/gems/sequel-5.17.0/lib/sequel/adapters/tinytds.rb:205:in
> `fields'
> /home/deploy/redacted/shared/bundle/ruby/2.6.0/gems/sequel-5.17.0/lib/sequel/adapters/tinytds.rb:205:in
> `block in fetch_rows'
> /home/deploy/redacted/shared/bundle/ruby/2.6.0/gems/sequel-5.17.0/lib/sequel/adapters/tinytds.rb:65:in
> `block in execute'
> /home/deploy/redacted/shared/bundle/ruby/2.6.0/gems/sequel-5.17.0/lib/sequel/connection_pool/threaded.rb:88:in
> `hold'
> /home/deploy/redacted/shared/bundle/ruby/2.6.0/gems/sequel-5.17.0/lib/sequel/database/connecting.rb:270:in
> `synchronize'
> /home/deploy/redacted/shared/bundle/ruby/2.6.0/gems/sequel-5.17.0/lib/sequel/adapters/tinytds.rb:34:in
> `execute'
> /home/deploy/redacted/shared/bundle/ruby/2.6.0/gems/sequel-5.17.0/lib/sequel/dataset/actions.rb:1087:in
> `execute'
> /home/deploy/redacted/shared/bundle/ruby/2.6.0/gems/sequel-5.17.0/lib/sequel/adapters/tinytds.rb:202:in
> `fetch_rows'
> /home/deploy/redacted/shared/bundle/ruby/2.6.0/gems/sequel-5.17.0/lib/sequel/dataset/actions.rb:152:in
> `each'
> /home/deploy/redacted/shared/bundle/ruby/2.6.0/gems/sequel-5.17.0/lib/sequel/dataset/actions.rb:590:in
> `block in paged_each'
> /home/deploy/redacted/shared/bundle/ruby/2.6.0/gems/sequel-5.17.0/lib/sequel/database/transactions.rb:245:in
> `_transaction'
> /home/deploy/redacted/shared/bundle/ruby/2.6.0/gems/sequel-5.17.0/lib/sequel/database/transactions.rb:220:in
> `block in transaction'
> /home/deploy/redacted/shared/bundle/ruby/2.6.0/gems/sequel-5.17.0/lib/sequel/connection_pool/threaded.rb:92:in
> `hold'
> /home/deploy/redacted/shared/bundle/ruby/2.6.0/gems/sequel-5.17.0/lib/sequel/database/connecting.rb:270:in
> `synchronize'
> /home/deploy/redacted/shared/bundle/ruby/2.6.0/gems/sequel-5.17.0/lib/sequel/database/transactions.rb:186:in
> `transaction'
> /home/deploy/redacted/shared/bundle/ruby/2.6.0/gems/sequel-5.17.0/lib/sequel/dataset/actions.rb:566:in
> `paged_each'
>
>
> Reads and writes:
>
> class Reader
> class << self
> def read(dataset)
> dataset.paged_each { |row| row }
> end
> end
> end
>
> class Writer
> class << self
> def write(dataset, &block)
> block.call.each_slice(1000) { |batch| dataset.multi_insert(batch) }
> end
> end
> end
>
> dataset = @database[@table_name]
> Writer.write(Reader.read(dataset))
>
>
Thank you for your example. I haven't actually tried it, but I'm not sure
how this code can work, because Writer.write is not called with a block, so
block.call should raise a NoMethodError.
You probably want something like:
writer_ds = @writer_database[@table_name]
@reader_database[@table_name].paged_each.each_slice(1000){|batch|
writer_ds.multi_insert(batch)}
Thanks,
Jeremy
--
You received this message because you are subscribed to the Google Groups
"sequel-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/sequel-talk.
To view this discussion on the web visit
https://groups.google.com/d/msgid/sequel-talk/c7e81002-2792-404f-8c47-6b343ec18a53%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
