You're correct, of course. I wasn't very careful when writing up that brief
example. The actual code is almost exactly like the example you provided
here.
FWIW, increasing the timeout on database objects that are using the TinyTDS
adapter seems to have resolved this specific issue. It turns out that the
database servers I'm working with are across AWS zones and the latency
between them isn't great and that is a problem for a different day.
Thanks again for your help and also for this wonderful library, Jeremy!
On Wednesday, May 8, 2019 at 5:40:47 PM UTC-7, Jeremy Evans wrote:
>
> On Wednesday, May 8, 2019 at 5:04:34 PM UTC-7, Jonathan Simpson wrote:
>>
>> Thanks, Jeremy!
>>
>> No, we are not using the same database object. Each source or destination
>> in these ETL jobs is it's own database object.
>>
>> What follows is the backtrace and I've cobbled together a quick/contrived
>> example that I hope better demonstrates what we're doing specifically. If
>> that doesn't help, I could work on putting together an actual example.
>> Thanks again for your help!
>>
>> Backtrace:
>>
>> /home/deploy/redacted/shared/bundle/ruby/2.6.0/gems/sequel-5.17.0/lib/sequel/adapters/tinytds.rb:205:in
>> `fields'
>> /home/deploy/redacted/shared/bundle/ruby/2.6.0/gems/sequel-5.17.0/lib/sequel/adapters/tinytds.rb:205:in
>> `block in fetch_rows'
>> /home/deploy/redacted/shared/bundle/ruby/2.6.0/gems/sequel-5.17.0/lib/sequel/adapters/tinytds.rb:65:in
>> `block in execute'
>> /home/deploy/redacted/shared/bundle/ruby/2.6.0/gems/sequel-5.17.0/lib/sequel/connection_pool/threaded.rb:88:in
>> `hold'
>> /home/deploy/redacted/shared/bundle/ruby/2.6.0/gems/sequel-5.17.0/lib/sequel/database/connecting.rb:270:in
>> `synchronize'
>> /home/deploy/redacted/shared/bundle/ruby/2.6.0/gems/sequel-5.17.0/lib/sequel/adapters/tinytds.rb:34:in
>> `execute'
>> /home/deploy/redacted/shared/bundle/ruby/2.6.0/gems/sequel-5.17.0/lib/sequel/dataset/actions.rb:1087:in
>> `execute'
>> /home/deploy/redacted/shared/bundle/ruby/2.6.0/gems/sequel-5.17.0/lib/sequel/adapters/tinytds.rb:202:in
>> `fetch_rows'
>> /home/deploy/redacted/shared/bundle/ruby/2.6.0/gems/sequel-5.17.0/lib/sequel/dataset/actions.rb:152:in
>> `each'
>> /home/deploy/redacted/shared/bundle/ruby/2.6.0/gems/sequel-5.17.0/lib/sequel/dataset/actions.rb:590:in
>> `block in paged_each'
>> /home/deploy/redacted/shared/bundle/ruby/2.6.0/gems/sequel-5.17.0/lib/sequel/database/transactions.rb:245:in
>> `_transaction'
>> /home/deploy/redacted/shared/bundle/ruby/2.6.0/gems/sequel-5.17.0/lib/sequel/database/transactions.rb:220:in
>> `block in transaction'
>> /home/deploy/redacted/shared/bundle/ruby/2.6.0/gems/sequel-5.17.0/lib/sequel/connection_pool/threaded.rb:92:in
>> `hold'
>> /home/deploy/redacted/shared/bundle/ruby/2.6.0/gems/sequel-5.17.0/lib/sequel/database/connecting.rb:270:in
>> `synchronize'
>> /home/deploy/redacted/shared/bundle/ruby/2.6.0/gems/sequel-5.17.0/lib/sequel/database/transactions.rb:186:in
>> `transaction'
>> /home/deploy/redacted/shared/bundle/ruby/2.6.0/gems/sequel-5.17.0/lib/sequel/dataset/actions.rb:566:in
>> `paged_each'
>>
>>
>> Reads and writes:
>>
>> class Reader
>> class << self
>> def read(dataset)
>> dataset.paged_each { |row| row }
>> end
>> end
>> end
>>
>> class Writer
>> class << self
>> def write(dataset, &block)
>> block.call.each_slice(1000) { |batch| dataset.multi_insert(batch) }
>> end
>> end
>> end
>>
>> dataset = @database[@table_name]
>> Writer.write(Reader.read(dataset))
>>
>>
> Thank you for your example. I haven't actually tried it, but I'm not sure
> how this code can work, because Writer.write is not called with a block, so
> block.call should raise a NoMethodError.
>
> You probably want something like:
>
> writer_ds = @writer_database[@table_name]
> @reader_database[@table_name].paged_each.each_slice(1000){|batch|
> writer_ds.multi_insert(batch)}
>
> Thanks,
> Jeremy
>
--
You received this message because you are subscribed to the Google Groups
"sequel-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/sequel-talk.
To view this discussion on the web visit
https://groups.google.com/d/msgid/sequel-talk/f10e9825-a139-4162-9137-d8644e1d3144%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
