Alright, so I'm new to Sequel and I'm enjoying it quite a bit after getting
over having to make a connection before defining a model class.
So here is the logic I need to use.
- Collect x rows of data, x = ~30k usually
- Loop through these records in batches of say, ~500
- Model import(those 500 records)
- Import Fail?
- Move these bad 500 to a low priority queue so I can loop through
them 1 by 1
- Do another 500
I know what your asking... Why do you have to do this? Because I didn't
design the part that the data comes from, or the database it ends up in.
I cannot "detect" this bad data because it is violating duplicate key
constraints on the db side. So I don't know if its bad until the database
tells me so.
So anyways, I have been playing with the options Sequel has available.
DB.transaction, Model.import(:slice => 1000) etc and trying to figure out
what's going to do the trick.
But for once I'm going to take a step back and try some advice first.
Anybody more experienced in this regard have any best practice advice for
me before I go much further? I'm over my wtf quota for today...
--
You received this message because you are subscribed to the Google Groups
"sequel-talk" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/sequel-talk/-/98bCqpTP540J.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/sequel-talk?hl=en.
