Dan, The proper place to ask is in the mailing list: "rhino-tools-dev" < [email protected]>, Answers below (and the mailing list is CCed).
On Fri, Apr 2, 2010 at 5:52 PM, Dan Jensen <[email protected]> wrote: > Paul Barrier (you linked to his talk about ETL recently) has since > commented and suggested a couple > of things to try. I tried the SqlBatch option, but it was much slower. So > now I've started over-riding the > Execute method of the SqlBulkInsertOperation in order to do smaller batches > of inserts at a time. Now > the bulk insert itself is small, but I'm still getting the memory > exception. I next profiled the app with ANTS, > and discovered that the cause of the exception was the collection of Row > objects that is being passed > between the operations in the process. So this is my question: Is there a > way around this? > > Hm, what is that collection? Who is maintaining it? Is it Rhino ETL? If so, what executer are you using? > I thought that by yielding to each subsequent operation, only one row at a > time would be passed between > the operations. > Yes, that is how it should work. Of the top of my head, I can't think of a reason it would behave in this manner. However, it looks like that collection continues to grow with each row > processed. So the > only solution I can think of is to do a row count on the source table > before starting, and then run the table > in chunks through the ETL process. This should prevent the Rows > collection from growing too large. > > > So if you have an idea of a more elegant solution, I'd really > appreciate the help! > > Thank you very much, > Dan > > P.S. I did have one other question, actually. I noticed in the code of > the SqlBulInsertOperation that you aren't disposing > the SqlBulkCopy object. I don't know much about this object, so I was > curious why you didn't? I tried disposing > it to see if that helped my problem, but it didn't. > I probably should dispose it, yes. -- You received this message because you are subscribed to the Google Groups "Rhino Tools Dev" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/rhino-tools-dev?hl=en.
