I have a process that I would typically use SSIS for but used RETL instead. The process is to run every row of a 190M row table through a transformation routine and execute some .Net code on the data and then update the row. This was very easy to set up and I'm using a SqlBatchOperation on the update side. The issue is that I can read the data from the source table really fast so within a few seconds my memory usage starts to skyrocket (6 GB) and if I just let it go eventually the throughout on the update operation grinds to a halt. I believe the issue is with the ThreadSafeEnumerator and it's use of a cache because if I change the default PipelineExecutor to be a SingleThreadedNonCachedPipelineExecuter then my memory usage stays below 50 MB and everything works but since I'm not multithreaded the performance is slower. Is there something I can do curtail the memory usage but keep the performance as it's peak level?
Thanks -- You received this message because you are subscribed to the Google Groups "Rhino Tools Dev" group. To view this discussion on the web visit https://groups.google.com/d/msg/rhino-tools-dev/-/ssGQEPL31VkJ. To post to this group, send email to rhino-tools-dev@googlegroups.com. To unsubscribe from this group, send email to rhino-tools-dev+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/rhino-tools-dev?hl=en.
