Hi, FWIW, I once had a problem where the caching enumerable can interact with the branching operation in such a way that the pipeline thinks it has finished when it hasn't. IIRC, this was because the rows had been read into the caching enumerables, but not necessarily processed by all the subsequent operations in all the branches - it appeared to terminate when the first branch finished.
The problem went away when I switched to the single-threaded pipeline executer, which is what I had meant to use in the first place, so I didn't investigate any further! @jalchr: if you have memory problems, use the single-threaded/simple pipeline executer when running your ETL to avoid all the rows being read into memory. This will avoid having to change the code in any of the operations. Miles On Sun, Jul 10, 2011 at 12:25 AM, Simone Busoli <[email protected]>wrote: > The operation result is shoved into a caching enumerabke because it's > iterated more than once. Please note that if you remove that then you're > effectively enumerating the source more than one time, which may not be what > you expect. > I can't tell why it gets stuck on the multi threaded branching operation, > have you tried then single threaded one? Do you have a repro for the issue? > On Jul 8, 2011 3:47 PM, "jalchr" <[email protected]> wrote: > > I'm trying to getting out of OutOfMemory Exception, so I noticed about > 1.5 > > Million rows are being cached in the CachingEnumerable object ... I > wonder > > why this doesn't stream ... why its cached. So I changed the following > > lines: > > > > > > protected IEnumerable<Row> GetRightEnumerable() > > { > > //IEnumerable<Row> rightEnumerable = new CachingEnumerable<Row>( > > // new EventRaisingEnumerator(right, right.Execute(null)) > > // ); > > IEnumerable<Row> rightEnumerable = new > > EventRaisingEnumerator(right, right.Execute(null)); > > > > > > and it works perfectly with much less memory ... I only had to increase > the > > input operation timeout value > > > > All is good except with MultiThreadedBranchingOperation ... where after > some > > processing it get stuck on: > > > > lock (sync) > > while (input.ConsumersLeft > 0) > > Monitor.Wait(sync); // <--------------------------- here > > > > > > Questions: > > why you cache to join? > > why it stucks there ? > > > > Thanks > > > > > > -- > > You received this message because you are subscribed to the Google Groups > "Rhino Tools Dev" group. > > To view this discussion on the web visit > https://groups.google.com/d/msg/rhino-tools-dev/-/rOzjgybGSYkJ. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > [email protected]. > > For more options, visit this group at > http://groups.google.com/group/rhino-tools-dev?hl=en. > > > > -- > You received this message because you are subscribed to the Google Groups > "Rhino Tools Dev" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/rhino-tools-dev?hl=en. > -- You received this message because you are subscribed to the Google Groups "Rhino Tools Dev" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/rhino-tools-dev?hl=en.
