I'd be willing to bet that you have an operation that isn't either iterating the input rows or returning data for the next operation to take. Keep in mind each operation reads the input rows and that if it doesn't, rows stop flowing through.
Here's an intro article http://www.codeproject.com/KB/cs/ETLWithCSharp.aspx It also seems like you have a bunch of unrelated operations in the same process, i.e. the output rows from CinemaProject probably don't have anything to do with the input for TheatreAddress. If that is the case you may want a process per type, each with two operations. If they are related, some form of branching and joining may be needed. On Dec 8, 10:57 pm, rbellamy <rbell...@pteradigm.com> wrote: > If this message should go to a different list, let me know and I'll > hop over there... > > I'm looking for some guidance in troubleshooting a problem with the my > load process. Maybe dropping a pipeline (or something like that). > Should I be using partial processes for each of the input/output > loads? I've tried debugging, and I need some help or a helping hand > with the best way to do so - that threadsafe enumerator with delegate > is kicking my a%%. > > 1. I register 10 processes - 5 input, 5 output, staggered like so: > protected override void Initialize() > { > Register(new StaticStatesFromDatabase > (this.options.SourceConnectionStringName)); > Register(new StaticStatesToDatabase > (this.options.TargetConnectionStringName)); > Register(new CinemaProjectAuditoriumFromDatabase > (this.options.SourceConnectionStringName)); > Register(new CinemaProjectAuditoriumToDatabase > (this.options.TargetConnectionStringName)); > Register(new TheatreAddressFromDatabase > (this.options.SourceConnectionStringName)); > Register(new TheatreAddressToDatabase > (this.options.TargetConnectionStringName)); > Register(new TheatreAuditoriumFromDatabase > (this.options.SourceConnectionStringName)); > Register(new TheatreAuditoriumToDatabase > (this.options.TargetConnectionStringName)); > Register(new TheatreFromDatabase > (this.options.SourceConnectionStringName)); > Register(new TheatreToDatabase > (this.options.TargetConnectionStringName));} > > 2. Everything performs as expected, until the end where the log shows > that the pipeline is dropping the last ~1100 rows of the > TheatreAuditoriumToDatabase process: > 2009-12-08 20:25:16,863 [3] NOTICE Etl.Professional.LoadProcess - > Finished CertificationToDatabase: 2076 Rows in 00:03:32.9030000 > 2009-12-08 20:26:39,287 [16] INFO Etl.Cinema.LoadProcess - Processed > 12000 rows in TheatreAuditoriumToDatabase > 2009-12-08 20:26:55,541 [18] INFO Etl.Professional.LoadProcess - > Processed 3000 rows in DesignerToDatabase > 2009-12-08 20:28:20,884 [16] INFO Etl.Cinema.LoadProcess - Processed > 13000 rows in TheatreAuditoriumToDatabase > 2009-12-08 20:28:37,629 [18] INFO Etl.Professional.LoadProcess - > Processed 4000 rows in DesignerToDatabase > 2009-12-08 20:30:03,224 [16] INFO Etl.Cinema.LoadProcess - Processed > 14000 rows in TheatreAuditoriumToDatabase > 2009-12-08 20:30:22,391 [18] INFO Etl.Professional.LoadProcess - > Processed 5000 rows in DesignerToDatabase > 2009-12-08 20:31:27,566 [18] NOTICE Etl.Professional.LoadProcess - > Finished DesignerToDatabase: 5627 Rows in 00:09:43.6060000 > 2009-12-08 20:31:27,569 [9] NOTICE > Rhino.Etl.Core.Pipelines.ThreadPoolPipelineExecuter - Completed > process LoadProcess in 00:09:43.6090000 -- You received this message because you are subscribed to the Google Groups "Rhino Tools Dev" group. To post to this group, send email to rhino-tools-...@googlegroups.com. To unsubscribe from this group, send email to rhino-tools-dev+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/rhino-tools-dev?hl=en.
