New reply on DataCleaner's online discussion forum (http://datacleaner.org/forum):
SteveH replied to subject 'Best practice for build complex pipelines?' ------------------- Kasper, Many thanks for your reply - it is very helpful, but I have a follow up question. I think I was a little too detailed in my example. Lets say that I am going to build a very large job - lets assume with 200 steps in it - and there is a lot of conditional branches in it. Lots of steps where I take different processing paths depending on difference data elements. And also a number of places where I need to bring these paths back together to start the next part of a chain of processing. Do I build this as one very large job, with lots of intermediate staging tables? Or... do I build a series of seperate jobs and string them together as child jobs orchestrated by a master job? Either route seems to have benefits and weaknesses, especially if I use very large datasets. I'm really looking to get a feel for which pattern you would suggest I start looking at first? ------------------- View the topic online to reply - go to http://datacleaner.org/topic/1098/Best-practice-for-build-complex-pipelines%3F -- You received this message because you are subscribed to the Google Groups "DataCleaner-notify" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/datacleaner-notify. For more options, visit https://groups.google.com/d/optout.
