The promise of being able to port application logic to a different execution engine by dropping in a different Pipeline implementation was one of the key reasons for adopting Crunch at Nuna Health. I'm strongly supportive of a third Pipeline implementation :)
On Sun, Jan 25, 2015 at 11:01 AM, Josh Wills <[email protected]> wrote: > :) > > I never want anyone to have to rewrite code in order to pick up and move > it to a different execution engine. At the very least, we should write a > wrapper that lets you run existing o.a.c.DoFn subclasses in Dataflow > pipelines, and maybe even a DataflowPipeline implementation to port > existing Crunch pipelines over to Dataflow once it's easier to bring Hadoop > Input/OutputFormats over. > > I have some thoughts on why I think Dataflow is interesting, esp. to > developers who are familiar with Crunch/Spark/Scalding, but I'll send them > out in a different email b/c they're getting kind of long. > > J > > On Sun, Jan 25, 2015 at 10:02 AM, Danny Morgan <[email protected]> > wrote: > >> So should I start porting my crunch code over to the Cloud Dataflow sdk? >> >> Danny >> > >
