Thanks for explaining your thoughts, Carl! Those answers all sound great to
Best of luck!
On Tue, Mar 6, 2018, 5:58 PM Carl Steinbach <cwsteinb...@gmail.com> wrote:
> Hi Mike,
> Why does Dr. Elephant make sense as a separate project instead of
> > contributing to Hadoop directly?
> Here are a couple reasons why I think Dr. Elephant is more likely to
> succeed as a separate project:
> * Dr. Elephant supports Hadoop *and* Spark, and may support other
> execution layers in the future. If we make Dr. Elephant a part of
> Hadoop I expect that it will discourage contributions from people
> who are interested mainly in Spark support, and vice versa.
> * If Dr. Elephant is added to Hadoop it will be necessary for the
> Hadoop project to declare a dependency on Spark. I doubt this change
> will get approved.
> * We don't want to tie Dr. Elephant to a specific version of Hadoop or
> Spark, or tie the Dr. Elephant release cycle to the Hadoop or Spark
> release cycles.
> * None of the current Dr. Elephant committers are Hadoop committers,
> and I doubt that the Hadoop PMC is going to give them a commit bit
> just to work on Dr. Elephant. As a result the existing committers
> would be effectively forfeiting their right to continue maintaining
> their own project. I think this is one of the reasons why many
> Hadoop contrib projects are poorly maintained.
> > What is the relationship between Dr. Elephant and the (now seemingly
> > defunct) Hadoop Vaidya?
> Vaidya was a command line tool for tuning Hadoop jobs. Dr. Elephant is
> an always-on service for tuning Hadoop and Spark jobs. We were unaware
> of Vaidya when we started working on Dr. Elephant.
> - Carl