Supporting the Hadoop 2.0 (not 2.2) YARN API would be a lot of coding effort. There was a huge API change between the two versions. Maybe we can find a technical solution to this political/legal problem: I'm going to build and try a Flink version against the "2.1.1-beta" (or similar) (official Apache Hadoop release) and see if that's working as well with CDH4. Then, we can provide a non-vendor specific binary that still solves the problem for our users. Our problem is not as severe as for Spark, since they have (in my understanding) support for both YARN APIs. So our issue with the CDH4 / Hadoop 2.1-beta is only related to the HDFS client, not the whole YARN API.
On Mon, Aug 18, 2014 at 7:43 PM, Stephan Ewen <[email protected]> wrote: > I like Sean's idea very much: Creating the three packages (Hadoop 1.x, > Hadoop 2.x, Hadoop 2.0 with Yarn beta). > > Any objections to creating a help site that says "For that vendor with this > version pick the following binary release" ? > > Stephan > > > > > >> On Mon, Aug 18, 2014 at 5:58 PM, Henry Saputra < > [email protected]> > > wrote: > > >>> As for Flink, for now the additional CDH4 packaged binary is to > > >>> support "non-standard" Hadoop version that some customers may already > > >>> have. > > >>> > > >>> Based on "not a question of supporting a vendor but a Hadoop version > > >>> combo.", would the approach that Flink had done to help customers get > > >>> go and running quickly seemed fair and good idea? > > >>> > > >>> There had been a lot of discussion about ASF release artifacts and > the > > >>> consistent answer is that ASF validate release of source code and not > > >>> binaries. > > >>> Release of binaries only used to help customers, which is the case > > >>> that Flink is doing with different Hadoop versions. > > >>> > > >>> - Henry > > >>> > > >>> On Mon, Aug 18, 2014 at 9:51 AM, Sean Owen <[email protected]> wrote: > > >>>> It's probably the same thing as with Spark. Spark doesn't actually > > >>>> work with YARN 'beta'-era releases, but works 'stable' and specially > > >>>> supports 'alpha'. CDH 4.{2-4} or so == YARN 'beta' (not > non-standard, > > >>>> but, is probably the only distro of it you'll still run into in > > >>>> circulation). (And so it's kind of unhelpful that Spark has build > > >>>> instructions for CDH 4.2 + YARN.) Yeah, that's the thing you may > > >>>> handle as a corner case, or not handle and punt to the vendor. But > > >>>> even that -- if that's the same issue -- it's not a question of > > >>>> supporting a vendor but a Hadoop version combo. > > >>>> > > >>>> On Mon, Aug 18, 2014 at 5:43 PM, Stephan Ewen <[email protected]> > > wrote: > > >>>>> I think the main problem was that CDH4 is a non standard build. All > > others > > >>>>> we tried worked with hadoop-1.2 and 2.2/2.4 builds. > > >>>>> > > >>>>> But I understand your points. > > >>>>> > > >>>>> So, instead of creating those packages, we can make a guide "how to > > pick > > >>>>> the right distribution", which points you to the hadoop-1.2 and > > 2.2/2.4 > > >>>>> builds. For some cases, the guide will ask you to > "compile-your-own". > > >>>>> > > >
