Cascading needs these dependencies as a compile dependency, at runtime they are optional.
I'm simply looking for them to exist so that i can automate tests against 2.4 and 2.6. I have no issues handling conditional dependencies (this is a requirement since every Hadoop release leaves stuff out anyway) https://github.com/cwensel/cascading/blob/wip-3.0/cascading-hadoop2-tez/build.gradle#L66 We take great pains with retaining compatibility: http://www.cascading.org/support/compatibility/ Also the missing deployable artifacts are a pain as well. I am already providing them to the Cascading community for testing purposes. https://github.com/cwensel/cascading/tree/wip-3.0/cascading-hadoop2-tez#running-a-tez-application As an aside, please feel free to update the README to reflect Cascading's ongoing support for Tez https://github.com/apache/tez/blob/master/README.md ckw > On Feb 25, 2015, at 11:45 AM, Hitesh Shah <[email protected]> wrote: > > Hello Chris > > From a supported point of view, both 2.4 and and 2.6 are supported. However, > ACLs support was not built into Timeline in Hadoop 2.4 hence a fully secure > setup cannot be supported with 2.4. Some of this is explained in > http://tez.apache.org/tez_yarn_timeline.html. > For the uploaded artifacts, not publishing the tez-yarn-timeline-history > artifact is something which I am aware of but I ignored it to some extent as > it does not contain any APIs that a user would need to compile against. Just > to be clear, the tez-yarn-timeline-history-with-acls artifact will not work > with 2.4 as it needs specific APIs introduced in hadoop 2.6. > > From a deployment of view, a few folks have called for the need of publishing > binary artifacts for convenience so that it makes it easy for users to deploy > Tez. In this regard, we ( the Tez community ) will probably need to provide > multiple sets compiled against different versions of hadoop. There are a > couple of reasons for this. The main one is obviously Timeline and what > features it supports or does not support in a particular version of Hadoop ( > and the fact that it did not exist in 2.2/2.3 ). The other is that YARN did > introduce some incompatibilities between 2.4 and 2.6 hence there are some > edge cases where an app compiled against 2.4 will not work on a 2.6 cluster. > > That said, are you looking to use any particular functionality from within > the tez-yarn-timeline-history-with-acls jar or looking to be able download > the Tez artifacts that one can deploy for use with a particular version of > hadoop? > > thanks > — Hitesh > > > On Feb 24, 2015, at 2:34 PM, Chris K Wensel <[email protected]> wrote: > >> >> As of Tez 0.6.0 I hear the intent is to support Hadoop 2.4 -> 2.6, and that >> 2.4/2.5 don’t have a secure yarn timeline server. >> >> I see there is a tez-yarn-timeline-history-with-acls artifact in maven for >> 0.6.0, which I assume is for a Hadoop 2.6+ deployment. This is also >> reflected in the poms for the 0.6.0 tez release. >> >> I see there is not a tez-yarn-timeline-history artifact in maven that would, >> I believe, provide backwards compatibility with 2.4. though the pom does >> list its in an alternative build profile. >> >> https://repository.apache.org/content/repositories/releases/org/apache/tez/tez-yarn-timeline-history/ >> >> <https://repository.apache.org/content/repositories/releases/org/apache/tez/tez-yarn-timeline-history/> >> >> Is not building and publishing this artifact by design, or an oversight? >> >> Or will the tez-yarn-timeline-history-with-acls artifact work with Hadoop >> 2.4/2.5? >> >> ckw >> >> — >> Chris K Wensel >> [email protected] >> >> >> >> > — Chris K Wensel [email protected]
