I think that list is all we've got. We only support Hadoop 2.x.x. We found that supporting both 1.x and 2.x has a cost that we couldn't afford. I believe there are fundamental differences between Hadoop 1.x and 2.x and that a good segment of Hadoop community still use 1.x. However, it has been a while since 1.x got a new release and so, I am not sure if it is worth investing time in making it work.
Also, seems to me that our Hadoop support is mainly for attracting existing users of Hadoop and so, I really think we should not invest in that area anymore. The only thing that I think we should continue doing is maybe add more tests (for different formats,etc). That is just my opinion :) What happened to Hadoop Compatibility Layer? Is that still a thing? On Thu, Jul 21, 2016 at 5:24 AM, Ian Maxon <[email protected]> wrote: > That's all the ways we use Hadoop at the moment that I can think of as > well. Maybe the two other minor ones are zookeeper and HDFS backup in > Managix. > > For 1) and 2) it's using Hadoop 2.2.0 right now. In my experimental branch > for 3) I'm using 2.6.0, it doesn't cause any more issues for me than 2.2.0. > I believe 1) used to support Hadoop 0.20.0 and other 1.x versions but I'm > not sure if that works anymore. > > On Wed, Jul 20, 2016 at 7:14 PM, Till Westmann <[email protected]> wrote: > > > Hi everybody, > > > > recently the topic of Hadoop support came up and I realized that my > > understanding is quite spotty so I’m trying to understand where we are. > > > > AFAIK we support > > 1) HDFS for (potentially indexed) external datasets, > > 2) YARN as a resource manager, and > > 3) HDFS as a basis for internal storage. > > Is this list complete or do we have other Hadoop touchpoints? > > > > I believe that 1) and 2) should be reasonable stable and that 3) is still > > in > > the works. Is that correct? > > > > Further I'm wondering > > a) which versions of Hadoop we support and > > b) which ones we should support for all the cases. > > Please chime in on this as well. > > > > Any other things that anybody working with AsterixDB and Hadoop should be > > aware > > of? > > > > Thanks! > > Till > > > > >
