For the upgrade it is probably best when you change the version number in [1] and then work through the build issues.
I would expect a number of dependency changes since the client classpath now has the Hadoop transitive dependencies shaded away [2]. Regarding the development process, see [3]. Thomas [1] https://github.com/apache/apex-core/blob/1f5425dc5ac1d3452a3b37db0e4c61485af9c6e1/pom.xml#L109 [2] https://hadoop.apache.org/docs/r3.0.0/ [3] http://apex.apache.org/contributing.html On Fri, Jan 11, 2019 at 5:14 PM Aaron Bossert <aa...@punchcyber.com> wrote: > at first blush, I would think, that the YARN node tagging and native Docker > support would be particularly intriguing...the GPU node support would be a > good implementation trick to be able to put operators that do deep learning > tasks... > > On Fri, Jan 11, 2019 at 2:42 PM Pramod Immaneni <pramod.imman...@gmail.com > > > wrote: > > > I agree we should just try changing the dependency first. I am in favor > of > > upgrading. It will be good to explore if and how we can leverage some of > > the new features in 3.x. > > > > On Fri, Jan 11, 2019 at 11:38 AM Vlad Rozov <vro...@apache.org> wrote: > > > > > Hi Aaron, > > > > > > I’d recommend to start with upgrading Hadoop dependency to 3.x, > > > recompiling and running unit test. Also, before you put significant > > effort > > > into Hadoop 3.x upgrade, let’s discuss and vote on dev@apex. Once > > > everyone agrees and vote passes, please open JIRA for tracking the > > changes. > > > > > > Thank you, > > > > > > Vlad > > > > > > > On Jan 10, 2019, at 07:58, Aaron Bossert <aa...@punchcyber.com> > wrote: > > > > > > > > As requested by Thomas, I am hoping we can kick off a discussion > about > > a > > > > short-term goal of adding support for Hadoop 3.X, though, if I am not > > > > mistaken, there was a pretty large change from 3.0 to 3.1...but will > > > > confirm, in which case, the ask is to support 3.1+ specifically. > > > > > > > > I program in Scala for the most part, but have not problem switching > to > > > > pure Java as needed. That being said, I am very unfamiliar with the > > Apex > > > > code base and also unsure of the scope and level of effort required > to > > > get > > > > this done. I look to the group for a few pointers on where to begin > as > > > > well as code style and such...I don't think I will need a lot of > > > > hand-holding, just a good shove in the right direction... > > > > > > > > Thanks again for the discussion and support on this...for background > > here > > > > is my personal and professional motivation for pressing forward: > > > > > > > > - Doing research work for clients as well as internal R&D for > > streaming > > > > data processing > > > > - very specific requirement to support the ability to alter > > > > topologies programatically and manually on the fly, hence the > > > interest in > > > > Apex, which is the only platform I know of that has this support > > > built > > > > in...I don't want to re-invent that wheel unless absolutely > > > necessary > > > > - General interest in supporting the open-source community, both > as a > > > > company and individually > > > > > > > > -- > > > > > > > > M. Aaron Bossert > > > > (202) 594-1703 > > > > Punch Cyber Analytics Group > > > > > > -- > > Thanks, > > Pramod > > http://ts.la/pramod3443 > > > > > -- > > M. Aaron Bossert > (202) 594-1703 > Punch Cyber Analytics Group >
