-0 I read the thread twice, it is not clear to me what benefit Apex users derive from this exercise. A branch normally contains development work that is eventually brought back to the main line and into a release. Here, the suggestion seems to be an open ended effort to play with latest tech, isn't that something anyone (including a group of folks) can do in a fork. I don't see value in a permanent branch for that, who is going to maintain such code and who will ever use it?
There was a point that we can find out about potential problems with later versions. The way to find such issues is to take the releases and run them on these later versions (that's what users do), not by changing the code! Regarding Java version: Our users don't use Apex in a vacuum. Please have a look at ASF Hadoop and the distros EOL policies. That will answer the question what Java version is appropriate. I would be surprised if something that works on Java 7 falls flat on the face with Java 8 as a lot of diligence goes into backward compatibility. Again the way to tests this is to run verification with existing Apex releases on Java 8 based stack. Regarding Hadoop version: This has been discussed off record several times and there are actual JIRA tickets marked accordingly so that the work is done when we move. It is a separate discussion, no need to mix Java versions and branching with it. I agree with what David said, if someone can show that we can move up to 2.6 based on EOL policies and what known Apex users have in production, then we should work on that upgrade. The way I imagine it would work is that we have a Hadoop-2.6 (or whatever version) branch, make all the upgrade related changes there (which should be a list of JIRAs) and then merge it back to master when we are satisfied. After that, the branch can be deleted. Thomas On Tue, Jul 12, 2016 at 8:36 AM, Chinmay Kolhatkar <[email protected]> wrote: > I'm -0 on this idea. > > Here is the reason: > Unless we see a real case where users want to see everything on latest, > this branch might quickly become low hanging fruit and eventually get > obsolete because its anyway a "no gaurantee" branch. > > We have a bunch of dependencies which we'll have to take care of to really > make it bleeding edge. Specially about malhar, its a long list. That looks > like quite significant work. > Moreover, if this branch is going to be in "may or may not work" state; I, > as a user or developer, would bank on what certainly works. > > I also think that, if its going to be "no gaurantee" then its worth > spending time contributions towards master rather than bleeding-edge > branch. > > If a question of "should we upgrade?" comes, the community is mature to > take that call then and work accordingly. > > -Chinmay. > > > > On Tue, Jul 12, 2016 at 11:42 AM, Priyanka Gugale <[email protected]> > wrote: > > > +1 for creating such branch. > > One of us will have to rebase it with master branch at intervals. I don't > > think everyone will cherry-pick their commits here. We can make it once > in > > a month activity. Are we considering updating all dependency library > > version as well? > > > > -Priyanka > > > > On Tue, Jul 12, 2016 at 2:34 AM, Munagala Ramanath <[email protected]> > > wrote: > > > > > Following up on some comments, wanted to clarify what I have in mind > for > > > this branch: > > > > > > 1. The main goal is to stay up-to-date with new releases, so if a > > question > > > of the form > > > "A new release of X is available, should we upgrade ?" comes up, > the > > > answer is > > > *always* an *emphatic* yes; otherwise it doesn't bleed enough (:-) > as > > > Sanjay points out. > > > 2. Pull requests are submitted as always; there is no requirement to > > > generate an additional > > > pull requests against this branch. It may get merged/cherry-picked > > > depending on who has the > > > time and inclination to do it. > > > 3. There is no expectation of dedication of any additional resources, > so > > > people work on > > > it as and when time is available. ("No guarantee" means exactly > > that). > > > So there is no > > > question of "maintaining" this branch. > > > 4. This branch is not to be encumbered with legacy and/or backward > > > compatibility issues. > > > 5. This branch is not an experimental sandbox to try out new > algorithms, > > > architectural changes > > > and other such changes. > > > > > > As always, I'm open to other ideas, but that is what I had in mind > when I > > > made the suggestion. > > > > > > Ram > > > > > > > > > > > > On Mon, Jul 11, 2016 at 1:45 PM, Sanjay Pujare <[email protected] > > > > > wrote: > > > > > > > As the name suggests the "bleeding-edge" branch ideally should use > > > bleeding > > > > edge versions so I would like to see Java 8 used there (and Hadoop 3 > > when > > > > it does eventually come out) to make the maintenance effort > > worthwhile... > > > > > > > > On Mon, Jul 11, 2016 at 12:05 PM, David Yan <[email protected]> > > > wrote: > > > > > > > > > I'm -0 on Java 8, but I'm +1 on the rest, and I'm especially strong > > +1 > > > > for > > > > > upgrading the Hadoop dependency version. > > > > > > > > > > Here are my reasons: > > > > > > > > > > - Hadoop 3 will require Java 8, but Hadoop 2.7.2 still supports > Java > > 7 > > > > and > > > > > there will probably be some time (I'm guessing more than one year) > > for > > > > > Hadoop 3 to become GA and for major distros to support Hadoop 3. > The > > > > > maintenance effort for having two branches, one for Java 7 and one > > for > > > > Java > > > > > 8 is not worth it at this time. > > > > > > > > > > - Apex currently uses Hadoop 2.2 dependencies, marked "provided". > And > > > > > Hadoop 2.4 has been released more than two years ago, and it added > a > > > lot > > > > of > > > > > features in the API that Apex can make use of. Most distros already > > > > bundle > > > > > Hadoop 2.6 or later. Although some old versions of Cloudera that > > > include > > > > > hadoop version earlier than 2.4 still have not reached end-of-life > > yet, > > > > the > > > > > number of users using those old versions is probably very small. > > > > > > > > > > David > > > > > > > > > > > > > > > On Mon, Jul 11, 2016 at 8:59 AM, Munagala Ramanath < > > > [email protected]> > > > > > wrote: > > > > > > > > > > > We've had a number of issues recently related to dependencies on > > old > > > > > > versions > > > > > > of various packages/libraries such as Hadoop itself, Google > guava, > > > > > > HTTPClient, > > > > > > mbassador, etc. > > > > > > > > > > > > How about we create a "bleeding-edge" branch in both Core and > > Malhar > > > > > which > > > > > > will use the latest versions of these various dependencies, > upgrade > > > to > > > > > Java > > > > > > 8 so > > > > > > we can use the new Java features, etc. ? > > > > > > > > > > > > This will give us an opportunity to discover these sorts of > > problems > > > > > early > > > > > > and, > > > > > > when we are ready to pull the trigger for a major version, we > have > > a > > > > > branch > > > > > > ready > > > > > > for merge with, hopefully, minimal additional effort. > > > > > > > > > > > > There will be no guarantees w.r.t. this branch so people using it > > use > > > > it > > > > > at > > > > > > their own > > > > > > risk. > > > > > > > > > > > > Ram > > > > > > > > > > > > > > > > > > > > >
