[
https://issues.apache.org/jira/browse/FALCON-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15295743#comment-15295743
]
Venkat Ranganathan commented on FALCON-1977:
--------------------------------------------
[~sriksun] Good question. Today falcon does work with 2.6.x and 2.7.x (in
fact after your comment, I checked the POM files and find a few places where we
were not strict with dependency scope (using compile instead of provided) which
might bring in hadoop libraries when not needed. Will file a separate JIRA
for that.
That said, if a 2.7.x specific change creeps in the future, we will not be able
to catch it if the default version is 2.7.1, but I find the chance for that to
be minimal, based on the APIs we use in Falcon core. One change in 2.7.x
line is the requirement that AWS and Azure FS dependency be explicit in the pom
file (they are removed from core hadoop common). That is something we have
to handle for hadoop-azure FS (currently done only in regression codebase).
Will check with [~yzheng-hortonworks] about including the Azure FS
implementation by default. This dependency change if introduced should still
work with 2.6.x
To have the ability to build falcon with 2.6.x (even if not default version),
we can make the hadoop version of falcon unit to be 2.7.1 explicitly, similar
to have it is set in regression and what was there before. This should help us
have the old behavior of Falcon having a different version from unit and
regression. Will update the patch for this.
Since unit is intertwined with the rest of falcon UT and IT testing, having it
on a different default hadoop version than main is odd though, unlike
regression, which is in the repo but is not part of the build profile.
> Move falcon-main hadoop version to match falcon-unit and falcon-regression
> hadoop versions
> ------------------------------------------------------------------------------------------
>
> Key: FALCON-1977
> URL: https://issues.apache.org/jira/browse/FALCON-1977
> Project: Falcon
> Issue Type: Bug
> Reporter: Venkat Ranganathan
> Assignee: Venkat Ranganathan
>
> Falcon-unit requires Hadoop 2.7.x APIs. This is different from the falcon
> main which uses 2.6.2 - There were some discussions on keeping falcon
> version to 2.6.2
> As part of cleaning up hadoop-2 profile, it would be good to move falcon-unit
> to addons like hivedr, hdfs-snapshot-mirroring etc which require higher
> versions of components and enabled with a specific profile. That way, users
> can build Falcon after explicitly updating the component versions.
> But given that Falcon unit (an integration test) and falcon regression both
> require higher version of Hadoop, it might be good to think about moving
> Falcon base to also default to Hadoop 2.7.1. Will create a separate JIRA for
> that.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
