[
https://issues.apache.org/jira/browse/FLINK-2268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16152224#comment-16152224
]
ASF GitHub Bot commented on FLINK-2268:
---------------------------------------
GitHub user aljoscha opened a pull request:
https://github.com/apache/flink/pull/4636
[FLINK-2268] Allow Flink binary release without Hadoop
This is a series of PRs that allows running a Flink without any Hadoop
dependencies in the lib folder. Each PR stands on its own but all of them are
necessary for the last commit to work. The commit's themselves clearly document
what is changed.
R: @zentol
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/aljoscha/flink hadoop-free-flink
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4636.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4636
----
commit 178c227f9a78b6fa8eca89d93b79f47d6d2cfda5
Author: Aljoscha Krettek <[email protected]>
Date: 2017-08-21T17:55:57Z
[FLINK-4048] Remove Hadoop from DataSet API
This removes all Hadoop-related methods from ExecutionEnvironment (there
are already equivalent methods in flink-hadoop-compatibility (see
HadoopUtils and HadoopInputs, etc.). This also removes Hadoop-specific
tests from flink-tests because these are duplicated by tests in
flink-hadoop-compatibility.
This also removes Hadoop-specic example code from flink-examples: the
DistCp example and related code.
commit b0f76980ccab2cd4b742e78453b788aea93c9680
Author: Aljoscha Krettek <[email protected]>
Date: 2017-08-22T14:40:28Z
[FLINK-2268] Remove Hadoop-related Akka Serializers from runtime
commit 4af83b119473f0245ea42be34dcb44099fd7af19
Author: Aljoscha Krettek <[email protected]>
Date: 2017-08-22T14:42:22Z
[FLINK-2268] Remove unused HDFS copy-utils from flink-streaming-java
commit 524f30bc8ffe329b7b0559ad8499148bf9707f3b
Author: Aljoscha Krettek <[email protected]>
Date: 2017-08-22T14:44:22Z
[FLINK-2268] Don't use Hadoop Writable in JoinOperatorTest
commit da76ff8b5ec34b5e2d0d689892950262538bc384
Author: Aljoscha Krettek <[email protected]>
Date: 2017-08-22T14:46:20Z
[FLINK-2268] Don't use commons-io ByteArrayOutputStream in NFATest
commons-io is only usable as a transitive dependency of the Hadoop
dependencies. We can just use the Java ByteArrayOutputStream and
get rid of that dependency.
commit 5c9ee77075d43f6e7fe8b2ccad53f2c37ed896d6
Author: Aljoscha Krettek <[email protected]>
Date: 2017-08-22T14:47:36Z
[FLINK-4048] Remove Hadoop GenericOptionsParser from ParameterTool
There are methods for this in flink-hadoop-compatibility.
commit 04ebf521d616e9370430a72d7054c6862bd96c3e
Author: Aljoscha Krettek <[email protected]>
Date: 2017-08-22T14:50:18Z
[FLINK-2268] Don't use Hadoop FileSystem in RocksDB tests
This was in there because of legacy reasons but is not required by the
test.
commit 1c417fdcf8e7068a5eb6f8429e63067475bc6fc0
Author: Aljoscha Krettek <[email protected]>
Date: 2017-08-22T14:51:53Z
[FLINK-2268] Don't use jets3t in MesosArtifactServer
This was only used for the Enum for a specific http response type. The
jets3t dependency is only available as a transitive dependency of the
Hadoop dependencies, that's why we remove it.
commit 92d28b7182d428cfd5bd78cfa1f9cbbd521f95f0
Author: Aljoscha Krettek <[email protected]>
Date: 2017-08-22T14:52:28Z
[FLINK-2268] Only print Hadoop env info if Hadoop is in the classpath
commit 80fe2708c773666c3a25add470b34b528ea6c75d
Author: Aljoscha Krettek <[email protected]>
Date: 2017-08-22T14:54:33Z
[FLINK-2268] Close Hadoop FS reflectively in TestBaseUtils
This removes the dependency on Hadoop and ensures that we only close if
Hadoop is available.
commit c876012e81fbc6224c1ff036707a4dcc684266e9
Author: Aljoscha Krettek <[email protected]>
Date: 2017-08-24T12:38:02Z
[FLINK-2268] Remove Writable support from Scala TypeInformation Macro
commit 810fb147618678b70826129d1ba24d6b140d60dd
Author: Aljoscha Krettek <[email protected]>
Date: 2017-08-24T12:22:26Z
[FLINK-2268] Dynamically load Hadoop security module when available
commit 384620a07b8483f811e802a72b763a9ca2f20c0a
Author: Aljoscha Krettek <[email protected]>
Date: 2017-08-24T12:32:14Z
[FLINK-2268] Don't include Hadoop deps in flink-core/flink-java
This also makes them optional in flink-runtime, which is enabled by the
previous changes to only use Hadoop dependencies if they are available.
This also requires adding a few explicit dependencies in other modules
because they were using transitive dependencies of the Hadoop deps. The
most common dependency there is, ha!, commons-io.
commit 05497812bc5b8a7a8717ba8a7d052502b7d6386a
Author: Aljoscha Krettek <[email protected]>
Date: 2017-08-24T13:30:17Z
[FLINK-2268] Allow not including the Hadoop uber jar
----
> Provide Flink binary release without Hadoop
> -------------------------------------------
>
> Key: FLINK-2268
> URL: https://issues.apache.org/jira/browse/FLINK-2268
> Project: Flink
> Issue Type: Improvement
> Components: Build System
> Reporter: Robert Metzger
> Assignee: Aljoscha Krettek
>
> Currently, all Flink releases ship with Hadoop 2.3.0 binaries.
> The big Hadoop distributions are usually not relying on vanilla Hadoop
> releases, but on custom patched versions.
> To provide the best user experience, we should offer a Flink binary that uses
> the Hadoop jars provided by the user (=hadoop distribution)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
