GitHub user aljoscha opened a pull request: https://github.com/apache/flink/pull/4636
[FLINK-2268] Allow Flink binary release without Hadoop This is a series of PRs that allows running a Flink without any Hadoop dependencies in the lib folder. Each PR stands on its own but all of them are necessary for the last commit to work. The commit's themselves clearly document what is changed. R: @zentol You can merge this pull request into a Git repository by running: $ git pull https://github.com/aljoscha/flink hadoop-free-flink Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/4636.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4636 ---- commit 178c227f9a78b6fa8eca89d93b79f47d6d2cfda5 Author: Aljoscha Krettek <aljoscha.kret...@gmail.com> Date: 2017-08-21T17:55:57Z [FLINK-4048] Remove Hadoop from DataSet API This removes all Hadoop-related methods from ExecutionEnvironment (there are already equivalent methods in flink-hadoop-compatibility (see HadoopUtils and HadoopInputs, etc.). This also removes Hadoop-specific tests from flink-tests because these are duplicated by tests in flink-hadoop-compatibility. This also removes Hadoop-specic example code from flink-examples: the DistCp example and related code. commit b0f76980ccab2cd4b742e78453b788aea93c9680 Author: Aljoscha Krettek <aljoscha.kret...@gmail.com> Date: 2017-08-22T14:40:28Z [FLINK-2268] Remove Hadoop-related Akka Serializers from runtime commit 4af83b119473f0245ea42be34dcb44099fd7af19 Author: Aljoscha Krettek <aljoscha.kret...@gmail.com> Date: 2017-08-22T14:42:22Z [FLINK-2268] Remove unused HDFS copy-utils from flink-streaming-java commit 524f30bc8ffe329b7b0559ad8499148bf9707f3b Author: Aljoscha Krettek <aljoscha.kret...@gmail.com> Date: 2017-08-22T14:44:22Z [FLINK-2268] Don't use Hadoop Writable in JoinOperatorTest commit da76ff8b5ec34b5e2d0d689892950262538bc384 Author: Aljoscha Krettek <aljoscha.kret...@gmail.com> Date: 2017-08-22T14:46:20Z [FLINK-2268] Don't use commons-io ByteArrayOutputStream in NFATest commons-io is only usable as a transitive dependency of the Hadoop dependencies. We can just use the Java ByteArrayOutputStream and get rid of that dependency. commit 5c9ee77075d43f6e7fe8b2ccad53f2c37ed896d6 Author: Aljoscha Krettek <aljoscha.kret...@gmail.com> Date: 2017-08-22T14:47:36Z [FLINK-4048] Remove Hadoop GenericOptionsParser from ParameterTool There are methods for this in flink-hadoop-compatibility. commit 04ebf521d616e9370430a72d7054c6862bd96c3e Author: Aljoscha Krettek <aljoscha.kret...@gmail.com> Date: 2017-08-22T14:50:18Z [FLINK-2268] Don't use Hadoop FileSystem in RocksDB tests This was in there because of legacy reasons but is not required by the test. commit 1c417fdcf8e7068a5eb6f8429e63067475bc6fc0 Author: Aljoscha Krettek <aljoscha.kret...@gmail.com> Date: 2017-08-22T14:51:53Z [FLINK-2268] Don't use jets3t in MesosArtifactServer This was only used for the Enum for a specific http response type. The jets3t dependency is only available as a transitive dependency of the Hadoop dependencies, that's why we remove it. commit 92d28b7182d428cfd5bd78cfa1f9cbbd521f95f0 Author: Aljoscha Krettek <aljoscha.kret...@gmail.com> Date: 2017-08-22T14:52:28Z [FLINK-2268] Only print Hadoop env info if Hadoop is in the classpath commit 80fe2708c773666c3a25add470b34b528ea6c75d Author: Aljoscha Krettek <aljoscha.kret...@gmail.com> Date: 2017-08-22T14:54:33Z [FLINK-2268] Close Hadoop FS reflectively in TestBaseUtils This removes the dependency on Hadoop and ensures that we only close if Hadoop is available. commit c876012e81fbc6224c1ff036707a4dcc684266e9 Author: Aljoscha Krettek <aljoscha.kret...@gmail.com> Date: 2017-08-24T12:38:02Z [FLINK-2268] Remove Writable support from Scala TypeInformation Macro commit 810fb147618678b70826129d1ba24d6b140d60dd Author: Aljoscha Krettek <aljoscha.kret...@gmail.com> Date: 2017-08-24T12:22:26Z [FLINK-2268] Dynamically load Hadoop security module when available commit 384620a07b8483f811e802a72b763a9ca2f20c0a Author: Aljoscha Krettek <aljoscha.kret...@gmail.com> Date: 2017-08-24T12:32:14Z [FLINK-2268] Don't include Hadoop deps in flink-core/flink-java This also makes them optional in flink-runtime, which is enabled by the previous changes to only use Hadoop dependencies if they are available. This also requires adding a few explicit dependencies in other modules because they were using transitive dependencies of the Hadoop deps. The most common dependency there is, ha!, commons-io. commit 05497812bc5b8a7a8717ba8a7d052502b7d6386a Author: Aljoscha Krettek <aljoscha.kret...@gmail.com> Date: 2017-08-24T13:30:17Z [FLINK-2268] Allow not including the Hadoop uber jar ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---