Hi Hequn & Vino,
Finally I rebuild the Flink by change the “hadoop.version” in the pom
file.
Because Flink use maven shaded plugin to shade the Hadoop dependency,
this also means I need to rebuild the hadoop shaded jar each time I upgrade
Flink version.
Best
Henry
> 在 2018年10月30日，下午12:35，Hequn Cheng <[email protected]> 写道：
>
> Hi Henry,
>
> You can specify a specific Hadoop version to build against:
> mvn clean install -DskipTests -Dhadoop.version=2.6.1
> More details here[1].
>
> Best, Hequn
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-master/flinkDev/building.html#hadoop-versions
>
> <https://ci.apache.org/projects/flink/flink-docs-master/flinkDev/building.html#hadoop-versions>
> On Tue, Oct 30, 2018 at 10:02 AM vino yang <[email protected]
> <mailto:[email protected]>> wrote:
> Hi Henry,
>
> You just need to change the node of "hadoop.version" in the parent pom file.
>
> Thanks, vino.
>
> 徐涛 <[email protected] <mailto:[email protected]>> 于2018年10月29日周一
> 下午11:23写道：
> Hi Vino,
> Because I build the project with Maven, maybe I can not use the jars
> directly download from the web.
> If built with Maven, how can I adjust the Hadoop version with the
> Hadoop version really used?
> Thanks a lot!!
>
> Best
> Henry
>
>> 在 2018年10月26日，上午10:02，vino yang <[email protected]
>> <mailto:[email protected]>> 写道：
>>
>> Hi Henry,
>>
>> When running flink on YARN, from ClusterEntrypoint the system environment
>> info is print out.
>> One of the info is "Hadoop version: 2.4.1”, I think it is from the
>> flink-shaded-hadoop2 jar. But actually the system Hadoop version is 2.7.2.
>>
>> I want to know is it OK if the version is different?
>>
>> > I don't think it is OK, because you will use a lower version of the client
>> > to access the higher version of the server.
>>
>> Is it a best practice to adjust flink Hadoop version to the Hadoop version
>> actually used?
>>
>> > I personally recommend that you keep the two versions consistent to
>> > eliminate the possibility of causing various potential problems.
>> In fact, Flink provides a bundle of Hadoop 2.7.x bundles for you to
>> download.[1]
>>
>> [1]:
>> https://www.apache.org/dyn/closer.lua/flink/flink-1.6.1/flink-1.6.1-bin-hadoop27-scala_2.11.tgz
>>
>> <https://www.apache.org/dyn/closer.lua/flink/flink-1.6.1/flink-1.6.1-bin-hadoop27-scala_2.11.tgz>
>>
>> Thanks, vino.
>>
>> 徐涛 <[email protected] <mailto:[email protected]>> 于2018年10月26日周五
>> 上午9:13写道：
>> Hi Experts
>> When running flink on YARN, from ClusterEntrypoint the system
>> environment info is print out.
>> One of the info is "Hadoop version: 2.4.1”, I think it is from the
>> flink-shaded-hadoop2 jar. But actually the system Hadoop version is 2.7.2.
>> I want to know is it OK if the version is different? Is it a best
>> practice to adjust flink Hadoop version to the Hadoop version actually used?
>>
>> Thanks a lot.
>>
>> Best
>> Henry
>
