[
https://issues.apache.org/jira/browse/BEAM-375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15349483#comment-15349483
]
Amit Sela commented on BEAM-375:
--------------------------------
But do we want Beam Hadoop components to support multiple Hadoop versions ?
with a default build like Spark does ?
Correct me if I'm wrong here, but going with the latest is not the best choice
anyway (unless there is a constraint) because AFAIK Hadoop is not something
users upgrade as fast as say, Spark/Flink versions.
WDYT ?
> HadoopIO and runners-spark conflict with hadoop.version
> -------------------------------------------------------
>
> Key: BEAM-375
> URL: https://issues.apache.org/jira/browse/BEAM-375
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-extensions
> Reporter: Pei He
> Assignee: Pei He
>
> HadoopIO currently uses 2.7.0 and runners-spark uses 2.2.0 for hadoop-client,
> hadoop-common.
> From [~amitsela]
> "Spark can be built against different hadoop versions, but the release in
> maven central is a 2.2.0 build (latest). ''
> For HadoopIO, I don't know why 2.7.0 is picked at the beginning. I can check
> if it will work with 2.2.0.
> I am creating this issue, since I think it there is a general question.
> In principle, HadoopIO and other sdks Sources should work with any runners.
> But, when one set of runners require version A, but the other set of runners
> require version B, we will need a general solution for it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
