[
https://issues.apache.org/jira/browse/BEAM-375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15967528#comment-15967528
]
liyuntian edited comment on BEAM-375 at 4/13/17 12:45 PM:
----------------------------------------------------------
Does HdfsIO must use hadoop2.7.0 or above? I use Hadoop2.6.0 in our system,but
It is blocking ,when I execute "pipeline.run()" with sparkrunner . If I change
Hadoop version to 2.7.0, it can run very well with sparkrunner.So,I think I
must use Hadoop2.7.0 in beam0.5?
was (Author: yuntian):
Does HdfsIO must use hadoop2.7.0 or above? I use Hadoop2.6.0 in our system,but
It is blocking ,when I execute "pipeline.run()" with sparkrunner . If I change
Hadoop version to 2.7.0, it can run very well with sparkrunner.So,I think I
must use Hadoop2.7.0,yes?
> HadoopIO and runners-spark conflict with hadoop.version
> -------------------------------------------------------
>
> Key: BEAM-375
> URL: https://issues.apache.org/jira/browse/BEAM-375
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-extensions
> Reporter: Pei He
> Assignee: Pei He
>
> HadoopIO currently uses 2.7.0 and runners-spark uses 2.2.0 for hadoop-client,
> hadoop-common.
> From [~amitsela]
> "Spark can be built against different hadoop versions, but the release in
> maven central is a 2.2.0 build (latest). ''
> For HadoopIO, I don't know why 2.7.0 is picked at the beginning. I can check
> if it will work with 2.2.0.
> I am creating this issue, since I think it there is a general question.
> In principle, HadoopIO and other sdks Sources should work with any runners.
> But, when one set of runners require version A, but the other set of runners
> require version B, we will need a general solution for it.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
