[
https://issues.apache.org/jira/browse/BEAM-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15952151#comment-15952151
]
ASF GitHub Bot commented on BEAM-1856:
--------------------------------------
GitHub user 397090770 opened a pull request:
https://github.com/apache/beam/pull/2399
[BEAM-1856]
As described in
[BEAM-1856](https://issues.apache.org/jira/browse/BEAM-1856) `HDFSFileSink`
class do not use the same configuration in master thread and slave thread.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/397090770/beam master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/beam/pull/2399.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2399
----
commit d1526d49563bba4820a43ca8e08ee7e8d1845d07
Author: iteblog <[email protected]>
Date: 2017-04-01T09:41:05Z
BEAM-1856
BEAM-1856
----
> HDFSFileSink class do not use the same configuration in master and slave
> ------------------------------------------------------------------------
>
> Key: BEAM-1856
> URL: https://issues.apache.org/jira/browse/BEAM-1856
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
> Affects Versions: 0.6.0
> Reporter: yangping wu
> Assignee: Davor Bonaci
>
> I have a code snippet as follow:
> {code}
> Read.Bounded<KV<LongWritable, Text>> from =
> Read.from(HDFSFileSource.from(options.getInputFile(), TextInputFormat.class,
> LongWritable.class, Text.class));
> PCollection<KV<LongWritable, Text>> data = p.apply(from);
> data.apply(MapElements.via(new SimpleFunction<KV<LongWritable, Text>,
> String>() {
> @Override
> public String apply(KV<LongWritable, Text> input) {
> return input.getValue() + "\t" + input.getValue();
> }
> })).apply(Write.to(HDFSFileSink.<String>toText(options.getOutputFile())));
> {code}
> and submit job like this:
> {code}
> spark-submit --class org.apache.beam.examples.WordCountHDFS --master
> yarn-client \
> ./target/word-count-beam-bundled-0.1.jar
> \
> --runner=SparkRunner
> \
> --inputFile=hdfs://master/tmp/input/
> \
> --outputFile=/tmp/output/
> {code}
> Then {{HDFSFileSink.validate}} function will check whether the local
> filesystem (not HDFS) exists {{/tmp/output/}} directory.
> But the final result will store in {{hdfs://master/tmp/output/}} directory in
> HDFS filesystem.
> The reason is {{HDFSFileSink}} class do not use the same configuration in
> master thread and slave thread.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
