Thanks for the detailed investigation Yuhong. This definitely sounds like a
bug; the code means to deduplicate identical paths, but uses only String
equality, not Path equality. I filed a JIRA issue in case someone wants to
work on fixing it: https://issues.apache.org/jira/browse/BEAM-11329
On Mon, Nov 23, 2020 at 5:32 PM 成雨虹 <mabelyuhong0...@gmail.com> wrote:
> Hi Beam,
>
> I am new to beam spark and recently got an error:
>
>
> Caused by: java.lang.IllegalArgumentException: The HadoopFileSystemRegistrar
> currently only supports at most a single Hadoop configuration.
>
> at
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument(Preconditions.java:141)
> ~[beam-vendor-guava-26_0-jre-0.1.jar:?]
> at
> org.apache.beam.sdk.io.hdfs.HadoopFileSystemRegistrar.fromOptions(HadoopFileSystemRegistrar.java:60)
> ~[beam-sdks-java-io-hadoop-file-system-3.2250.5.jar:?]
> at
> org.apache.beam.sdk.io.FileSystems.verifySchemesAreUnique(FileSystems.java:496)
> ~[beam-sdks-java-core-3.2250.5.jar:?]
> at
> org.apache.beam.sdk.io.FileSystems.setDefaultPipelineOptions(FileSystems.java:486)
> ~[beam-sdks-java-core-3.2250.5.jar:?]
> at
> org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:47)
> ~[beam-sdks-java-core-3.2250.5.jar:?]
> at org.apache.beam.sdk.Pipeline.create(Pipeline.java:149)
> ~[beam-sdks-java-core-3.2250.5.jar:?]
>
>
> I tried to debug and printed some logs using
>
> List<Configuration> configurations =
> pipelineOpts.as(HadoopFileSystemOptions.class).getHdfsConfiguration();
> LOG.info("print hdfsConfiguration for testing: " + configurations.toString());
>
>
> 2020-11-19 18:02:26.289 [main] HelloBeam [INFO] print hdfsConfiguration for
> testing:
>
> [Configuration:
> /export/content/lid/apps/samza-yarn-nodemanager/1d5c39c31bb33e3dd8e8149168167870328a014b/genConfig/core-site.xml,
>
> Configuration:
> /export/content/lid/apps/samza-yarn-nodemanager/1d5c39c31bb33e3dd8e8149168167870328a014b/genConfig/core-site.xml]
>
>
> as you can see the hdfsConfiguration is a list and contains two same
> elements, which caused the error.
>
> I noticed that the configurations are generated according to HADOOP_CONF_DIR
> and YARN_CONF_DIR. In the class, a set is used to dedup,
>
> however, in my test environment, the two dirs are:
>
>
> HADOOP_CONF_DIR=/export/content/lid/apps/samza-yarn-nodemanager/1d5c39c31bb33e3dd8e8149168167870328a014b/bin/../genConfig*/*
>
> YARN_CONF_DIR=/export/content/lid/apps/samza-yarn-nodemanager/1d5c39c31bb33e3dd8e8149168167870328a014b/bin/../genConfig
>
>
> HADOOP_CONF_DIR contains a '/' at the end so these two dir are considered to
> be different and then got added twice.
>
>
> I am not sure this is what we expected or is it a bug we should fix?
>
>
> Thanks in advance. Hope can hear from you soon.
>
>
> Best,
>
> Yuhong
>
>
>
>
