MorganRoff-UnlikelyAI opened a new issue, #28603: URL: https://github.com/apache/beam/issues/28603
Hello there, I have a Beam job running on Spark that reads data from S3. I have tried using the [S3FileSystem](https://github.com/apache/beam/blob/master/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/s3/S3FileSystem.java) to read an s3:// path directly, but found this to be much slower than when I use s3a:// via HDFS in a non-Beam job. I believe it should be possible to read s3a:// paths using the [HadoopFileSystem](https://github.com/apache/beam/blob/master/sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystem.java), but I can't seem to get this working. If I include the org.apache.beam:beam-sdks-java-io-hadoop-file-system dependency, I still see this error: ``` java.lang.IllegalArgumentException: No filesystem found for scheme s3a at org.apache.beam.sdk.io.FileSystems.getFileSystemInternal(FileSystems.java:515) ``` This is in an environment where I know the org.apache.hadoop:hadoop-aws dependency is already included (AWS EMR), and s3a:// paths work out-of-the-box with a plain Spark job. To be safe, I tried also including org.apache.hadoop:hadoop-aws and org.apache.hadoop:hadoop-client directly, as recommended in the [Hadoop docs](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#Getting_Started), but that still gave the same errors as above and below. From looking at the [HadoopFileSystemRegistrar](https://github.com/apache/beam/blob/master/sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemRegistrar.java), it looks like the only way to register a custom scheme is to use an option like `--hdfsConfiguration=[{\"fs.default.name\":\"s3a://{bucket_path}\"}]` to get the scheme registered, but this still results in an error: ``` org.apache.beam.sdk.util.UserCodeException: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "s3a" at org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:39) at org.apache.beam.sdk.io.FileIO$MatchAll$MatchFn$DoFnInvoker.invokeProcessElement(Unknown Source) ``` Has anyone managed to make this work? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
