Bharath created BEAM-10592:
------------------------------
Summary: Issue after bumping Beam version from 2.13 to 2.19
Key: BEAM-10592
URL: https://issues.apache.org/jira/browse/BEAM-10592
Project: Beam
Issue Type: Bug
Components: io-java-hadoop-file-system
Affects Versions: 2.22.0, 2.19.0, 2.18.0
Environment: Azure Databricks
Reporter: Bharath
Bumping from Beam 2.13.0 to 2.19.0 and above we see broken pipelines running on
spark/HDFS.
Platfrom: Azure databricks.
Beam 2.13.0 works fine, Have issues only after migrating to 2.19 and above.
Caused by: java.io.IOException: Unable to rename resource
wasbs://*****/output/npstand75k0727_1/np/.temp-beam-64a00562-5dcd-4bcd-9c5a-be7cff1231f3/483d5498-ed9c-46fd-b1ce-8647fa5c8a06
to
wasbs://*******/output/npstand75k0727_1/np/confinements/part-00000-of-00001.txt.
No further information provided by underlying filesystem.Caused by:
java.io.IOException: Unable to rename resource
wasbs://****/output/npstand75k0727_1/np/.temp-beam-64a00562-5dcd-4bcd-9c5a-be7cff1231f3/483d5498-ed9c-46fd-b1ce-8647fa5c8a06
to
wasbs://****/output/npstand75k0727_1/np/confinements/part-00000-of-00001.txt.
No further information provided by underlying filesystem. at
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.rename(HadoopFileSystem.java:287)
at org.apache.beam.sdk.io.FileSystems.rename(FileSystems.java:327) at
org.apache.beam.sdk.io.FileBasedSink$WriteOperation.moveToOutputFiles(FileBasedSink.java:755)
at
org.apache.beam.sdk.io.WriteFiles$FinalizeTempFileBundles$FinalizeFn.process(WriteFiles.java:850)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
