damccorm opened a new issue, #21319:
URL: https://github.com/apache/beam/issues/21319
Hi Team,
We are running a beam pipeline & writing the final output to a file as
below. This runs fine most times but fails sometimes with error
"FileNotFoundError: [Errno 2] No such file or directory". This has been
intermittent and there is no issue with the file location, other jobs using the
same location run fine at the same time & this too works fine if we immediately
re-try. Please check and advise, I couldn't find any solution/details for this
online. Thanks in advance.
| 'write to file' \>\> beam.io.WriteToText(self.output_file,
header=','.join(output_cols_header),
shard_name_template=''
)
*****ERROR:*****
[2022-01-17 15:20:43,512] \{bash_operator.py:126} INFO - Traceback (most
recent call last):
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
"apache_beam/runners/common.py", line 1213, in
apache_beam.runners.common.DoFnRunner.process
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
"apache_beam/runners/common.py", line 742, in
apache_beam.runners.common.PerWindowInvoker.invoke_process
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
"apache_beam/runners/common.py", line 867, in
apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
"/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/iobase.py",
line 1148, in process
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - writer =
self.sink.open_writer(init_result, str(uuid.uuid4()))
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
"/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/options/value_provider.py",
line 135, in _f
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - return
fnc(self, *args, **kwargs)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
"/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/filebasedsink.py",
line 196, in open_writer
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - return
FileBasedSinkWriter(self, writer_path)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
"/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/filebasedsink.py",
line 417, in __init__
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO -
self.temp_handle = self.sink.open(temp_shard_path)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
"/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/textio.py",
line 405, in open
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - file_handle =
super(_TextSink, self).open(temp_path)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
"/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/options/value_provider.py",
line 135, in _f
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - return
fnc(self, *args, **kwargs)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
"/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/filebasedsink.py",
line 138, in open
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - return
FileSystems.create(temp_path, self.mime_type, self.compression_type)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
"/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/filesystems.py",
line 229, in create
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - return
filesystem.create(path, mime_type, compression_type)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
"/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/localfilesystem.py",
line 168, in create
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - return
self._path_open(path, 'wb', mime_type, compression_type)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
"/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/localfilesystem.py",
line 143, in _path_open
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - raw_file =
io.open(path, mode)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - FileNotFoundError:
[Errno 2] No such file or directory:
'/dropbox/weather/prd/kumulus/data/earthsat/daily/2022/01/v2/beam-temp-earthsat_daily_4cast_20220117.csv-0163f65477a911ecb8d80aec9c027dc6/da21a024-9253-4500-9d10-576bdb6a4711.earthsat_daily_4cast_20220117.csv'
Imported from Jira
[BEAM-13706](https://issues.apache.org/jira/browse/BEAM-13706). Original Jira
may contain additional context.
Reported by: vinodraj.konda.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
