[
https://issues.apache.org/jira/browse/BEAM-13706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chamikara Madhusanka Jayalath updated BEAM-13706:
-------------------------------------------------
Status: Open (was: Triage Needed)
> Beam IO WriteToText - FileNotFoundError
> ---------------------------------------
>
> Key: BEAM-13706
> URL: https://issues.apache.org/jira/browse/BEAM-13706
> Project: Beam
> Issue Type: Bug
> Components: io-py-files
> Affects Versions: 2.26.0
> Environment: RHEL 7 & python 3.6
> Reporter: Vinodraj Konda
> Priority: P2
>
> Hi Team,
> We are running a beam pipeline & writing the final output to a file as below.
> This runs fine most times but fails sometimes with error "FileNotFoundError:
> [Errno 2] No such file or directory". This has been intermittent and there is
> no issue with the file location, other jobs using the same location run fine
> at the same time & this too works fine if we immediately re-try. Please check
> and advise, I couldn't find any solution/details for this online. Thanks in
> advance.
>
> | 'write to file' >> beam.io.WriteToText(self.output_file,
>
> header=','.join(output_cols_header),
> shard_name_template=''
> )
>
> +*ERROR:*+
> [2022-01-17 15:20:43,512] \{bash_operator.py:126} INFO - Traceback (most
> recent call last):
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
> "apache_beam/runners/common.py", line 1213, in
> apache_beam.runners.common.DoFnRunner.process
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
> "apache_beam/runners/common.py", line 742, in
> apache_beam.runners.common.PerWindowInvoker.invoke_process
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
> "apache_beam/runners/common.py", line 867, in
> apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
> "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/iobase.py",
> line 1148, in process
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - writer =
> self.sink.open_writer(init_result, str(uuid.uuid4()))
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
> "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/options/value_provider.py",
> line 135, in _f
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - return fnc(self,
> *args, **kwargs)
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
> "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/filebasedsink.py",
> line 196, in open_writer
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - return
> FileBasedSinkWriter(self, writer_path)
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
> "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/filebasedsink.py",
> line 417, in __init__
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - self.temp_handle
> = self.sink.open(temp_shard_path)
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
> "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/textio.py",
> line 405, in open
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - file_handle =
> super(_TextSink, self).open(temp_path)
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
> "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/options/value_provider.py",
> line 135, in _f
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - return fnc(self,
> *args, **kwargs)
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
> "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/filebasedsink.py",
> line 138, in open
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - return
> FileSystems.create(temp_path, self.mime_type, self.compression_type)
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
> "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/filesystems.py",
> line 229, in create
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - return
> filesystem.create(path, mime_type, compression_type)
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
> "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/localfilesystem.py",
> line 168, in create
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - return
> self._path_open(path, 'wb', mime_type, compression_type)
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
> "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/localfilesystem.py",
> line 143, in _path_open
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - raw_file =
> io.open(path, mode)
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - FileNotFoundError:
> [Errno 2] No such file or directory:
> '/dropbox/weather/prd/kumulus/data/earthsat/daily/2022/01/v2/beam-temp-earthsat_daily_4cast_20220117.csv-0163f65477a911ecb8d80aec9c027dc6/da21a024-9253-4500-9d10-576bdb6a4711.earthsat_daily_4cast_20220117.csv'
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
