Unsubscribe
Jay Linux <jaylinuxg...@gmail.com> schrieb am Mi. 12. Juli 2023 um 14:04:
> Hello,
>
> I'm running Airflow v2.5.3 on a Dockerized setup. Here I have a strange
> issue with the SSH timeout for a rsync command.
>
> Here not sure only for the first try I'm getting this SSH timeout error.
> after the first retry there is no timeout error. This happens for all the
> runs. Please advise.
>
> I can conform there is no network issue. And I suspect the webserver and
> scheduler container are in unhealthy state.
>
>
>
> ===
> rsync_work_to_scratch_command = """
> rsync -av "{{ dag_run.conf["work_dir"] }}/" "{{
> dag_run.conf["scratch_dir"] }}/"
> """
>
> rsync_work_to_scratch_task = SSHOperator(
> task_id='rsync_work_to_scratch',
> ssh_hook=ssh_hook,
> command=rsync_work_to_scratch_command,
> on_success_callback=None,
> on_failure_callback=update_failure,
> retries=15,
> dag=dag
> )
> ===
>
> Error log for the SSH timeout.
> ===
> [2023-07-12T14:47:46.179+0400] {ssh.py:516} INFO -
> Data/Intensities/BaseCalls/L001/C1.1/L001_2.cbcl
> [2023-07-12T14:47:56.202+0400] {taskinstance.py:1776} ERROR - Task failed
> with exception
> Traceback (most recent call last):
> File
> "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/ssh/operators/ssh.py",
> line 173, in execute
> result = self.run_ssh_client_command(ssh_client, self.command,
> context=context)
> File
> "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/ssh/operators/ssh.py",
> line 159, in run_ssh_client_command
> ssh_client, command, environment=self.environment, get_pty=self.get_pty
> File
> "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/ssh/hooks/ssh.py",
> line 541, in exec_ssh_client_command
> raise AirflowException("SSH command timed out")
> airflow.exceptions.AirflowException: SSH command timed out
> [2023-07-12T14:47:56.216+0400] {taskinstance.py:1332} INFO - Marking task
> as UP_FOR_RETRY. dag_id=10X_sequence, task_id=rsync_work_to_scratch,
> execution_date=20230712T104644, start_date=20230712T104647,
> end_date=20230712T104756
> [2023-07-12T14:47:56.230+0400] {standard_task_runner.py:105} ERROR -
> Failed to execute job 82 for task rsync_work_to_scratch (SSH command timed
> out; 12951)
> [2023-07-12T14:47:56.260+0400] {local_task_job.py:212} INFO - Task exited
> with return code 1
> [2023-07-12T14:47:56.339+0400] {taskinstance.py:2596} INFO - 0 downstream
> tasks scheduled from follow-on schedule check
> ====
>
> Thanks
> Jay
>
--
christoph.gmei...@gmail.com
