[
https://issues.apache.org/jira/browse/AIRFLOW-4382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jarek Potiuk resolved AIRFLOW-4382.
-----------------------------------
Resolution: Fixed
Fix Version/s: 2.0.0
Resolved by AIRFLOW-4401
> Limited Parallelism test is flaky
> ---------------------------------
>
> Key: AIRFLOW-4382
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4382
> Project: Apache Airflow
> Issue Type: Wish
> Components: tests
> Reporter: jack
> Priority: Major
> Fix For: 1.10.4, 2.0.0
>
>
> -The SFTPOperator tests seems to fail many times.-
> Limited parallelism test is flaky. AIRFLOW-4401 is the root cause.
> Previously the diagnosis of that test indicated that the problem is with
> SFTPOperator, but in fact all those log files there are expected. All the
> builds below fail because of limited parallelism test:
> This is the failure message:
>
> {code:java}
> ======================================================================
> 49) FAIL: test_execution_limited_parallelism
> (tests.executors.test_local_executor.LocalExecutorTest)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> tests/executors/test_local_executor.py line 77 in
> test_execution_limited_parallelism
> self.execution_parallelism(parallelism=test_parallelism)
> tests/executors/test_local_executor.py line 61 in execution_parallelism
> self.assertEqual(len(executor.running), 0)
> AssertionError: 1 != 0
>
> {code}
>
> Examples for PRs where this test fails:
> [https://travis-ci.org/apache/airflow/jobs/522789194]
> [https://travis-ci.org/apache/airflow/jobs/522931911]
> [https://travis-ci.org/apache/airflow/jobs/523348590]
>
> THIS IS NOT AN error: (mistakenly identified as one):
> {code:java}
>
> INFO [airflow.task] Dependencies all met for <TaskInstance:
> unit_teststest_schedule_dag_once.test_sftp 2019-04-22 06:02:24.255945+00:00
> [None]> INFO [airflow.task]
> --------------------------------------------------------------------------------
> INFO [airflow.task] Starting attempt 1 of 1 INFO [airflow.task]
> --------------------------------------------------------------------------------
> INFO [airflow.task] Executing <Task(SFTPOperator): test_sftp> on 2019-04-22
> 06:02:24.255945+00:00 INFO [paramiko.transport] Connected (version 2.0,
> client OpenSSH_7.2p2) INFO [paramiko.transport] Authentication (publickey)
> successful! INFO [paramiko.transport.sftp] [chan 0] Opened sftp connection
> (server version 3) INFO [airflow.task.operators] Starting to transfer from
> /tmp/test_remote_file to /tmp/tmp2/test_local_file ERROR [airflow.task] Error
> while transferring from /tmp/test_remote_file to /tmp/tmp2/test_local_file,
> error: [Errno 2] No such file or directory: '/tmp/tmp2/test_local_file'
> Traceback (most recent call last): File
> "/app/airflow/contrib/operators/sftp_operator.py", line 138, in execute
> sftp_client.get(self.remote_filepath, self.local_filepath) File
> "/app/.tox/py35-backend_sqlite-env_docker/lib/python3.5/site-packages/paramiko/sftp_client.py",
> line 801, in get with open(localpath, "wb") as fl: FileNotFoundError:
> [Errno 2] No such file or directory: '/tmp/tmp2/test_local_file' During
> handling of the above exception, another exception occurred: Traceback
> (most recent call last): File "/app/airflow/models/taskinstance.py", line
> 888, in _run_raw_task result = task_copy.execute(context=context) File
> "/app/airflow/contrib/operators/sftp_operator.py", line 155, in execute
> .format(file_msg, str(e))) airflow.exceptions.AirflowException: Error while
> transferring from /tmp/test_remote_file to /tmp/tmp2/test_local_file, error:
> [Errno 2] No such file or directory: '/tmp/tmp2/test_local_file'{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
