[
https://issues.apache.org/jira/browse/AIRFLOW-2488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829757#comment-16829757
]
Ash Berlin-Taylor edited comment on AIRFLOW-2488 at 4/29/19 10:01 PM:
----------------------------------------------------------------------
I've just landed AIRFLOW-4358 and AIRFLOW-4394 which takes the total time now
down to ~28min, and slowest individual tests are now:
[success] 4.34%
tests.test_impersonation.ImpersonationTest.test_impersonation_subdag:
{color:#FF0000}44.6452s{color}
[success] 2.93%
tests.executors.test_celery_executor.CeleryExecutorTest.test_celery_integration_1_redis_redis_6379_0:
{color:#FF0000}30.0785s{color}
[success] 2.36%
tests.operators.test_subdag_operator.SubDagOperatorTests.test_subdag_deadlock:
{color:red}24.2686s{color}
[success] 2.10%
tests.test_jobs.BackfillJobTest.test_run_ignores_all_dependencies:
{color:red}21.5965s{color}
[success] 1.96% tests.test_jobs.BackfillJobTest.test_backfill_run_backwards:
{color:red}20.1814s{color}
[success] 1.96%
tests.test_impersonation.ImpersonationTest.test_impersonation_custom:
{color:red}20.1730s{color}
[success] 1.96%
tests.executors.test_celery_executor.CeleryExecutorTest.test_celery_integration_0_amqp_guest_guest_rabbitmq_5672:
{color:red}20.1652s{color}
[success] 1.78% tests.cli.test_cli.TestCLI.test_next_execution:
{color:red}18.3203s{color}
[success] 1.47%
tests.sensors.test_timeout_sensor.SensorTimeoutTest.test_timeout:
{color:red}15.1303s{color}
[success] 1.27%
tests.transplant_class.<locals>.C.test_cli_webserver_shutdown_when_gunicorn_master_is_killed:
{color:red}13.0136s{color}
[success] 1.13%
tests.test_impersonation.ImpersonationTest.test_no_impersonation:
{color:red}11.6442s{color}
[success] 1.13% tests.test_impersonation.ImpersonationTest.test_impersonation:
{color:red}11.5675s{color}
[success] 1.13%
tests.test_impersonation.ImpersonationTest.test_default_impersonation:
{color:red}11.5617s{color}
The "unit" tests now run in 25mins and the kube tests take 28mins, so speeding
up those would be worth it
was (Author: ashb):
I've just landed AIRFLOW-4358 and AIRFLOW-4393 which takes the total time now
down to ~28min, and slowest individual tests are now:
[success] 4.34%
tests.test_impersonation.ImpersonationTest.test_impersonation_subdag:
{color:#FF0000}44.6452s{color}
[success] 2.93%
tests.executors.test_celery_executor.CeleryExecutorTest.test_celery_integration_1_redis_redis_6379_0:
{color:#FF0000}30.0785s{color}
[success] 2.36%
tests.operators.test_subdag_operator.SubDagOperatorTests.test_subdag_deadlock:
{color:red}24.2686s{color}
[success] 2.10%
tests.test_jobs.BackfillJobTest.test_run_ignores_all_dependencies:
{color:red}21.5965s{color}
[success] 1.96% tests.test_jobs.BackfillJobTest.test_backfill_run_backwards:
{color:red}20.1814s{color}
[success] 1.96%
tests.test_impersonation.ImpersonationTest.test_impersonation_custom:
{color:red}20.1730s{color}
[success] 1.96%
tests.executors.test_celery_executor.CeleryExecutorTest.test_celery_integration_0_amqp_guest_guest_rabbitmq_5672:
{color:red}20.1652s{color}
[success] 1.78% tests.cli.test_cli.TestCLI.test_next_execution:
{color:red}18.3203s{color}
[success] 1.47%
tests.sensors.test_timeout_sensor.SensorTimeoutTest.test_timeout:
{color:red}15.1303s{color}
[success] 1.27%
tests.transplant_class.<locals>.C.test_cli_webserver_shutdown_when_gunicorn_master_is_killed:
{color:red}13.0136s{color}
[success] 1.13%
tests.test_impersonation.ImpersonationTest.test_no_impersonation:
{color:red}11.6442s{color}
[success] 1.13% tests.test_impersonation.ImpersonationTest.test_impersonation:
{color:red}11.5675s{color}
[success] 1.13%
tests.test_impersonation.ImpersonationTest.test_default_impersonation:
{color:red}11.5617s{color}
The "unit" tests now run in 25mins and the kube tests take 28mins, so speeding
up those would be worth it
> Speed up the CI
> ---------------
>
> Key: AIRFLOW-2488
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2488
> Project: Apache Airflow
> Issue Type: Improvement
> Reporter: Fokko Driesprong
> Priority: Major
>
> Some of the tests take far to long:
> {noformat}
> [success] 10.48% tests.test_jobs.BackfillJobTest.test_backfill_examples:
> 177.9813s
> [success] 6.24% tests.test_jobs.BackfillJobTest.test_backfill_multi_dates:
> 106.0157s
> [success] 3.59% tests.transplant_class.<locals>.C.test_backfill: 61.0503s
> [success] 3.55%
> tests.contrib.sensors.test_python_sensor.PythonSensorTests.test_python_sensor_false:
> 60.2099s
> [success] 2.66%
> tests.test_impersonation.ImpersonationTest.test_impersonation_subdag: 45.1117s
> [success] 1.86%
> tests.test_jobs.SchedulerJobTest.test_dagrun_root_after_dagrun_unfinished:
> 31.5533s
> [success] 1.77%
> tests.executors.test_celery_executor.CeleryExecutorTest.test_celery_integration_1_redis_redis_6379_0:
> 30.0873s
> [success] 1.48%
> tests.operators.test_subdag_operator.SubDagOperatorTests.test_subdag_deadlock:
> 25.0747s
> [success] 1.40%
> tests.test_jobs.SchedulerJobTest.test_new_import_error_replaces_old: 23.7093s
> [success] 1.36% tests.test_jobs.SchedulerJobTest.test_dagrun_success: 23.1645s
> [success] 1.33%
> tests.test_jobs.BackfillJobTest.test_cli_backfill_depends_on_past_backwards:
> 22.5161s
> [success] 1.30%
> tests.test_jobs.SchedulerJobTest.test_dagrun_deadlock_ignore_depends_on_past:
> 22.1402s
> [success] 1.30%
> tests.test_jobs.SchedulerJobTest.test_dagrun_deadlock_ignore_depends_on_past_advance_ex_date:
> 22.1153s
> [success] 1.28%
> tests.test_jobs.BackfillJobTest.test_cli_backfill_depends_on_past: 21.7944s
> [success] 1.27%
> tests.test_jobs.SchedulerJobTest.test_dagrun_root_fail_unfinished: 21.5010s
> [success] 1.25%
> tests.test_jobs.BackfillJobTest.test_run_ignores_all_dependencies: 21.3037s
> [success] 1.25% tests.test_jobs.SchedulerJobTest.test_dagrun_root_fail:
> 21.1455s
> [success] 1.20%
> tests.test_impersonation.ImpersonationTest.test_impersonation_custom: 20.3233s
> [success] 1.19%
> tests.executors.test_celery_executor.CeleryExecutorTest.test_celery_integration_0_amqp_guest_guest_rabbitmq_5672:
> 20.1648s
> [success] 1.14% tests.test_jobs.BackfillJobTest.test_backfill_run_backwards:
> 19.4043s
> [success] 1.06% tests.cli.test_cli.TestCLI.test_next_execution: 17.9290s
> {noformat}
> (List updated April 23, 2019)
> These tests should be refactored to make them faster.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
