ashb commented on issue #6627: [AIRFLOW-5931] Use os.fork when appropriate to speed up task execution. URL: https://github.com/apache/airflow/pull/6627#issuecomment-557132298 My benchmarking results of this change below. All my tests were of `BashOperator(..., bash_command="true")`. This change helps bring down the average (14.062799955 to 0.9318446075, x15 faster) and the maximum time (from 22.927918 to 1.865752, x12 faster!) ** Without change, 40 dag runs**: ``` $ psql airflow -c "select dag_id, task_id, execution_date, state, count(*), min(duration) as min_dur, avg(duration) as avg_dur, max(duration) as max_dur, sum(duration) as running_time, max(end_date)-min(start_date) as total_duration, avg(start_date - queued_dttm) as queue_delay from task_instance where state is not null group by grouping sets ((dag_id, task_id, state), (dag_id,execution_date, state), (dag_id, state)) order by 1,2,3" dag_id | task_id | execution_date | state | count | min_dur | avg_dur | max_dur | running_time | total_duration | queue_delay -------------+--------------+------------------------+---------+-------+-----------+--------------+-----------+--------------+-----------------+----------------- example_dag | print_date1 | | success | 10 | 13.540218 | 14.2083814 | 14.745729 | 142.083814 | 00:00:17.130217 | 00:00:14.927132 example_dag | print_date10 | | success | 10 | 11.480039 | 13.4748527 | 14.403048 | 134.748527 | 00:01:45.278854 | 00:09:42.822216 example_dag | print_date11 | | success | 10 | 10.084806 | 13.2927993 | 14.148362 | 132.927993 | 00:01:43.667161 | 00:09:44.954266 example_dag | print_date12 | | success | 10 | 7.529389 | 13.1063045 | 14.152888 | 131.063045 | 00:01:41.729084 | 00:09:48.473788 example_dag | print_date13 | | success | 10 | 13.352645 | 14.171775 | 15.294458 | 141.71775 | 00:05:34.40845 | 00:07:12.078185 example_dag | print_date14 | | success | 10 | 12.685846 | 14.1814867 | 15.92092 | 141.814867 | 00:05:54.331766 | 00:07:00.183005 example_dag | print_date15 | | success | 10 | 13.261651 | 13.6241582 | 14.4745 | 136.241582 | 00:06:01.605317 | 00:06:46.927206 example_dag | print_date16 | | success | 10 | 13.463467 | 13.9461268 | 14.327437 | 139.461268 | 00:06:03.748068 | 00:06:43.670468 example_dag | print_date17 | | success | 10 | 12.838915 | 13.7431276 | 14.127847 | 137.431276 | 00:06:16.971633 | 00:06:39.727049 example_dag | print_date18 | | success | 10 | 13.478566 | 16.0986632 | 19.783983 | 160.986632 | 00:06:39.378521 | 00:06:20.19122 example_dag | print_date19 | | success | 10 | 13.427385 | 15.7544223 | 22.927918 | 157.544223 | 00:06:40.99252 | 00:06:52.527796 example_dag | print_date2 | | success | 10 | 13.692921 | 14.3700994 | 14.829447 | 143.700994 | 00:00:46.73697 | 00:00:24.922475 example_dag | print_date20 | | success | 10 | 13.749788 | 15.9622433 | 21.03353 | 159.622433 | 00:06:46.79555 | 00:06:47.494261 example_dag | print_date21 | | success | 10 | 13.533584 | 15.3698928 | 20.041363 | 153.698928 | 00:07:01.543776 | 00:06:40.238809 example_dag | print_date22 | | success | 10 | 13.292851 | 14.3784663 | 15.897209 | 143.784663 | 00:07:07.038294 | 00:06:22.919986 example_dag | print_date23 | | success | 10 | 13.571403 | 14.7570801 | 16.540063 | 147.570801 | 00:06:58.41123 | 00:06:17.252477 example_dag | print_date24 | | success | 10 | 13.698992 | 14.5122418 | 17.094243 | 145.122418 | 00:06:51.823565 | 00:06:14.655425 example_dag | print_date25 | | success | 10 | 11.696784 | 13.6606921 | 14.262454 | 136.606921 | 00:07:14.91126 | 00:06:10.941776 example_dag | print_date26 | | success | 10 | 13.115977 | 13.8040275 | 14.581811 | 138.040275 | 00:07:22.865953 | 00:05:55.113162 example_dag | print_date27 | | success | 10 | 13.203122 | 13.8406964 | 14.061522 | 138.406964 | 00:07:21.852219 | 00:05:50.572565 example_dag | print_date28 | | success | 10 | 13.684101 | 14.1242237 | 14.625835 | 141.242237 | 00:07:52.563805 | 00:05:45.37103 example_dag | print_date29 | | success | 10 | 13.392751 | 13.8983641 | 14.356764 | 138.983641 | 00:07:43.298189 | 00:05:33.6447 example_dag | print_date3 | | success | 10 | 11.072175 | 13.7151706 | 14.351104 | 137.151706 | 00:00:47.050922 | 00:00:40.070479 example_dag | print_date30 | | success | 10 | 13.220995 | 13.6871487 | 14.455784 | 136.871487 | 00:08:08.386634 | 00:05:19.734116 example_dag | print_date31 | | success | 10 | 12.529552 | 13.7031299 | 14.13769 | 137.031299 | 00:08:00.575368 | 00:05:07.899519 example_dag | print_date32 | | success | 10 | 13.318386 | 14.0317772 | 14.622018 | 140.317772 | 00:08:23.871494 | 00:05:02.388743 example_dag | print_date33 | | success | 10 | 13.380067 | 14.1291853 | 14.981017 | 141.291853 | 00:08:24.882959 | 00:04:52.273283 example_dag | print_date34 | | success | 10 | 12.821078 | 13.7153179 | 14.443366 | 137.153179 | 00:08:25.688936 | 00:04:45.993094 example_dag | print_date35 | | success | 10 | 12.781473 | 13.671214 | 14.770308 | 136.71214 | 00:08:54.559094 | 00:04:38.23618 example_dag | print_date36 | | success | 10 | 13.293374 | 14.0982102 | 14.837228 | 140.982102 | 00:08:54.508936 | 00:04:27.575895 example_dag | print_date37 | | success | 10 | 13.657723 | 13.9828665 | 14.399264 | 139.828665 | 00:08:44.443426 | 00:04:24.720492 example_dag | print_date38 | | success | 10 | 12.415427 | 13.4307027 | 14.742275 | 134.307027 | 00:09:11.21312 | 00:04:07.207969 example_dag | print_date39 | | success | 10 | 12.977849 | 13.9103038 | 14.494109 | 139.103038 | 00:10:45.931352 | 00:06:18.113138 example_dag | print_date4 | | success | 10 | 13.027779 | 13.9621166 | 14.589971 | 139.621166 | 00:01:59.062323 | 00:09:15.243934 example_dag | print_date40 | | success | 10 | 12.899318 | 13.6532077 | 14.3119 | 136.532077 | 00:05:36.128265 | 00:07:45.713126 example_dag | print_date5 | | success | 10 | 13.364638 | 13.9671966 | 14.606845 | 139.671966 | 00:01:57.868484 | 00:09:15.995714 example_dag | print_date6 | | success | 10 | 13.454817 | 14.1199325 | 14.922081 | 141.199325 | 00:01:50.984789 | 00:09:18.331126 example_dag | print_date7 | | success | 10 | 13.316182 | 14.0112888 | 14.782732 | 140.112888 | 00:01:49.556653 | 00:09:20.466958 example_dag | print_date8 | | success | 10 | 10.542196 | 13.6500996 | 14.425894 | 136.500996 | 00:00:46.269235 | 00:00:15.901673 example_dag | print_date9 | | success | 10 | 3.733766 | 12.7930044 | 14.344923 | 127.930044 | 00:01:36.054977 | 00:09:49.170081 example_dag | | 2018-01-01 00:00:00+00 | success | 40 | 12.695903 | 14.51603295 | 20.041363 | 580.641318 | 00:13:04.943902 | 00:04:04.183146 example_dag | | 2018-01-01 00:05:00+00 | success | 40 | 11.072175 | 14.490386525 | 19.999951 | 579.615461 | 00:13:04.031953 | 00:04:01.462174 example_dag | | 2018-01-01 00:10:00+00 | success | 40 | 10.542196 | 14.67187005 | 22.927918 | 586.874802 | 00:12:53.483644 | 00:04:01.159616 example_dag | | 2018-01-01 00:15:00+00 | success | 40 | 11.696784 | 14.1309867 | 18.610719 | 565.239468 | 00:13:09.216955 | 00:04:06.713355 example_dag | | 2018-01-01 00:20:00+00 | success | 40 | 12.838915 | 13.97867895 | 19.082492 | 559.147158 | 00:12:32.573585 | 00:06:25.172815 example_dag | | 2018-01-01 00:25:00+00 | success | 40 | 3.733766 | 13.2246647 | 14.763435 | 528.986588 | 00:13:11.267078 | 00:07:08.041724 example_dag | | 2018-01-01 00:30:00+00 | success | 40 | 12.786672 | 13.91106905 | 14.638537 | 556.442762 | 00:12:38.990129 | 00:06:58.554466 example_dag | | 2018-01-01 00:35:00+00 | success | 40 | 12.415427 | 13.84971725 | 14.589971 | 553.98869 | 00:11:49.876577 | 00:06:52.566075 example_dag | | 2018-01-01 00:40:00+00 | success | 40 | 12.685846 | 13.92356865 | 14.829447 | 556.942746 | 00:12:01.941327 | 00:08:30.816995 example_dag | | 2018-01-01 00:45:00+00 | success | 40 | 13.027779 | 13.931024725 | 14.770308 | 557.240989 | 00:12:05.666341 | 00:08:49.990763 example_dag | | | success | 400 | 3.733766 | 14.062799955 | 22.927918 | 5625.119982 | 00:13:13.096719 | 00:06:05.866113 ``` **With change, 40 dag runs** ``` $ psql airflow -c "select dag_id, task_id, execution_date, state, count(*), min(duration) as min_dur, avg(duration) as avg_dur, max(duration) as max_dur, sum(duration) as running_time, max(end_date)-min(start_date) as total_duration, avg(start_date - queued_dttm) as queue_delay from task_instance where state is not null group by grouping sets ((dag_id, task_id, state), (dag_id,execution_date, state), (dag_id, state)) order by 1,2,3" dag_id | task_id | execution_date | state | count | min_dur | avg_dur | max_dur | running_time | total_duration | queue_delay -------------+--------------+------------------------+---------+-------+----------+--------------+----------+--------------+-----------------+----------------- example_dag | print_date1 | | success | 10 | 1.146504 | 1.4268349 | 1.694726 | 14.268349 | 00:00:02.171718 | 00:00:13.903464 example_dag | print_date10 | | success | 10 | 0.593307 | 0.8229632 | 1.042284 | 8.229632 | 00:00:41.72518 | 00:06:17.844413 example_dag | print_date11 | | success | 10 | 0.62847 | 0.8636334 | 1.173789 | 8.636334 | 00:00:40.747711 | 00:06:18.975095 example_dag | print_date12 | | success | 10 | 0.396318 | 0.8446297 | 1.207533 | 8.446297 | 00:00:38.101728 | 00:06:20.102727 example_dag | print_date13 | | success | 10 | 0.845284 | 1.0616951 | 1.417735 | 10.616951 | 00:03:26.884824 | 00:04:13.506178 example_dag | print_date14 | | success | 10 | 0.710254 | 0.9734616 | 1.304478 | 9.734616 | 00:03:30.503875 | 00:04:08.587778 example_dag | print_date15 | | success | 10 | 0.729193 | 0.9978077 | 1.659975 | 9.978077 | 00:03:34.779666 | 00:04:02.680916 example_dag | print_date16 | | success | 10 | 0.736481 | 1.1479496 | 1.660697 | 11.479496 | 00:03:38.17759 | 00:03:58.284443 example_dag | print_date17 | | success | 10 | 0.721791 | 1.089326 | 1.865752 | 10.89326 | 00:03:43.899201 | 00:03:52.045602 example_dag | print_date18 | | success | 10 | 0.82254 | 1.0861933 | 1.602212 | 10.861933 | 00:03:46.563687 | 00:03:45.934179 example_dag | print_date19 | | success | 10 | 0.694369 | 0.9150857 | 1.185493 | 9.150857 | 00:03:52.568222 | 00:04:02.353 example_dag | print_date2 | | success | 10 | 0.806452 | 1.0095171 | 1.230248 | 10.095171 | 00:00:04.82439 | 00:00:20.988461 example_dag | print_date20 | | success | 10 | 0.743303 | 0.8742865 | 0.983655 | 8.742865 | 00:03:53.99485 | 00:03:58.726793 example_dag | print_date21 | | success | 10 | 0.684588 | 0.8367918 | 1.023203 | 8.367918 | 00:03:55.738761 | 00:03:54.74774 example_dag | print_date22 | | success | 10 | 0.589924 | 0.8825712 | 1.029147 | 8.825712 | 00:03:58.548224 | 00:03:50.84068 example_dag | print_date23 | | success | 10 | 0.767636 | 0.8957238 | 0.986097 | 8.957238 | 00:04:04.217391 | 00:03:46.144873 example_dag | print_date24 | | success | 10 | 0.74223 | 0.9251518 | 1.239134 | 9.251518 | 00:04:05.765597 | 00:03:42.429237 example_dag | print_date25 | | success | 10 | 0.594681 | 0.8781399 | 1.121106 | 8.781399 | 00:04:05.490946 | 00:03:39.28868 example_dag | print_date26 | | success | 10 | 0.663151 | 0.8689899 | 1.052992 | 8.689899 | 00:04:09.399829 | 00:03:34.559888 example_dag | print_date27 | | success | 10 | 0.531863 | 0.8011121 | 1.065627 | 8.011121 | 00:04:13.508844 | 00:03:28.977669 example_dag | print_date28 | | success | 10 | 0.623941 | 0.8106821 | 1.027548 | 8.106821 | 00:04:14.959353 | 00:03:25.053474 example_dag | print_date29 | | success | 10 | 0.744605 | 0.8441154 | 0.964904 | 8.441154 | 00:04:11.377597 | 00:03:22.705902 example_dag | print_date3 | | success | 10 | 0.442053 | 0.9375549 | 1.567445 | 9.375549 | 00:00:15.142371 | 00:00:15.495181 example_dag | print_date30 | | success | 10 | 0.675152 | 0.8852712 | 1.107291 | 8.852712 | 00:04:14.102078 | 00:03:18.933734 example_dag | print_date31 | | success | 10 | 0.519751 | 0.843545 | 1.042148 | 8.43545 | 00:04:24.862378 | 00:03:12.841289 example_dag | print_date32 | | success | 10 | 0.483924 | 0.8193908 | 0.998283 | 8.193908 | 00:04:25.487382 | 00:03:08.866348 example_dag | print_date33 | | success | 10 | 0.562085 | 0.9288077 | 1.227881 | 9.288077 | 00:04:23.784358 | 00:03:06.368832 example_dag | print_date34 | | success | 10 | 0.744678 | 1.0033072 | 1.408322 | 10.033072 | 00:04:44.668717 | 00:02:55.888481 example_dag | print_date35 | | success | 10 | 0.652233 | 0.9915692 | 1.152128 | 9.915692 | 00:05:01.192531 | 00:02:48.305163 example_dag | print_date36 | | success | 10 | 0.553713 | 0.933974 | 1.250523 | 9.33974 | 00:05:02.069341 | 00:02:40.600184 example_dag | print_date37 | | success | 10 | 0.688596 | 1.0893281 | 1.532913 | 10.893281 | 00:05:20.416081 | 00:02:29.20499 example_dag | print_date38 | | success | 10 | 0.565303 | 0.9242697 | 1.503941 | 9.242697 | 00:06:15.347112 | 00:04:09.462437 example_dag | print_date39 | | success | 10 | 0.558524 | 0.8705852 | 1.049251 | 8.705852 | 00:03:20.690685 | 00:04:42.223374 example_dag | print_date4 | | success | 10 | 0.581263 | 0.895164 | 1.130376 | 8.95164 | 00:00:41.89009 | 00:05:49.37534 example_dag | print_date40 | | success | 10 | 0.65816 | 0.9504808 | 1.267472 | 9.504808 | 00:03:23.057506 | 00:04:39.000299 example_dag | print_date5 | | success | 10 | 0.615587 | 0.8474517 | 1.051941 | 8.474517 | 00:00:42.64173 | 00:05:50.834835 example_dag | print_date6 | | success | 10 | 0.675339 | 0.9569241 | 1.506232 | 9.569241 | 00:00:42.380933 | 00:05:51.989674 example_dag | print_date7 | | success | 10 | 0.7404 | 0.9408764 | 1.439553 | 9.408764 | 00:00:41.822069 | 00:05:53.02817 example_dag | print_date8 | | success | 10 | 0.514494 | 0.8535814 | 1.28338 | 8.535814 | 00:00:16.014045 | 00:00:12.423858 example_dag | print_date9 | | success | 10 | 0.354399 | 0.7450411 | 0.949153 | 7.450411 | 00:00:35.280334 | 00:06:20.324074 example_dag | | 2018-01-01 00:00:00+00 | success | 40 | 0.354399 | 0.903148125 | 1.621628 | 36.125925 | 00:07:43.652252 | 00:02:38.275005 example_dag | | 2018-01-01 00:05:00+00 | success | 40 | 0.627125 | 0.967237575 | 1.532913 | 38.689503 | 00:07:31.482163 | 00:02:36.899465 example_dag | | 2018-01-01 00:10:00+00 | success | 40 | 0.442053 | 0.94155605 | 1.52581 | 37.662242 | 00:07:09.337594 | 00:02:37.102932 example_dag | | 2018-01-01 00:15:00+00 | success | 40 | 0.520419 | 0.896940775 | 1.452449 | 35.877631 | 00:07:34.200548 | 00:02:41.155998 example_dag | | 2018-01-01 00:20:00+00 | success | 40 | 0.60063 | 0.878752225 | 1.230248 | 35.150089 | 00:07:14.674988 | 00:03:55.471032 example_dag | | 2018-01-01 00:25:00+00 | success | 40 | 0.554367 | 0.981219525 | 1.621297 | 39.248781 | 00:07:12.30197 | 00:04:14.648725 example_dag | | 2018-01-01 00:30:00+00 | success | 40 | 0.558524 | 0.9800735 | 1.660697 | 39.20294 | 00:07:18.737654 | 00:04:14.88136 example_dag | | 2018-01-01 00:35:00+00 | success | 40 | 0.483924 | 0.9204226 | 1.603143 | 36.816904 | 00:07:25.533756 | 00:04:15.256257 example_dag | | 2018-01-01 00:40:00+00 | success | 40 | 0.664791 | 0.977641 | 1.865752 | 39.10564 | 00:07:39.449438 | 00:05:15.616032 example_dag | | 2018-01-01 00:45:00+00 | success | 40 | 0.515129 | 0.8714547 | 1.282473 | 34.858188 | 00:07:40.570824 | 00:05:26.655057 example_dag | | | success | 400 | 0.354399 | 0.9318446075 | 1.865752 | 372.737843 | 00:07:44.01714 | 00:03:47.596186 (51 rows) ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
