[jira] [Commented] (AIRFLOW-6920) AIRFLOW Feature Parity with LUIGI & CONTROLM

[ https://issues.apache.org/jira/browse/AIRFLOW-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17292260#comment-17292260 ] t oo commented on AIRFLOW-6920: --- I was using LocalExecutor > AIRFLOW Feature Parity with LUIGI &

[jira] [Updated] (AIRFLOW-6920) AIRFLOW Feature Parity with LUIGI & CONTROLM

[ https://issues.apache.org/jira/browse/AIRFLOW-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6920: -- Description: *LUIGI* vs *AIRFLOW*   200 sequential tasks (so no parallelism):   +LUIGI:+ mkdir -p

[jira] [Updated] (AIRFLOW-6920) AIRFLOW Feature Parity with LUIGI & CONTROLM

[ https://issues.apache.org/jira/browse/AIRFLOW-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6920: -- Description: *LUIGI* vs *AIRFLOW*   200 sequential tasks (so no parallelism):   +LUIGI:+ mkdir -p

[jira] [Updated] (AIRFLOW-6920) AIRFLOW Feature Parity with LUIGI & CONTROLM

[ https://issues.apache.org/jira/browse/AIRFLOW-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6920: -- Description: *LUIGI* vs *AIRFLOW*   200 sequential tasks (so no parallelism):   +LUIGI:+ mkdir -p

[jira] [Updated] (AIRFLOW-6920) AIRFLOW Feature Parity with LUIGI & CONTROLM

[ https://issues.apache.org/jira/browse/AIRFLOW-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6920: -- Description: *LUIGI* vs *AIRFLOW*   200 sequential tasks (so no parallelism):   +LUIGI:+ mkdir -p

[jira] [Updated] (AIRFLOW-6920) AIRFLOW Feature Parity with LUIGI & CONTROLM

[ https://issues.apache.org/jira/browse/AIRFLOW-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6920: -- Description: *LUIGI* vs *AIRFLOW*   200 sequential tasks (so no parallelism):   +LUIGI:+ mkdir -p

[jira] [Commented] (AIRFLOW-4481) DummyOperator performance improvement - currently needs +87 seconds to do nothing

[ https://issues.apache.org/jira/browse/AIRFLOW-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164348#comment-17164348 ] t oo commented on AIRFLOW-4481: --- still seeing 6.88 seconds for Success state and 25.08 seconds for

[jira] [Commented] (AIRFLOW-7113) Airflow failed to render gantt view due to missing start_date

[ https://issues.apache.org/jira/browse/AIRFLOW-7113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17080634#comment-17080634 ] t oo commented on AIRFLOW-7113: --- [~Cong Zhu] pr is merged so can close this? > Airflow failed to render

[jira] [Work started] (AIRFLOW-4785) Config flag to exclude dummyoperator from having Unknown 'Dependencies Blocking Task From Getting Scheduled'

[ https://issues.apache.org/jira/browse/AIRFLOW-4785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-4785 started by t oo. - > Config flag to exclude dummyoperator from having Unknown 'Dependencies > Blocking Task

[jira] [Commented] (AIRFLOW-6157) Separate out executor protocol

[ https://issues.apache.org/jira/browse/AIRFLOW-6157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073448#comment-17073448 ] t oo commented on AIRFLOW-6157: --- use case "is there a way to set executor per task? ie I want all tasks

[jira] [Commented] (AIRFLOW-6515) log level of INFO/WARN when ERROR happened

[ https://issues.apache.org/jira/browse/AIRFLOW-6515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072817#comment-17072817 ] t oo commented on AIRFLOW-6515: --- sounds good! > log level of INFO/WARN when ERROR happened >

[jira] [Commented] (AIRFLOW-7116) Tasks get stuck in the "removed" state

[ https://issues.apache.org/jira/browse/AIRFLOW-7116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072442#comment-17072442 ] t oo commented on AIRFLOW-7116: --- see if https://github.com/apache/airflow/pull/6954 helps > Tasks get

[jira] [Commented] (AIRFLOW-6515) log level of INFO/WARN when ERROR happened

[ https://issues.apache.org/jira/browse/AIRFLOW-6515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072439#comment-17072439 ] t oo commented on AIRFLOW-6515: --- yes > log level of INFO/WARN when ERROR happened >

[jira] [Commented] (AIRFLOW-5454) security - hide all password/secret/credentials/tokens from log

[ https://issues.apache.org/jira/browse/AIRFLOW-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071886#comment-17071886 ] t oo commented on AIRFLOW-5454: --- sure, u can > security - hide all password/secret/credentials/tokens

[jira] [Commented] (AIRFLOW-6420) ui - show summary of task/dagrun stats across ALL dags

[ https://issues.apache.org/jira/browse/AIRFLOW-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066587#comment-17066587 ] t oo commented on AIRFLOW-6420: --- 1. sum the totals from across all dags 2. not since webserver up, just

[jira] [Commented] (AIRFLOW-6518) Task did not retry when there was temporary metastore db connectivity loss

[ https://issues.apache.org/jira/browse/AIRFLOW-6518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17059645#comment-17059645 ] t oo commented on AIRFLOW-6518: --- depends on what level of fault-tolerance we want to achieve from

[jira] [Created] (AIRFLOW-7052) spark 3.0.0 does not work with sparksubmitoperator

t oo created AIRFLOW-7052: - Summary: spark 3.0.0 does not work with sparksubmitoperator Key: AIRFLOW-7052 URL: https://issues.apache.org/jira/browse/AIRFLOW-7052 Project: Apache Airflow Issue Type:

[jira] [Updated] (AIRFLOW-6994) SparkSubmitOperator re launches spark driver even when original driver still running

[ https://issues.apache.org/jira/browse/AIRFLOW-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6994: -- Fix Version/s: 1.10.10 > SparkSubmitOperator re launches spark driver even when original driver still >

[jira] [Work started] (AIRFLOW-6994) SparkSubmitOperator re launches spark driver even when original driver still running

[ https://issues.apache.org/jira/browse/AIRFLOW-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-6994 started by t oo. - > SparkSubmitOperator re launches spark driver even when original driver still > running >

[jira] [Updated] (AIRFLOW-6994) SparkSubmitOperator re launches spark driver even when original driver still running

[ https://issues.apache.org/jira/browse/AIRFLOW-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6994: -- Affects Version/s: (was: 1.10.6) 1.10.8 1.10.9 >

[jira] [Updated] (AIRFLOW-6994) SparkSubmitOperator re launches spark driver even when original driver still running

[ https://issues.apache.org/jira/browse/AIRFLOW-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6994: -- Fix Version/s: (was: 1.10.8) > SparkSubmitOperator re launches spark driver even when original driver

[jira] [Updated] (AIRFLOW-6994) SparkSubmitOperator re launches spark driver even when original driver still running

[ https://issues.apache.org/jira/browse/AIRFLOW-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6994: -- Description: https://issues.apache.org/jira/browse/AIRFLOW-6229 introduced a bug Due to temporary network

[jira] [Created] (AIRFLOW-6994) SparkSubmitOperator re launches spark driver even when original driver still running

t oo created AIRFLOW-6994: - Summary: SparkSubmitOperator re launches spark driver even when original driver still running Key: AIRFLOW-6994 URL: https://issues.apache.org/jira/browse/AIRFLOW-6994 Project:

[jira] [Commented] (AIRFLOW-6747) UI - Show count of tasks in each dag on the main dags page

[ https://issues.apache.org/jira/browse/AIRFLOW-6747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17049657#comment-17049657 ] t oo commented on AIRFLOW-6747: --- as part of this could use the new column to indicate whether a dagid is

[jira] [Comment Edited] (AIRFLOW-6747) UI - Show count of tasks in each dag on the main dags page

[ https://issues.apache.org/jira/browse/AIRFLOW-6747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048645#comment-17048645 ] t oo edited comment on AIRFLOW-6747 at 3/1/20 5:56 PM: --- [~maubeh1] you can

[jira] [Commented] (AIRFLOW-6747) UI - Show count of tasks in each dag on the main dags page

[ https://issues.apache.org/jira/browse/AIRFLOW-6747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048645#comment-17048645 ] t oo commented on AIRFLOW-6747: --- [~maubeh1] you can contact me if u get stuck > UI - Show count of tasks

[jira] [Commented] (AIRFLOW-6747) UI - Show count of tasks in each dag on the main dags page

[ https://issues.apache.org/jira/browse/AIRFLOW-6747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048527#comment-17048527 ] t oo commented on AIRFLOW-6747: --- you can [~maubeh1] > UI - Show count of tasks in each dag on the main

[jira] [Created] (AIRFLOW-6934) max_active_runs from different dag in dagbag stopping any task from running

t oo created AIRFLOW-6934: - Summary: max_active_runs from different dag in dagbag stopping any task from running Key: AIRFLOW-6934 URL: https://issues.apache.org/jira/browse/AIRFLOW-6934 Project: Apache

[jira] [Commented] (AIRFLOW-6389) add config for 'allow_multi_scheduler_instances' default True

[ https://issues.apache.org/jira/browse/AIRFLOW-6389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17045895#comment-17045895 ] t oo commented on AIRFLOW-6389: --- t https://jira.apache.org/jira/browse/AIRFLOW-6389 any suggestions on

[jira] [Updated] (AIRFLOW-6389) add config for 'allow_multi_scheduler_instances' default True

[ https://issues.apache.org/jira/browse/AIRFLOW-6389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6389: -- Description: right now common deployment pattern with blue/green build is: 1. on EC2 1, start scheduler 2.

[jira] [Closed] (AIRFLOW-4502) new cli command - task_states_for_dag_run

[ https://issues.apache.org/jira/browse/AIRFLOW-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo closed AIRFLOW-4502. - Resolution: Fixed > new cli command - task_states_for_dag_run > - > >

[jira] [Created] (AIRFLOW-6920) AIRFLOW Feature Parity with LUIGI & CONTROLM

t oo created AIRFLOW-6920: - Summary: AIRFLOW Feature Parity with LUIGI & CONTROLM Key: AIRFLOW-6920 URL: https://issues.apache.org/jira/browse/AIRFLOW-6920 Project: Apache Airflow Issue Type:

[jira] [Updated] (AIRFLOW-6389) add config for 'allow_multi_scheduler_instances' default True

[ https://issues.apache.org/jira/browse/AIRFLOW-6389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6389: -- Description: right now common deployment pattern with blue/green build is: 1. on EC2 1, start scheduler 2.

[jira] [Updated] (AIRFLOW-6389) add config for 'allow_multi_scheduler_instances' default True

[ https://issues.apache.org/jira/browse/AIRFLOW-6389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6389: -- Description: right now common deployment pattern with blue/green build is: 1. on EC2 1, start scheduler 2.

[jira] [Updated] (AIRFLOW-6388) SparkSubmitOperator polling should not 'consume' a pool slot

[ https://issues.apache.org/jira/browse/AIRFLOW-6388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6388: -- Summary: SparkSubmitOperator polling should not 'consume' a pool slot (was: SparkSubmitOperator polling

[jira] [Updated] (AIRFLOW-6443) Pool name - increase length > 50, make cli give error if too large

[ https://issues.apache.org/jira/browse/AIRFLOW-6443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6443: -- Labels: gsoc gsoc2020 mentor pool (was: pool) > Pool name - increase length > 50, make cli give error if

[jira] [Commented] (AIRFLOW-6518) Task did not retry when there was temporary metastore db connectivity loss

[ https://issues.apache.org/jira/browse/AIRFLOW-6518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17044969#comment-17044969 ] t oo commented on AIRFLOW-6518: --- [~ash] [~potiuk] thoughts? > Task did not retry when there was

[jira] [Updated] (AIRFLOW-6747) UI - Show count of tasks in each dag on the main dags page

[ https://issues.apache.org/jira/browse/AIRFLOW-6747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6747: -- Labels: gsoc gsoc2020 mentor (was: ) > UI - Show count of tasks in each dag on the main dags page >

[jira] [Updated] (AIRFLOW-6918) don't use 'is' in if conditions comparing STATE

[ https://issues.apache.org/jira/browse/AIRFLOW-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6918: -- Description: Trivial code tidy-up - 'is' usually is used for testing booleans or None presence. This change

[jira] [Updated] (AIRFLOW-6918) don't use 'is' in if conditions comparing STATE

[ https://issues.apache.org/jira/browse/AIRFLOW-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6918: -- Priority: Trivial (was: Major) > don't use 'is' in if conditions comparing STATE >

[jira] [Work started] (AIRFLOW-6918) don't use 'is' in if conditions comparing STATE

[ https://issues.apache.org/jira/browse/AIRFLOW-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-6918 started by t oo. - > don't use 'is' in if conditions comparing STATE >

[jira] [Updated] (AIRFLOW-6918) don't use 'is' in if conditions comparing STATE

[ https://issues.apache.org/jira/browse/AIRFLOW-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6918: -- Summary: don't use 'is' in if conditions comparing STATE (was: don't use is in if comparing state) > don't

[jira] [Created] (AIRFLOW-6918) don't use is in if comparing state

t oo created AIRFLOW-6918: - Summary: don't use is in if comparing state Key: AIRFLOW-6918 URL: https://issues.apache.org/jira/browse/AIRFLOW-6918 Project: Apache Airflow Issue Type: Improvement

[jira] [Created] (AIRFLOW-6747) UI - Show count of tasks in each dag on the main dags page

t oo created AIRFLOW-6747: - Summary: UI - Show count of tasks in each dag on the main dags page Key: AIRFLOW-6747 URL: https://issues.apache.org/jira/browse/AIRFLOW-6747 Project: Apache Airflow

[jira] [Updated] (AIRFLOW-6474) list_dag_runs cli command should allow exec_date between start/end range and print start/end times

[ https://issues.apache.org/jira/browse/AIRFLOW-6474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6474: -- Description: 1. accept argument exec_date_from, exec_date_to to filter execution_dates returned, ie show

[jira] [Closed] (AIRFLOW-6356) make cli only print output relevant to dag argument passed in rather than from any random dag in bag

[ https://issues.apache.org/jira/browse/AIRFLOW-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo closed AIRFLOW-6356. - Resolution: Fixed > make cli only print output relevant to dag argument passed in rather than > from any

[jira] [Closed] (AIRFLOW-6387) print details of success/skipped task

[ https://issues.apache.org/jira/browse/AIRFLOW-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo closed AIRFLOW-6387. - Resolution: Fixed > print details of success/skipped task > - > >

[jira] [Closed] (AIRFLOW-6473) Show conf in response of dag_state cli command

[ https://issues.apache.org/jira/browse/AIRFLOW-6473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo closed AIRFLOW-6473. - Resolution: Fixed > Show conf in response of dag_state cli command >

[jira] [Comment Edited] (AIRFLOW-6518) Task did not retry when there was temporary metastore db connectivity loss

[ https://issues.apache.org/jira/browse/AIRFLOW-6518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016625#comment-17016625 ] t oo edited comment on AIRFLOW-6518 at 1/16/20 7:27 AM: some options: 1.

[jira] [Comment Edited] (AIRFLOW-6518) Task did not retry when there was temporary metastore db connectivity loss

[ https://issues.apache.org/jira/browse/AIRFLOW-6518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016625#comment-17016625 ] t oo edited comment on AIRFLOW-6518 at 1/16/20 7:24 AM: some options: 1.

[jira] [Commented] (AIRFLOW-6518) Task did not retry when there was temporary metastore db connectivity loss

[ https://issues.apache.org/jira/browse/AIRFLOW-6518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016625#comment-17016625 ] t oo commented on AIRFLOW-6518: --- some options: 1. introduce db retry config -

[jira] [Work started] (AIRFLOW-4355) Externally triggered DAG is marked as 'success' even if a task has been 'removed'!

[ https://issues.apache.org/jira/browse/AIRFLOW-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-4355 started by t oo. - > Externally triggered DAG is marked as 'success' even if a task has been > 'removed'! >

[jira] [Work started] (AIRFLOW-6473) Show conf in response of dag_state cli command

[ https://issues.apache.org/jira/browse/AIRFLOW-6473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-6473 started by t oo. - > Show conf in response of dag_state cli command >

[jira] [Work started] (AIRFLOW-6562) mushroom cloud error when clicking 'mark failed/success' from graph view of dag that has never been run yet

[ https://issues.apache.org/jira/browse/AIRFLOW-6562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-6562 started by t oo. - > mushroom cloud error when clicking 'mark failed/success' from graph view of > dag that

[jira] [Created] (AIRFLOW-6562) mushroom cloud error when clicking 'mark failed/success' from graph view of dag that has never been run yet

t oo created AIRFLOW-6562: - Summary: mushroom cloud error when clicking 'mark failed/success' from graph view of dag that has never been run yet Key: AIRFLOW-6562 URL: https://issues.apache.org/jira/browse/AIRFLOW-6562

[jira] [Updated] (AIRFLOW-6562) mushroom cloud error when clicking 'mark failed/success' from graph view of dag that has never been run yet

[ https://issues.apache.org/jira/browse/AIRFLOW-6562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6562: -- Description: # create a new dag # go to graph view # click on one of the tasks (it should have a white

[jira] [Work started] (AIRFLOW-6555) mushroom cloud error when clicking 'task instance details' from graph view of dag that has never been run yet

[ https://issues.apache.org/jira/browse/AIRFLOW-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-6555 started by t oo. - > mushroom cloud error when clicking 'task instance details' from graph view of > dag that

[jira] [Created] (AIRFLOW-6555) mushroom cloud error when clicking 'task instance details' from graph view of dag that has never been run yet

t oo created AIRFLOW-6555: - Summary: mushroom cloud error when clicking 'task instance details' from graph view of dag that has never been run yet Key: AIRFLOW-6555 URL: https://issues.apache.org/jira/browse/AIRFLOW-6555

[jira] [Updated] (AIRFLOW-6469) Enforce dagrun_timeout for non-scheduled (ie externally triggered) dags without # of active DagRuns == max_active_runs limitation

[ https://issues.apache.org/jira/browse/AIRFLOW-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6469: -- Description: dagrun_timeout is very limited right now see

[jira] [Updated] (AIRFLOW-6469) Enforce dagrun_timeout for non-scheduled (ie externally triggered) dags without # of active DagRuns == max_active_runs limitation

[ https://issues.apache.org/jira/browse/AIRFLOW-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6469: -- Description: dagrun_timeout is very limited right now see https://github.com/apache/airflow/pull/4782/files

[jira] [Updated] (AIRFLOW-6469) Enforce dagrun_timeout for non-scheduled (ie externally triggered) dags without # of active DagRuns == max_active_runs limitation

[ https://issues.apache.org/jira/browse/AIRFLOW-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6469: -- Description: dagrun_timeout is very limited right now see https://github.com/apache/airflow/pull/4782/files

[jira] [Work started] (AIRFLOW-6251) add config for max tasks per dag to prevent scheduler denial of service

[ https://issues.apache.org/jira/browse/AIRFLOW-6251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-6251 started by t oo. - > add config for max tasks per dag to prevent scheduler denial of service >

[jira] [Updated] (AIRFLOW-6251) add config for max tasks per dag to prevent scheduler denial of service

[ https://issues.apache.org/jira/browse/AIRFLOW-6251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6251: -- Summary: add config for max tasks per dag to prevent scheduler denial of service (was: add config for max

[jira] [Updated] (AIRFLOW-6251) add config for max tasks per dag to prevent scheduler denial of service

[ https://issues.apache.org/jira/browse/AIRFLOW-6251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6251: -- Description: airflow is often shared between users of varying experience levels. to prevent a novice user

[jira] [Created] (AIRFLOW-6538) prevent autocomplete of username in login UI

t oo created AIRFLOW-6538: - Summary: prevent autocomplete of username in login UI Key: AIRFLOW-6538 URL: https://issues.apache.org/jira/browse/AIRFLOW-6538 Project: Apache Airflow Issue Type:

[jira] [Updated] (AIRFLOW-6454) add test for time taken by scheduler to run dag of diff num of tasks (2 vs 20 vs 200 vs 2000 vs 20000 simple 1 line print tasks)

[ https://issues.apache.org/jira/browse/AIRFLOW-6454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6454: -- Description: *LUIGI* vs *AIRFLOW*   200 sequential tasks (so no parallelism):   +LUIGI:+ mkdir -p

[jira] [Updated] (AIRFLOW-6454) add test for time taken by scheduler to run dag of diff num of tasks (2 vs 20 vs 200 vs 2000 vs 20000 simple 1 line print tasks)

[ https://issues.apache.org/jira/browse/AIRFLOW-6454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6454: -- Description: *LUIGI* vs *AIRFLOW*   200 sequential tasks (so no parallelism):   +LUIGI:+ mkdir -p

[jira] [Work started] (AIRFLOW-6451) self._print_stat() in dag_processing.py should be skippable by config option

[ https://issues.apache.org/jira/browse/AIRFLOW-6451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-6451 started by t oo. - > self._print_stat() in dag_processing.py should be skippable by config option >

[jira] [Commented] (AIRFLOW-6524) Make trigger_dag & clear CLIs throw an error if 'shape' of dag has changed since prior dagrun

[ https://issues.apache.org/jira/browse/AIRFLOW-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013024#comment-17013024 ] t oo commented on AIRFLOW-6524: --- it doesn't, none of the tasks get into running state > Make trigger_dag

[jira] [Commented] (AIRFLOW-6454) add test for time taken by scheduler to run dag of diff num of tasks (2 vs 20 vs 200 vs 2000 vs 20000 simple 1 line print tasks)

[ https://issues.apache.org/jira/browse/AIRFLOW-6454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013021#comment-17013021 ] t oo commented on AIRFLOW-6454: --- I did run against 1.10.7, 160ms for a single task is fine, just the

[jira] [Updated] (AIRFLOW-6388) SparkSubmitOperator polling should not 'consume' a slot

[ https://issues.apache.org/jira/browse/AIRFLOW-6388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6388: -- Description: Spark jobs can often take many minutes (or even hours) to complete. The spark submit operator

[jira] [Updated] (AIRFLOW-6454) add test for time taken by scheduler to run dag of diff num of tasks (2 vs 20 vs 200 vs 2000 vs 20000 simple 1 line print tasks)

[ https://issues.apache.org/jira/browse/AIRFLOW-6454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6454: -- Description: *LUIGI* vs *AIRFLOW*   200 sequential tasks (so no parallelism):   +LUIGI:+ mkdir -p

[jira] [Updated] (AIRFLOW-6454) add test for time taken by scheduler to run dag of diff num of tasks (2 vs 20 vs 200 vs 2000 vs 20000 simple 1 line print tasks)

[ https://issues.apache.org/jira/browse/AIRFLOW-6454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6454: -- Description: *LUIGI* vs *AIRFLOW*   200 sequential tasks (so no parallelism):   +LUIGI:+ mkdir -p

[jira] [Updated] (AIRFLOW-6454) add test for time taken by scheduler to run dag of diff num of tasks (2 vs 20 vs 200 vs 2000 vs 20000 simple 1 line print tasks)

[ https://issues.apache.org/jira/browse/AIRFLOW-6454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6454: -- Description: *LUIGI* vs *AIRFLOW*   200 sequential tasks (so no parallelism):   +LUIGI:+ mkdir -p

[jira] [Updated] (AIRFLOW-6454) add test for time taken by scheduler to run dag of diff num of tasks (2 vs 20 vs 200 vs 2000 vs 20000 simple 1 line print tasks)

[ https://issues.apache.org/jira/browse/AIRFLOW-6454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6454: -- Description: *LUIGI* vs *AIRFLOW*   200 sequential tasks (so no parallelism):   +LUIGI:+ mkdir -p

[jira] [Updated] (AIRFLOW-6454) add test for time taken by scheduler to run dag of diff num of tasks (2 vs 20 vs 200 vs 2000 vs 20000 simple 1 line print tasks)

[ https://issues.apache.org/jira/browse/AIRFLOW-6454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6454: -- Description: *LUIGI* vs *AIRFLOW*   200 sequential tasks (so no parallelism):   +LUIGI:+ mkdir -p

[jira] [Updated] (AIRFLOW-6454) add test for time taken by scheduler to run dag of diff num of tasks (2 vs 20 vs 200 vs 2000 vs 20000 simple 1 line print tasks)

[ https://issues.apache.org/jira/browse/AIRFLOW-6454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6454: -- Description: *LUIGI* vs *AIRFLOW* 200 sequential tasks: LUIGI: mkdir -p test_output8 #*8.3secs* total time

[jira] [Created] (AIRFLOW-6524) Make trigger_dag & clear CLIs throw an error if 'shape' of dag has changed since prior dagrun

t oo created AIRFLOW-6524: - Summary: Make trigger_dag & clear CLIs throw an error if 'shape' of dag has changed since prior dagrun Key: AIRFLOW-6524 URL: https://issues.apache.org/jira/browse/AIRFLOW-6524

[jira] [Created] (AIRFLOW-6518) Task did not retry when there was temporary metastore db connectivity loss

t oo created AIRFLOW-6518: - Summary: Task did not retry when there was temporary metastore db connectivity loss Key: AIRFLOW-6518 URL: https://issues.apache.org/jira/browse/AIRFLOW-6518 Project: Apache

[jira] [Created] (AIRFLOW-6515) log level of INFO/WARN when ERROR happened

t oo created AIRFLOW-6515: - Summary: log level of INFO/WARN when ERROR happened Key: AIRFLOW-6515 URL: https://issues.apache.org/jira/browse/AIRFLOW-6515 Project: Apache Airflow Issue Type: Bug

[jira] [Updated] (AIRFLOW-6515) log level of INFO/WARN when ERROR happened

[ https://issues.apache.org/jira/browse/AIRFLOW-6515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6515: -- Description: log level should be error on some (but there are false positives): grep -iE

[jira] [Work started] (AIRFLOW-6451) self._print_stat() in dag_processing.py should be skippable by config option

[ https://issues.apache.org/jira/browse/AIRFLOW-6451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-6451 started by t oo. - > self._print_stat() in dag_processing.py should be skippable by config option >

[jira] [Created] (AIRFLOW-6503) support more schedule expressions (ie last day of the month)

t oo created AIRFLOW-6503: - Summary: support more schedule expressions (ie last day of the month) Key: AIRFLOW-6503 URL: https://issues.apache.org/jira/browse/AIRFLOW-6503 Project: Apache Airflow

[jira] [Updated] (AIRFLOW-6501) UI - show next_execution date, first exec_date, creation date (ie timestamp when first parsed into dagbag)

[ https://issues.apache.org/jira/browse/AIRFLOW-6501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6501: -- Summary: UI - show next_execution date, first exec_date, creation date (ie timestamp when first parsed into

[jira] [Created] (AIRFLOW-6501) UI - show next_execution date

t oo created AIRFLOW-6501: - Summary: UI - show next_execution date Key: AIRFLOW-6501 URL: https://issues.apache.org/jira/browse/AIRFLOW-6501 Project: Apache Airflow Issue Type: New Feature

[jira] [Updated] (AIRFLOW-6419) dag_processor_manager/webserver/scheduler logs should be created under date folder

[ https://issues.apache.org/jira/browse/AIRFLOW-6419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6419: -- Description: dag level logs are written under separate date folders. This is great because the old dates

[jira] [Updated] (AIRFLOW-6451) self._print_stat() in dag_processing.py should be skippable by config option

[ https://issues.apache.org/jira/browse/AIRFLOW-6451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6451: -- Description: perf benefit clean up extra poll, logs, typos was:perf benefit > self._print_stat() in

[jira] [Work started] (AIRFLOW-6452) scheduler_job.py - remove excess sleep/log/duration calls

[ https://issues.apache.org/jira/browse/AIRFLOW-6452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-6452 started by t oo. - > scheduler_job.py - remove excess sleep/log/duration calls >

[jira] [Resolved] (AIRFLOW-4428) Externally triggered DagRun marked as 'success' state but all tasks for it are 'None' state

[ https://issues.apache.org/jira/browse/AIRFLOW-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo resolved AIRFLOW-4428. --- Fix Version/s: 2.0.0 Resolution: Fixed > Externally triggered DagRun marked as 'success' state but

[jira] [Closed] (AIRFLOW-6437) sql filters - remove in (NULL)

[ https://issues.apache.org/jira/browse/AIRFLOW-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo closed AIRFLOW-6437. - Fix Version/s: master 2.0.0 Resolution: Fixed > sql filters - remove in (NULL) >

[jira] [Created] (AIRFLOW-6478) new cli command to set taskinstance state

t oo created AIRFLOW-6478: - Summary: new cli command to set taskinstance state Key: AIRFLOW-6478 URL: https://issues.apache.org/jira/browse/AIRFLOW-6478 Project: Apache Airflow Issue Type: New

[jira] [Created] (AIRFLOW-6477) new cli command for dagrun operations

t oo created AIRFLOW-6477: - Summary: new cli command for dagrun operations Key: AIRFLOW-6477 URL: https://issues.apache.org/jira/browse/AIRFLOW-6477 Project: Apache Airflow Issue Type: New Feature

[jira] [Created] (AIRFLOW-6474) list_dag_runs cli command should allow exec_date between start/end range and print start/end times

t oo created AIRFLOW-6474: - Summary: list_dag_runs cli command should allow exec_date between start/end range and print start/end times Key: AIRFLOW-6474 URL: https://issues.apache.org/jira/browse/AIRFLOW-6474

[jira] [Commented] (AIRFLOW-3128) SparkSqlOperator prints blank info endlessly after SparkContext stopped

[ https://issues.apache.org/jira/browse/AIRFLOW-3128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17008401#comment-17008401 ] t oo commented on AIRFLOW-3128: --- AIRFLOW-5339 fixed similar issue > SparkSqlOperator prints blank info

[jira] [Commented] (AIRFLOW-1220) Can't clear airflow jobs when using schedule_interval of None

[ https://issues.apache.org/jira/browse/AIRFLOW-1220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17008399#comment-17008399 ] t oo commented on AIRFLOW-1220: --- no longer an issue, can u close [~ssen-ichain] ? > Can't clear airflow

[jira] [Commented] (AIRFLOW-2721) Add config param in the DagRun UI

[ https://issues.apache.org/jira/browse/AIRFLOW-2721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17008394#comment-17008394 ] t oo commented on AIRFLOW-2721: --- [~niranda] can u update the title to 'allow passing in conf when

[jira] [Created] (AIRFLOW-6473) Show conf in response of dag_state cli command

t oo created AIRFLOW-6473: - Summary: Show conf in response of dag_state cli command Key: AIRFLOW-6473 URL: https://issues.apache.org/jira/browse/AIRFLOW-6473 Project: Apache Airflow Issue Type:

[jira] [Updated] (AIRFLOW-6369) clear cli command needs a 'conf' option

[ https://issues.apache.org/jira/browse/AIRFLOW-6369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6369: -- Summary: clear cli command needs a 'conf' option (was: clear command needs a 'conf' option) > clear cli

[jira] [Created] (AIRFLOW-6469) Enforce dagrun_timeout for non-scheduled (ie externally triggered) dags without # of active DagRuns == max_active_runs limitation

t oo created AIRFLOW-6469: - Summary: Enforce dagrun_timeout for non-scheduled (ie externally triggered) dags without # of active DagRuns == max_active_runs limitation Key: AIRFLOW-6469 URL:

[jira] [Created] (AIRFLOW-6468) fix install on python 3.7

t oo created AIRFLOW-6468: - Summary: fix install on python 3.7 Key: AIRFLOW-6468 URL: https://issues.apache.org/jira/browse/AIRFLOW-6468 Project: Apache Airflow Issue Type: New Feature

  1   2   3   4   >