[jira] [Commented] (AIRFLOW-7063) dag.clear() slowness caused by multiple UNION statements and tis.count()

2020-03-14 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-7063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17059319#comment-17059319 ] Qian Yu commented on AIRFLOW-7063: -- Here's a self-contained test to reproduce this slowness. 

[jira] [Updated] (AIRFLOW-7063) dag.clear() slowness caused by multiple UNION statements and tis.count()

2020-03-14 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-7063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu updated AIRFLOW-7063: - Description: When multiple {{ExternalTaskMarker}} are used, {{dag.clear()}} becomes very slow when

[jira] [Created] (AIRFLOW-7063) dag.clear() slowness caused by multiple UNION statements and tis.count()

2020-03-14 Thread Qian Yu (Jira)
Qian Yu created AIRFLOW-7063: Summary: dag.clear() slowness caused by multiple UNION statements and tis.count() Key: AIRFLOW-7063 URL: https://issues.apache.org/jira/browse/AIRFLOW-7063 Project: Apache

[jira] [Commented] (AIRFLOW-1230) Upstream_failed tasks are not executed when the DAG is restarted after failure

2020-02-28 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-1230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048223#comment-17048223 ] Qian Yu commented on AIRFLOW-1230: -- The problem is seems to be caused by what you are doing here:

[jira] [Updated] (AIRFLOW-6834) Fix flaky test test_scheduler_job.py::TestDagFileProcessor::test_dag_file_processor_process_task_instances_depends_on_past

2020-02-23 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu updated AIRFLOW-6834: - Description: test_scheduler_job.py has a few flaky tests. Some are marked with pytest.mark.xfail, but

[jira] [Assigned] (AIRFLOW-6834) Fix flaky test test_scheduler_job.py::TestDagFileProcessor::test_dag_file_processor_process_task_instances_depends_on_past

2020-02-19 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu reassigned AIRFLOW-6834: Assignee: Qian Yu > Fix flaky test >

[jira] [Commented] (AIRFLOW-6840) Bump up version of future to 0.18.2 in v-10-testing to get closer to python 3.8

2020-02-19 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039793#comment-17039793 ] Qian Yu commented on AIRFLOW-6840: -- Tagging some people who might know the answer: [~jlowin]

[jira] [Updated] (AIRFLOW-6840) Bump up version of future to 0.18.2 in v-10-testing to get closer to python 3.8

2020-02-19 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu updated AIRFLOW-6840: - Summary: Bump up version of future to 0.18.2 in v-10-testing to get closer to python 3.8 (was: Bump

[jira] [Created] (AIRFLOW-6840) Bump up version of future to 0.18.2 in v-10-testing

2020-02-19 Thread Qian Yu (Jira)
Qian Yu created AIRFLOW-6840: Summary: Bump up version of future to 0.18.2 in v-10-testing Key: AIRFLOW-6840 URL: https://issues.apache.org/jira/browse/AIRFLOW-6840 Project: Apache Airflow Issue

[jira] [Created] (AIRFLOW-6834) Fix flaky test test_scheduler_job.py::TestDagFileProcessor::test_dag_file_processor_process_task_instances_depends_on_past

2020-02-18 Thread Qian Yu (Jira)
Qian Yu created AIRFLOW-6834: Summary: Fix flaky test test_scheduler_job.py::TestDagFileProcessor::test_dag_file_processor_process_task_instances_depends_on_past Key: AIRFLOW-6834 URL:

[jira] [Closed] (AIRFLOW-5444) action_logging missing important fields such as Dag Id and Task Id for POST actions

2020-02-15 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu closed AIRFLOW-5444. > action_logging missing important fields such as Dag Id and Task Id for POST > actions >

[jira] [Created] (AIRFLOW-6800) SSHHook: Close file object when reading ssh config

2020-02-13 Thread Qian Yu (Jira)
Qian Yu created AIRFLOW-6800: Summary: SSHHook: Close file object when reading ssh config Key: AIRFLOW-6800 URL: https://issues.apache.org/jira/browse/AIRFLOW-6800 Project: Apache Airflow Issue

[jira] [Created] (AIRFLOW-6704) TaskInstance.operator is not set when task is marked success or failed in the Web UI

2020-02-01 Thread Qian Yu (Jira)
Qian Yu created AIRFLOW-6704: Summary: TaskInstance.operator is not set when task is marked success or failed in the Web UI Key: AIRFLOW-6704 URL: https://issues.apache.org/jira/browse/AIRFLOW-6704

[jira] [Assigned] (AIRFLOW-6704) TaskInstance.operator is not set when task is marked success or failed in the Web UI

2020-02-01 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu reassigned AIRFLOW-6704: Assignee: Qian Yu > TaskInstance.operator is not set when task is marked success or failed in

[jira] [Updated] (AIRFLOW-6704) TaskInstance.operator is not set when task is marked success or failed in the Web UI

2020-02-01 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu updated AIRFLOW-6704: - Issue Type: Bug (was: Improvement) > TaskInstance.operator is not set when task is marked success or

[jira] [Commented] (AIRFLOW-6657) Deprecate BaseBranchOperator

2020-02-01 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028041#comment-17028041 ] Qian Yu commented on AIRFLOW-6657: -- I renamed the JIRA to be "Deprecate BaseBranchOperator".  

[jira] [Updated] (AIRFLOW-6657) Deprecate BaseBranchOperator

2020-02-01 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu updated AIRFLOW-6657: - Description: This PR added a base class {{BaseBranchOperator}} that is intended to let people

[jira] [Updated] (AIRFLOW-6657) Deprecate BaseBranchOperator

2020-02-01 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu updated AIRFLOW-6657: - Summary: Deprecate BaseBranchOperator (was: Implement BranchPythonOperator as BaseBranchOperator) >

[jira] [Assigned] (AIRFLOW-5391) Clearing a task skipped by BranchPythonOperator will cause the task to execute

2020-01-28 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu reassigned AIRFLOW-5391: Assignee: Qian Yu > Clearing a task skipped by BranchPythonOperator will cause the task to

[jira] [Commented] (AIRFLOW-6657) Implement BranchPythonOperator as BaseBranchOperator

2020-01-28 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024976#comment-17024976 ] Qian Yu commented on AIRFLOW-6657: -- Okay nvm it does look very awkward to implement 

[jira] [Commented] (AIRFLOW-6657) Implement BranchPythonOperator as BaseBranchOperator

2020-01-28 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024954#comment-17024954 ] Qian Yu commented on AIRFLOW-6657: -- [~kaxilnaik] [~ash] [~chronitis]  It is not clear to me why the

[jira] [Created] (AIRFLOW-6657) Implement BranchPythonOperator as BaseBranchOperator

2020-01-28 Thread Qian Yu (Jira)
Qian Yu created AIRFLOW-6657: Summary: Implement BranchPythonOperator as BaseBranchOperator Key: AIRFLOW-6657 URL: https://issues.apache.org/jira/browse/AIRFLOW-6657 Project: Apache Airflow

[jira] [Commented] (AIRFLOW-6527) Error sending Celery task:Timeout in send_task_to_executor

2020-01-12 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013670#comment-17013670 ] Qian Yu commented on AIRFLOW-6527: -- [~kaxilnaik] [~kamil.bregula] I put up a PR here to make the

[jira] [Commented] (AIRFLOW-6527) Error sending Celery task:Timeout in send_task_to_executor

2020-01-09 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17012447#comment-17012447 ] Qian Yu commented on AIRFLOW-6527: -- Tagging some people who might know:

[jira] [Updated] (AIRFLOW-6527) Error sending Celery task:Timeout in send_task_to_executor

2020-01-09 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu updated AIRFLOW-6527: - Description: We use Airflow with CeleryExecutor and redis broker. Our airflow scheduler often

[jira] [Created] (AIRFLOW-6527) Error sending Celery task:Timeout in send_task_to_executor

2020-01-09 Thread Qian Yu (Jira)
Qian Yu created AIRFLOW-6527: Summary: Error sending Celery task:Timeout in send_task_to_executor Key: AIRFLOW-6527 URL: https://issues.apache.org/jira/browse/AIRFLOW-6527 Project: Apache Airflow

[jira] [Commented] (AIRFLOW-6497) Scheduler creates DagBag in the same process with outdated info

2020-01-07 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010303#comment-17010303 ] Qian Yu commented on AIRFLOW-6497: -- [~kamil.bregula] Do you want to create a PR? Or if you can give

[jira] [Assigned] (AIRFLOW-2279) Clearing Tasks Across DAGs

2020-01-07 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu reassigned AIRFLOW-2279: Assignee: Qian Yu > Clearing Tasks Across DAGs > -- > >

[jira] [Commented] (AIRFLOW-6497) Scheduler creates DagBag in the same process with outdated info

2020-01-06 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009401#comment-17009401 ] Qian Yu commented on AIRFLOW-6497: -- [~bolke] [~potiuk] [~kamil.bregula] any idea / suggestions? >

[jira] [Created] (AIRFLOW-6497) Scheduler creates DagBag in the same process with outdated info

2020-01-06 Thread Qian Yu (Jira)
Qian Yu created AIRFLOW-6497: Summary: Scheduler creates DagBag in the same process with outdated info Key: AIRFLOW-6497 URL: https://issues.apache.org/jira/browse/AIRFLOW-6497 Project: Apache Airflow

[jira] [Assigned] (AIRFLOW-6250) on_failure_callback does not know the task_id when handle_failure() is called without passing context

2019-12-12 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu reassigned AIRFLOW-6250: Assignee: Qian Yu > on_failure_callback does not know the task_id when handle_failure() is

[jira] [Created] (AIRFLOW-6250) on_failure_callback does not know the task_id when handle_failure() is called without passing context

2019-12-12 Thread Qian Yu (Jira)
Qian Yu created AIRFLOW-6250: Summary: on_failure_callback does not know the task_id when handle_failure() is called without passing context Key: AIRFLOW-6250 URL: https://issues.apache.org/jira/browse/AIRFLOW-6250

[jira] [Closed] (AIRFLOW-5648) Add ClearTaskOperator to allow clearing/re-running tasks from within a DAG

2019-11-22 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu closed AIRFLOW-5648. Resolution: Won't Fix See comments on the closed PR: [https://github.com/apache/airflow/pull/6392]  

[jira] [Created] (AIRFLOW-5840) Add a link to the external task that ExternalTaskSensor is waiting for

2019-11-03 Thread Qian Yu (Jira)
Qian Yu created AIRFLOW-5840: Summary: Add a link to the external task that ExternalTaskSensor is waiting for Key: AIRFLOW-5840 URL: https://issues.apache.org/jira/browse/AIRFLOW-5840 Project: Apache

[jira] [Commented] (AIRFLOW-2279) Clearing Tasks Across DAGs

2019-11-03 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16965639#comment-16965639 ] Qian Yu commented on AIRFLOW-2279: -- I'm in the process of working on something I call

[jira] [Assigned] (AIRFLOW-5648) Add ClearTaskOperator to allow clearing/re-running tasks from within a DAG

2019-10-17 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu reassigned AIRFLOW-5648: Assignee: Qian Yu > Add ClearTaskOperator to allow clearing/re-running tasks from within a DAG

[jira] [Updated] (AIRFLOW-5648) Add ClearTaskOperator to allow clearing/re-running tasks from within a DAG

2019-10-14 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu updated AIRFLOW-5648: - Description: There are use cases where some external conditions have changed and a section of the DAG

[jira] [Updated] (AIRFLOW-5648) Add ClearTaskOperator to allow clearing/re-running tasks from within a DAG

2019-10-14 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu updated AIRFLOW-5648: - Description: There are use cases where some external conditions have changed and a section of the DAG

[jira] [Updated] (AIRFLOW-5648) Add ClearTaskOperator to allow clearing/re-running tasks from within a DAG

2019-10-13 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu updated AIRFLOW-5648: - Labels: airflow clear duplicate operator re-run rerun task (was: clear duplicate operator re-run

[jira] [Updated] (AIRFLOW-5648) Add ClearTaskOperator to allow clearing/re-running tasks from within a DAG

2019-10-13 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu updated AIRFLOW-5648: - Labels: ClearTaskOperator airflow clear duplicate operator re-run rerun task (was: airflow clear

[jira] [Updated] (AIRFLOW-5648) Add ClearTaskOperator to allow clearing/re-running tasks from within a DAG

2019-10-13 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu updated AIRFLOW-5648: - Description: There are use cases where some external conditions have changed and a section of the DAG

[jira] [Created] (AIRFLOW-5648) Add ClearTaskOperator to allow clearing/re-running tasks from within a DAG

2019-10-13 Thread Qian Yu (Jira)
Qian Yu created AIRFLOW-5648: Summary: Add ClearTaskOperator to allow clearing/re-running tasks from within a DAG Key: AIRFLOW-5648 URL: https://issues.apache.org/jira/browse/AIRFLOW-5648 Project: Apache

[jira] [Commented] (AIRFLOW-2219) Race condition to DagRun.verify_integrity between Scheduler and Webserver

2019-09-23 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936370#comment-16936370 ] Qian Yu commented on AIRFLOW-2219: -- Also this looks related or the same as: 

[jira] [Commented] (AIRFLOW-2219) Race condition to DagRun.verify_integrity between Scheduler and Webserver

2019-09-23 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936368#comment-16936368 ] Qian Yu commented on AIRFLOW-2219: -- Just saw this error today too.   This JIRA is marked trivial

[jira] [Updated] (AIRFLOW-5462) Google Authentication redirection page crashes with KeyError: 'login'

2019-09-18 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu updated AIRFLOW-5462: - Priority: Minor (was: Major) > Google Authentication redirection page crashes with KeyError: 'login'

[jira] [Updated] (AIRFLOW-5410) AirflowConfigParser no longer find conf entries with capitalized letters

2019-09-18 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu updated AIRFLOW-5410: - Priority: Minor (was: Major) > AirflowConfigParser no longer find conf entries with capitalized

[jira] [Created] (AIRFLOW-5462) Google Authentication redirection page crashes with KeyError: 'login'

2019-09-11 Thread Qian Yu (Jira)
Qian Yu created AIRFLOW-5462: Summary: Google Authentication redirection page crashes with KeyError: 'login' Key: AIRFLOW-5462 URL: https://issues.apache.org/jira/browse/AIRFLOW-5462 Project: Apache

[jira] [Created] (AIRFLOW-5444) action_logging missing important fields such as Dag Id and Task Id for POST actions

2019-09-08 Thread Qian Yu (Jira)
Qian Yu created AIRFLOW-5444: Summary: action_logging missing important fields such as Dag Id and Task Id for POST actions Key: AIRFLOW-5444 URL: https://issues.apache.org/jira/browse/AIRFLOW-5444

[jira] [Commented] (AIRFLOW-4052) To allow filtering using "event" and "owner" in "Log" view

2019-09-08 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925288#comment-16925288 ] Qian Yu commented on AIRFLOW-4052: -- The "Fix Version" here is shown as 1.10.3, but clearly this is

[jira] [Created] (AIRFLOW-5410) AirflowConfigParser no longer find conf entries with capitalized letters

2019-09-05 Thread Qian Yu (Jira)
Qian Yu created AIRFLOW-5410: Summary: AirflowConfigParser no longer find conf entries with capitalized letters Key: AIRFLOW-5410 URL: https://issues.apache.org/jira/browse/AIRFLOW-5410 Project: Apache

[jira] [Updated] (AIRFLOW-5391) Clearing a task skipped by BranchPythonOperator will cause the task to execute

2019-09-03 Thread Qian Yu (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu updated AIRFLOW-5391: - Description: I tried this on 1.10.3 and 1.10.4, both have this issue:  E.g. in this example from the

[jira] [Created] (AIRFLOW-5391) Clearing a task skipped by BranchPythonOperator will cause the task to execute

2019-09-03 Thread Qian Yu (Jira)
Qian Yu created AIRFLOW-5391: Summary: Clearing a task skipped by BranchPythonOperator will cause the task to execute Key: AIRFLOW-5391 URL: https://issues.apache.org/jira/browse/AIRFLOW-5391 Project:

[jira] [Assigned] (AIRFLOW-4013) Mark success/failed is picking all execution date

2019-08-09 Thread Qian Yu (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-4013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Yu reassigned AIRFLOW-4013: Assignee: Qian Yu > Mark success/failed is picking all execution date >