[jira] [Resolved] (AIRFLOW-5811) Add metric for externally killed task count
[ https://issues.apache.org/jira/browse/AIRFLOW-5811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-5811. --- Fix Version/s: 1.10.7 Resolution: Fixed > Add metric for externally killed task count > --- > > Key: AIRFLOW-5811 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5811 > Project: Apache Airflow > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.0.0 >Reporter: QP Hou >Assignee: QP Hou >Priority: Trivial > Fix For: 1.10.7 > > > Number of externally killed task would be very useful for monitoring init > container error when using k8s executor. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5562) Skip grant single DAG permissions for Admin role
[ https://issues.apache.org/jira/browse/AIRFLOW-5562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-5562. --- Fix Version/s: 1.10.7 Assignee: Liu Xuesi Resolution: Fixed > Skip grant single DAG permissions for Admin role > > > Key: AIRFLOW-5562 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5562 > Project: Apache Airflow > Issue Type: Improvement > Components: security >Affects Versions: 1.10.3, 1.10.4, 1.10.5 >Reporter: Liu Xuesi >Assignee: Liu Xuesi >Priority: Major > Labels: security, security-groups > Fix For: 1.10.7 > > Attachments: admin_permission_full_of_dags.jpg > > Original Estimate: 168h > Remaining Estimate: 168h > > From AIRFLOW-2267，there is a function named *update_admin_perm_view* will > refresh admin permission then add ALL permission to Admin role. > But, DAG level access make each DAG a MenuView, these views will be grant to > Admin role. As Admin role already have access to *all_dags*, these > permissions actually make Admin role's permission more chaotic. > In my project, it is hard to check permissions in webUI and actually this > lead to some performance issues. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5732) Add unit test for the version command
[
https://issues.apache.org/jira/browse/AIRFLOW-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Feng resolved AIRFLOW-5732.
---
Fix Version/s: 1.10.7
Resolution: Fixed
> Add unit test for the version command
> -
>
> Key: AIRFLOW-5732
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5732
> Project: Apache Airflow
> Issue Type: Test
> Components: cli, tests
>Affects Versions: 1.10.5
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Minor
> Fix For: 1.10.7
>
>
> It seems there's no unit test for the {{airflow version}} command. That's a
> trivial test but it'd be better if it is for avoiding an accidental
> regression.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5715) Make email, owner context available
[ https://issues.apache.org/jira/browse/AIRFLOW-5715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-5715. --- Fix Version/s: 2.0.0 Resolution: Fixed > Make email, owner context available > --- > > Key: AIRFLOW-5715 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5715 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 1.10.5 >Reporter: Tao Feng >Assignee: Tao Feng >Priority: Minor > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5715) Make email, owner context available
[ https://issues.apache.org/jira/browse/AIRFLOW-5715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng updated AIRFLOW-5715: -- Priority: Minor (was: Major) > Make email, owner context available > --- > > Key: AIRFLOW-5715 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5715 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 1.10.5 >Reporter: Tao Feng >Assignee: Tao Feng >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5715) Make email, owner context available
Tao Feng created AIRFLOW-5715: - Summary: Make email, owner context available Key: AIRFLOW-5715 URL: https://issues.apache.org/jira/browse/AIRFLOW-5715 Project: Apache Airflow Issue Type: Improvement Components: core Affects Versions: 1.10.5 Reporter: Tao Feng Assignee: Tao Feng -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5147) Annotations for k8s executors should support extended alphabet (like '/'))
[
https://issues.apache.org/jira/browse/AIRFLOW-5147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Feng resolved AIRFLOW-5147.
---
Resolution: Fixed
> Annotations for k8s executors should support extended alphabet (like '/'))
> ---
>
> Key: AIRFLOW-5147
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5147
> Project: Apache Airflow
> Issue Type: Bug
> Components: executor-kubernetes, executors
>Affects Versions: 1.10.3, 1.10.4
>Reporter: Andrei Loginov
>Assignee: Daniel Imberman
>Priority: Major
>
> The fix to introduce k8s annotations for executors
> ([https://github.com/apache/airflow/pull/4589] for
> https://issues.apache.org/jira/browse/AIRFLOW-3766) limited the character set
> allowed for the annotation key to [-._a-zA-Z0-9] set. However many
> annotations contain `/` in it, for example:
> {code:java}
> injector.tumblr.com/request{code}
> or
> {code:java}
> iam.amazonaws.com/role{code}
> Which would not be allowed in the current solution.
>
> I believe original solution should be completely revisited. And instead of
> using a separate *kubernetes_annotations* section there should be a key which
> will contain a set of key:value annotations in some format. E.g. json:
> {code:java}
> [kubernetes]
> annotations = { "iam.amazonaws.com/role":
> "arn:aws:iam:::role/some-role-CKU5HL9BIPXG", "some-other-anno-key":
> "some/value" }
> {code}
>
> Supported character set for annotations:
> https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/#syntax-and-character-set
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (AIRFLOW-5274) dag.loading-duration metric name too long
[
https://issues.apache.org/jira/browse/AIRFLOW-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Feng closed AIRFLOW-5274.
-
Resolution: Fixed
> dag.loading-duration metric name too long
> -
>
> Key: AIRFLOW-5274
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5274
> Project: Apache Airflow
> Issue Type: Improvement
> Components: logging
>Affects Versions: 1.10.4
>Reporter: Ben Marengo
>Assignee: Tao Feng
>Priority: Major
> Fix For: 1.10.5
>
>
> see conversation here -
> https://github.com/apache/airflow/pull/5350/files/5375a9af21db970651c17f947558b6fc180f0dd3#r315774306
> problem:
> when a single file generates a lot of dags, the metric name is huge
> possible solution:
> - is it not possible to append the name of the file where the dags are
> generated instead of appending {{'_'.join(dag_ids)}} ?
> - update the description in `docs/metrics.rst` to match the implementation
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Assigned] (AIRFLOW-5274) dag.loading-duration metric name too long
[
https://issues.apache.org/jira/browse/AIRFLOW-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Feng reassigned AIRFLOW-5274:
-
Assignee: Tao Feng
> dag.loading-duration metric name too long
> -
>
> Key: AIRFLOW-5274
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5274
> Project: Apache Airflow
> Issue Type: Improvement
> Components: logging
>Affects Versions: 1.10.4
>Reporter: Ben Marengo
>Assignee: Tao Feng
>Priority: Major
>
> see conversation here -
> https://github.com/apache/airflow/pull/5350/files/5375a9af21db970651c17f947558b6fc180f0dd3#r315774306
> problem:
> when a single file generates a lot of dags, the metric name is huge
> possible solution:
> - is it not possible to append the name of the file where the dags are
> generated instead of appending {{'_'.join(dag_ids)}} ?
> - update the description in `docs/metrics.rst` to match the implementation
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (AIRFLOW-4888) Add migration system for adding RBAC permissions to existing roles
[
https://issues.apache.org/jira/browse/AIRFLOW-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877984#comment-16877984
]
Tao Feng commented on AIRFLOW-4888:
---
[~ash] I vaguely remembered the issue was due to the fact that I changed the
permission for the default roles?I assume if we are going to make a migration
script, it is mostly for migration from old 1.10 setup to 1.10.2/1.10.3?
The only concern is that RBAC model has many different tables. I am not sure
how easily to cover it in a migration script. And should we move the model
files inside airflow as well? If you feel it is not complicated, I am +1 for
the migration script.
> Add migration system for adding RBAC permissions to existing roles
> --
>
> Key: AIRFLOW-4888
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4888
> Project: Apache Airflow
> Issue Type: Bug
> Components: core
>Affects Versions: 2.0.0
>Reporter: Ash Berlin-Taylor
>Priority: Major
> Labels: permissions
>
> In our clusters we don't allow any users to be Admin, so we use the Op, User
> and Viewer roles. It turns out that these roles are missing the
> {{can_dagrun_success}} and {{can_dagrun_failure}} permissions.
> Fixing this for new installs is easy, but due to AIRFLOW-3271
> (https://github.com/apache/airflow/pull/4118) we won't alter the roles if
> they already exist, so having some mechanism for adding permissions to roles
> via migrations might be useful.
> As a palyground I started working on
> https://gist.github.com/ashb/f43741740fb0eae59948d52634cda575 - I'm not sure
> if this is too complex or not. (It's also not a complete solution yet)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4591) Tag tasks with default pool
[ https://issues.apache.org/jira/browse/AIRFLOW-4591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-4591. --- Resolution: Fixed Fix Version/s: 2.0.0 > Tag tasks with default pool > --- > > Key: AIRFLOW-4591 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4591 > Project: Apache Airflow > Issue Type: New Feature > Components: core >Affects Versions: 2.0.0 >Reporter: Chao-Han Tsai >Assignee: Chao-Han Tsai >Priority: Major > Fix For: 2.0.0 > > > Currently the number of running tasks without a pool specified will be > limited by `non_pooled_task_slot_count`. It limits the number of tasks > launched per scheduler loop but does not limit the number of tasks running in > parallel. > This ticket proposes that we assign tasks without a pool specified to default > pool which limits the number of running tasks in parallel. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (AIRFLOW-4591) Tag tasks with default pool
[ https://issues.apache.org/jira/browse/AIRFLOW-4591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng reopened AIRFLOW-4591: --- > Tag tasks with default pool > --- > > Key: AIRFLOW-4591 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4591 > Project: Apache Airflow > Issue Type: New Feature > Components: core >Affects Versions: 2.0.0 >Reporter: Chao-Han Tsai >Assignee: Chao-Han Tsai >Priority: Major > > Currently the number of running tasks without a pool specified will be > limited by `non_pooled_task_slot_count`. It limits the number of tasks > launched per scheduler loop but does not limit the number of tasks running in > parallel. > This ticket proposes that we assign tasks without a pool specified to default > pool which limits the number of running tasks in parallel. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4591) Tag tasks with default pool
[ https://issues.apache.org/jira/browse/AIRFLOW-4591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-4591. --- Resolution: Fixed > Tag tasks with default pool > --- > > Key: AIRFLOW-4591 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4591 > Project: Apache Airflow > Issue Type: New Feature > Components: core >Affects Versions: 2.0.0 >Reporter: Chao-Han Tsai >Assignee: Chao-Han Tsai >Priority: Major > > Currently the number of running tasks without a pool specified will be > limited by `non_pooled_task_slot_count`. It limits the number of tasks > launched per scheduler loop but does not limit the number of tasks running in > parallel. > This ticket proposes that we assign tasks without a pool specified to default > pool which limits the number of running tasks in parallel. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4716) Instrument dag loading time duration
Tao Feng created AIRFLOW-4716: - Summary: Instrument dag loading time duration Key: AIRFLOW-4716 URL: https://issues.apache.org/jira/browse/AIRFLOW-4716 Project: Apache Airflow Issue Type: Improvement Components: DAG Affects Versions: 1.10.3 Reporter: Tao Feng Assignee: Tao Feng Fix For: 2.0.0 Sometimes it is useful to instrument the dag loading time and see which dag takes the longest the load. This is helpful for debugging once the UI becomes a bit unstable(e.g gunicorn timeout etc) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4521) Pause dag also pause subdags
[ https://issues.apache.org/jira/browse/AIRFLOW-4521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-4521. --- Resolution: Fixed Fix Version/s: 2.0.0 > Pause dag also pause subdags > > > Key: AIRFLOW-4521 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4521 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Chao-Han Tsai >Assignee: Chao-Han Tsai >Priority: Major > Fix For: 2.0.0 > > > Currently pause dag only pause the parent DAG but not pause the subdags. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4520) RBAC user cannot clear from tree view
[
https://issues.apache.org/jira/browse/AIRFLOW-4520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840095#comment-16840095
]
Tao Feng commented on AIRFLOW-4520:
---
[https://github.com/apache/airflow/pull/5220] should fix this issue.
> RBAC user cannot clear from tree view
> -
>
> Key: AIRFLOW-4520
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4520
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.10.3
>Reporter: Kevin Campbell
>Priority: Minor
>
> An RBAC user who can only access certain dags will receive an "Access is
> denied" in the tree view. The issue is that the clear operation performs a
> POST request, but the has_dag_access expects to handle GET requests when it
> calls
> {code:java}
> dag_id = request.args.get('dag_id'){code}
> This is fixed by AIRFLOW-4240 in commit 22c559dce
> Raising a JIRA ticket for completeness as there doesn't seem to be a record
> of the UI bug affecting users on 1.10.3
> As a workaround you can clear tasks via Browse > TaskInstances
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4420) Backfill respects task_concurrency
[ https://issues.apache.org/jira/browse/AIRFLOW-4420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-4420. --- Resolution: Fixed Fix Version/s: 1.10.4 > Backfill respects task_concurrency > -- > > Key: AIRFLOW-4420 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4420 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Chao-Han Tsai >Assignee: Chao-Han Tsai >Priority: Major > Fix For: 1.10.4 > > > Airflow backfill should respect > [task_concurrency|https://github.com/apache/airflow/blob/af3090786b170baf32c75fbd03c5f277c3ffaef8/airflow/models/baseoperator.py#L195-L197]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4459) To fix the issue that the DAG count in /home page may be wrong when DAG count is 0
[ https://issues.apache.org/jira/browse/AIRFLOW-4459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-4459. --- Resolution: Fixed > To fix the issue that the DAG count in /home page may be wrong when DAG count > is 0 > -- > > Key: AIRFLOW-4459 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4459 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Affects Versions: 1.10.3 >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > Fix For: 1.10.4 > > > When there is no DAG (maybe no DAG file deployed yet, or no matching result > when we search), the the DAG count at the right bottom corner may be wrong. > It will be "Showing 1 to 0 of 0 entries", while it should be "Showing 0 to 0 > of 0 entries". > On the other hand, if we provide URL argument "page" a too big value, let's > say we only have 10 DAGs, and we visit > "[http://localhost:8080/home?page=|http://localhost:8080/home?page=10]3;, we > may see results like "Showing 21 to 10 of 10 entries", which is wrong for > sure as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4146) CgroupTaskRunner is not functioning
[ https://issues.apache.org/jira/browse/AIRFLOW-4146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-4146. --- Resolution: Fixed Fix Version/s: 1.10.4 > CgroupTaskRunner is not functioning > --- > > Key: AIRFLOW-4146 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4146 > Project: Apache Airflow > Issue Type: Bug >Reporter: Junda Yang >Assignee: Junda Yang >Priority: Major > Fix For: 1.10.4 > > > Tried to switch from StandardTaskRunner to CgroupTaskRunner but tasks were > stuck and unable to finish. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4452) Webserver and Scheduler keep crashing because of slackclient update
[
https://issues.apache.org/jira/browse/AIRFLOW-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Feng resolved AIRFLOW-4452.
---
Resolution: Fixed
Fix Version/s: 1.10.4
> Webserver and Scheduler keep crashing because of slackclient update
> ---
>
> Key: AIRFLOW-4452
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4452
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler, webserver
>Affects Versions: 1.10.1
>Reporter: Abhishek Ray
>Assignee: Tao Feng
>Priority: Blocker
> Fix For: 1.10.4
>
>
> Webserver and Scheduler get into a crash loop if Airflow is installed with
> slack dependencies.
> Airflow relies on slackclient which released a new major version (2.0.0)
> today ([https://pypi.org/project/slackclient/#history]). This new version
> seems to be incompatible with Airflow causing the webserver to get into a
> crash loop.
> The root cause of the issue is that Airflow doesn't pin requirements for
> slackclient:
> [https://github.com/apache/airflow/blob/v1-10-stable/setup.py#L229]
> {code:java}
> slack = ['slackclient>=1.0.0']{code}
>
> This is the exception in the logs due to this error:
>
> {code:java}
> File "/Users/abhishek.ray/airflow/dags/test_dag.py", line 3, in
> from airflow.operators import SlackAPIPostOperator
> File
> "/Users/abhishek.ray/.virtualenvs/airflow-test/lib/python3.6/site-packages/airflow/utils/helpers.py",
> line 372, in __getattr__
> loaded_attribute = self._load_attribute(attribute)
> File
> "/Users/abhishek.ray/.virtualenvs/airflow-test/lib/python3.6/site-packages/airflow/utils/helpers.py",
> line 336, in _load_attribute
> self._loaded_modules[module] = imp.load_module(module, f, filename,
> description)
> File "/Users/abhishek.ray/.virtualenvs/airflow-test/lib/python3.6/imp.py",
> line 235, in load_module
> return load_source(name, filename, file)
> File "/Users/abhishek.ray/.virtualenvs/airflow-test/lib/python3.6/imp.py",
> line 172, in load_source
> module = _load(spec)
> File
> "/Users/abhishek.ray/.virtualenvs/airflow-test/lib/python3.6/site-packages/airflow/operators/slack_operator.py",
> line 24, in
> from airflow.hooks.slack_hook import SlackHook
> File
> "/Users/abhishek.ray/.virtualenvs/airflow-test/lib/python3.6/site-packages/airflow/hooks/slack_hook.py",
> line 20, in
> from slackclient import SlackClient
> File
> "/Users/abhishek.ray/.virtualenvs/airflow-test/lib/python3.6/site-packages/slackclient/__init__.py",
> line 1, in
> from .client import SlackClient # noqa
> File
> "/Users/abhishek.ray/.virtualenvs/airflow-test/lib/python3.6/site-packages/slackclient/client.py",
> line 8, in
> from .server import Server
> File
> "/Users/abhishek.ray/.virtualenvs/airflow-test/lib/python3.6/site-packages/slackclient/server.py",
> line 14, in
> from websocket import create_connection
> ModuleNotFoundError: No module named 'websocket'
> {code}
>
>
> This is how to reproduce this issue:
> Install apache airflow with slack:
> {code:java}
> pip install apache-airflow[slack]==1.10.1{code}
>
> Create a DAG which uses *SlackAPIPostOperator*
> {code:java}
> from airflow import DAG
> from airflow.operators.bash_operator import BashOperator
> from airflow.operators import SlackAPIPostOperator
> dag_default_args = {
> "owner": "airflow",
> "depends_on_past": False,
> "start_date": datetime(2019, 4, 22),
> "email": ["airf...@airflow.com"],
> "email_on_failure": False,
> "email_on_retry": False,
> "retries": 1,
> "catchup": True,
> }
> dag = DAG("test_dag", default_args=dag_default_args,
> schedule_interval="@daily")
> BashOperator(task_id="print_date", bash_command="date", dag=dag){code}
>
> I think the fix should be pretty straightforward to add a max version for
> slackclient.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-4452) Webserver and Scheduler keep crashing because of slackclient update
[
https://issues.apache.org/jira/browse/AIRFLOW-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Feng reassigned AIRFLOW-4452:
-
Assignee: Tao Feng
> Webserver and Scheduler keep crashing because of slackclient update
> ---
>
> Key: AIRFLOW-4452
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4452
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler, webserver
>Affects Versions: 1.10.1
>Reporter: Abhishek Ray
>Assignee: Tao Feng
>Priority: Blocker
>
> Webserver and Scheduler get into a crash loop if Airflow is installed with
> slack dependencies.
> Airflow relies on slackclient which released a new major version (2.0.0)
> today ([https://pypi.org/project/slackclient/#history]). This new version
> seems to be incompatible with Airflow causing the webserver to get into a
> crash loop.
> The root cause of the issue is that Airflow doesn't pin requirements for
> slackclient:
> [https://github.com/apache/airflow/blob/v1-10-stable/setup.py#L229]
> {code:java}
> slack = ['slackclient>=1.0.0']{code}
>
> This is the exception in the logs due to this error:
>
> {code:java}
> File "/Users/abhishek.ray/airflow/dags/test_dag.py", line 3, in
> from airflow.operators import SlackAPIPostOperator
> File
> "/Users/abhishek.ray/.virtualenvs/airflow-test/lib/python3.6/site-packages/airflow/utils/helpers.py",
> line 372, in __getattr__
> loaded_attribute = self._load_attribute(attribute)
> File
> "/Users/abhishek.ray/.virtualenvs/airflow-test/lib/python3.6/site-packages/airflow/utils/helpers.py",
> line 336, in _load_attribute
> self._loaded_modules[module] = imp.load_module(module, f, filename,
> description)
> File "/Users/abhishek.ray/.virtualenvs/airflow-test/lib/python3.6/imp.py",
> line 235, in load_module
> return load_source(name, filename, file)
> File "/Users/abhishek.ray/.virtualenvs/airflow-test/lib/python3.6/imp.py",
> line 172, in load_source
> module = _load(spec)
> File
> "/Users/abhishek.ray/.virtualenvs/airflow-test/lib/python3.6/site-packages/airflow/operators/slack_operator.py",
> line 24, in
> from airflow.hooks.slack_hook import SlackHook
> File
> "/Users/abhishek.ray/.virtualenvs/airflow-test/lib/python3.6/site-packages/airflow/hooks/slack_hook.py",
> line 20, in
> from slackclient import SlackClient
> File
> "/Users/abhishek.ray/.virtualenvs/airflow-test/lib/python3.6/site-packages/slackclient/__init__.py",
> line 1, in
> from .client import SlackClient # noqa
> File
> "/Users/abhishek.ray/.virtualenvs/airflow-test/lib/python3.6/site-packages/slackclient/client.py",
> line 8, in
> from .server import Server
> File
> "/Users/abhishek.ray/.virtualenvs/airflow-test/lib/python3.6/site-packages/slackclient/server.py",
> line 14, in
> from websocket import create_connection
> ModuleNotFoundError: No module named 'websocket'
> {code}
>
>
> This is how to reproduce this issue:
> Install apache airflow with slack:
> {code:java}
> pip install apache-airflow[slack]==1.10.1{code}
>
> Create a DAG which uses *SlackAPIPostOperator*
> {code:java}
> from airflow import DAG
> from airflow.operators.bash_operator import BashOperator
> from airflow.operators import SlackAPIPostOperator
> dag_default_args = {
> "owner": "airflow",
> "depends_on_past": False,
> "start_date": datetime(2019, 4, 22),
> "email": ["airf...@airflow.com"],
> "email_on_failure": False,
> "email_on_retry": False,
> "retries": 1,
> "catchup": True,
> }
> dag = DAG("test_dag", default_args=dag_default_args,
> schedule_interval="@daily")
> BashOperator(task_id="print_date", bash_command="date", dag=dag){code}
>
> I think the fix should be pretty straightforward to add a max version for
> slackclient.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4447) Display task duration as human friendly format in Tree View
[ https://issues.apache.org/jira/browse/AIRFLOW-4447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-4447. --- Resolution: Fixed Fix Version/s: 1.10.4 > Display task duration as human friendly format in Tree View > --- > > Key: AIRFLOW-4447 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4447 > Project: Apache Airflow > Issue Type: Improvement > Components: webserver >Reporter: Ping Zhang >Assignee: Ping Zhang >Priority: Minor > Fix For: 1.10.4 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4419) Concurrency calculation in scheduler is slow and has a bug
[ https://issues.apache.org/jira/browse/AIRFLOW-4419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-4419. --- Resolution: Fixed Fix Version/s: 2.0.0 > Concurrency calculation in scheduler is slow and has a bug > -- > > Key: AIRFLOW-4419 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4419 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Kevin Yang >Assignee: Kevin Yang >Priority: Minor > Fix For: 2.0.0 > > > The pool, dag and task concurrency calculation logic is slow and can be > optimized. Also current task concurrency should not increase if the executor > already has the task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4228) DatabricksRunNowOperator does not show up under airflow docs
[ https://issues.apache.org/jira/browse/AIRFLOW-4228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-4228. --- Resolution: Fixed Fix Version/s: 2.0.0 > DatabricksRunNowOperator does not show up under airflow docs > > > Key: AIRFLOW-4228 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4228 > Project: Apache Airflow > Issue Type: Bug >Reporter: Thomas Dziedzic >Assignee: Thomas Elvey >Priority: Trivial > Fix For: 2.0.0 > > > [https://airflow.apache.org/_modules/airflow/contrib/operators/databricks_operator.html] > contains DatabricksRunNowOperator but when you visit > [https://airflow.apache.org/integration.html?highlight=databricks#databricks] > there is no mention of the DatabricksRunNowOperator even though it is > available and working. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4409) UI duration view can be broken by task faile null duration column
[ https://issues.apache.org/jira/browse/AIRFLOW-4409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-4409. --- Resolution: Fixed Fix Version/s: 2.0.0 > UI duration view can be broken by task faile null duration column > - > > Key: AIRFLOW-4409 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4409 > Project: Apache Airflow > Issue Type: Improvement > Components: ui >Affects Versions: 1.10.0 >Reporter: Yingbo Wang >Assignee: Yingbo Wang >Priority: Minor > Fix For: 2.0.0 > > Attachments: Screen Shot 2019-04-24 at 1.54.53 PM.png > > Original Estimate: 1h > Remaining Estimate: 1h > > Airflow DAG UI has a view "task duration". Due to our recent migration. There > are some records in the task_fail table which has null duration (caused by > missing start date). This is affecting the UI view of task duration !Screen > Shot 2019-04-24 at 1.54.53 PM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4306) Global operator extra links
[ https://issues.apache.org/jira/browse/AIRFLOW-4306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-4306. --- Resolution: Fixed Fix Version/s: 2.0.0 > Global operator extra links > --- > > Key: AIRFLOW-4306 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4306 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Chao-Han Tsai >Assignee: Chao-Han Tsai >Priority: Major > Fix For: 2.0.0 > > > A way to register global operator extra links that are shared by all the > operators. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4402) Update super() calls for nvd3
[
https://issues.apache.org/jira/browse/AIRFLOW-4402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Feng resolved AIRFLOW-4402.
---
Resolution: Fixed
> Update super() calls for nvd3
> -
>
> Key: AIRFLOW-4402
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4402
> Project: Apache Airflow
> Issue Type: Sub-task
>Reporter: Chao-Han Tsai
>Assignee: Chao-Han Tsai
>Priority: Major
> Fix For: 2.0.0
>
>
> In all classes under nvd3, replace {{super(__class__, self).__init__(...)}}
> by {{super().__init__(...)}}
> Similarly for any other {{super}} calls for other methods.
> (In Python 3 {{super(__class__, self) == super()}})
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4402) Update super() calls for nvd3
[
https://issues.apache.org/jira/browse/AIRFLOW-4402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16824830#comment-16824830
]
Tao Feng commented on AIRFLOW-4402:
---
[~milton0825] didn't quite follow all the prs recently. But I found what you
are talking about. Just raise the questions in that pr as I am not 100% sure.
> Update super() calls for nvd3
> -
>
> Key: AIRFLOW-4402
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4402
> Project: Apache Airflow
> Issue Type: Sub-task
>Reporter: Chao-Han Tsai
>Assignee: Chao-Han Tsai
>Priority: Major
> Fix For: 2.0.0
>
>
> In all classes under nvd3, replace {{super(__class__, self).__init__(...)}}
> by {{super().__init__(...)}}
> Similarly for any other {{super}} calls for other methods.
> (In Python 3 {{super(__class__, self) == super()}})
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4402) Update super() calls for nvd3
[
https://issues.apache.org/jira/browse/AIRFLOW-4402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16824820#comment-16824820
]
Tao Feng commented on AIRFLOW-4402:
---
hold on, I don't think we should change any of the files under vendor folder as
all the file under that folder meant to be copy paste from upstream dependency.
> Update super() calls for nvd3
> -
>
> Key: AIRFLOW-4402
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4402
> Project: Apache Airflow
> Issue Type: Sub-task
>Reporter: Chao-Han Tsai
>Assignee: Chao-Han Tsai
>Priority: Major
> Fix For: 2.0.0
>
>
> In all classes under nvd3, replace {{super(__class__, self).__init__(...)}}
> by {{super().__init__(...)}}
> Similarly for any other {{super}} calls for other methods.
> (In Python 3 {{super(__class__, self) == super()}})
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4339) Cleared backfill tasks do not get picked by the scheduler
[ https://issues.apache.org/jira/browse/AIRFLOW-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820258#comment-16820258 ] Tao Feng commented on AIRFLOW-4339: --- [~Ignas] I am in a conference, could take a look in the night, but all the clear tasks should be picked by scheduler. Which executor do you use for backfill? The patch has been running in our prod without any issues so far. > Cleared backfill tasks do not get picked by the scheduler > - > > Key: AIRFLOW-4339 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4339 > Project: Apache Airflow > Issue Type: Bug > Components: backfill, scheduler >Affects Versions: 1.10.1 >Reporter: Ignas >Priority: Blocker > > When a task in a successful backfill run is cleared DagRun with running state > is created but it is not picked by the scheduler. > If there are many of such DagRuns and the backfill command is executed, all > the DagRuns will be ran at the same time, not following the max_active_runs > parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-161) Redirection to external url
[ https://issues.apache.org/jira/browse/AIRFLOW-161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-161. -- Resolution: Fixed Fix Version/s: 2.0.0 > Redirection to external url > --- > > Key: AIRFLOW-161 > URL: https://issues.apache.org/jira/browse/AIRFLOW-161 > Project: Apache Airflow > Issue Type: Improvement > Components: webserver >Reporter: Sumit Maheshwari >Assignee: Max Payton >Priority: Major > Fix For: 2.0.0 > > > Hi, > I am not able to find a good way (apart from loading everything upfront), > where I can redirect someone to a external service url, using the information > stored in airflow. There could be many use cases like downloading a signed > file from s3, redirecting to hadoop job tracker, or a direct case on which I > am working which is linking airflow tasks to qubole commands. > I already have a working model and will open a PR soon. Please let me know if > there existing ways already. > Thanks, > Sumit -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4251) Instrument DagRun schedule delay
[ https://issues.apache.org/jira/browse/AIRFLOW-4251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-4251. --- Resolution: Fixed Fix Version/s: 2.0.0 > Instrument DagRun schedule delay > > > Key: AIRFLOW-4251 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4251 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Chao-Han Tsai >Assignee: Chao-Han Tsai >Priority: Major > Fix For: 2.0.0 > > > Instrument DagRun schedule delay - time between expected DagRun start date > and the actual DagRun start date. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4267) Fix Task Instance duration in Graph View in Fab-Based UI
[ https://issues.apache.org/jira/browse/AIRFLOW-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-4267. --- Resolution: Fixed Fix Version/s: 1.10.3 > Fix Task Instance duration in Graph View in Fab-Based UI > > > Key: AIRFLOW-4267 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4267 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Affects Versions: 1.10.2, 1.10.3 >Reporter: Kaxil Naik >Assignee: Kaxil Naik >Priority: Critical > Fix For: 1.10.3 > > Attachments: Screenshot 2019-04-09 at 21.58.54.png, Screenshot > 2019-04-09 at 22.01.42.png > > > The duration field is missing in the Graph View for the Fab-based RBAC UI. > This is because of the bug in the code. Instead of *task.duration* it should > be *ti.duration*. > The duration is associated with a *Task Instance* and no a *Task*. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4265) Lineage backend did not work normally
[
https://issues.apache.org/jira/browse/AIRFLOW-4265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Feng resolved AIRFLOW-4265.
---
Resolution: Fixed
Fix Version/s: 2.0.0
> Lineage backend did not work normally
> -
>
> Key: AIRFLOW-4265
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4265
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Bryan Yang
>Assignee: Bryan Yang
>Priority: Major
> Fix For: 2.0.0
>
>
> 1. Lineage backend comes from {{import_string}} function, and
> {{import_string}} will try to seperate module_path and class_name by dot.
> {quote}def import_string(dotted_path):
> """
> Import a dotted module path and return the attribute/class designated by the
> last name in the path. Raise ImportError if the import failed.
> """
> try:
> module_path, class_name = dotted_path.rsplit('.', 1)
> except ValueError:
> raise ImportError("{} doesn't look like a module path".format(dotted_path))
> module = import_module(module_path)
> {quote}
> But in the current document: {{backend = airflow.lineage.backend.atlas}}
> which is just a module path without class name \{{AtlasBackend}}.
> 2. Continuously, this function will return a module or class instead of an
> instance. So that when {{apply_lineage}} want to call
> {{backend.send_lineage}}, it will fail.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4261) To fix some minor issues in airflow/jobs.py
[ https://issues.apache.org/jira/browse/AIRFLOW-4261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-4261. --- Resolution: Fixed Fix Version/s: 2.0.0 > To fix some minor issues in airflow/jobs.py > --- > > Key: AIRFLOW-4261 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4261 > Project: Apache Airflow > Issue Type: Improvement > Components: scheduler >Affects Versions: 1.10.2 >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > Fix For: 2.0.0 > > > There are some minor issues in airflow/jobs.py. Clearing them can help 1. > make code cleaner; 2. introduce minor performance improvement. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4262) SlackWebhookOperator is not working properly
[ https://issues.apache.org/jira/browse/AIRFLOW-4262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-4262. --- Resolution: Fixed Fix Version/s: 2.0.0 > SlackWebhookOperator is not working properly > > > Key: AIRFLOW-4262 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4262 > Project: Apache Airflow > Issue Type: Bug >Reporter: Junda Yang >Assignee: Junda Yang >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4240) State changing actions shouldn't be GET requests
[ https://issues.apache.org/jira/browse/AIRFLOW-4240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-4240. --- Resolution: Fixed Assignee: Ash Berlin-Taylor Fix Version/s: 1.10.3 > State changing actions shouldn't be GET requests > > > Key: AIRFLOW-4240 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4240 > Project: Apache Airflow > Issue Type: Bug >Reporter: Ash Berlin-Taylor >Assignee: Ash Berlin-Taylor >Priority: Major > Fix For: 1.10.3 > > > We have a number of actions which perform actions (trigger, clear, etc) that > are performed over GET requests. > That should be avoided as browsers/corporate proxies might prefetch the URLs > causing things to behave oddly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4163) IntervalCheckOperator support relative difference ratio and can ignore zero
[ https://issues.apache.org/jira/browse/AIRFLOW-4163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-4163. --- Resolution: Fixed Fix Version/s: 2.0.0 > IntervalCheckOperator support relative difference ratio and can ignore zero > > > Key: AIRFLOW-4163 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4163 > Project: Apache Airflow > Issue Type: Bug >Reporter: Chao-Han Tsai >Assignee: Chao-Han Tsai >Priority: Major > Fix For: 2.0.0 > > > - IntervalCheckOperator takes max/min ratio of two values for a metric and > returns true if it is less than threshold. Currently if one of the values is > 0, it assigns the ratio as None. In python comparison None < Number is always > true. We should add an option to fail the task if one of the value is 0. > - Currently it only supports Max/Min. It would be useful to support > calculating ratio with relative difference. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4194) set dag_run state to failed when user terminate backfill
[ https://issues.apache.org/jira/browse/AIRFLOW-4194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-4194. --- Resolution: Fixed Fix Version/s: 2.0.0 > set dag_run state to failed when user terminate backfill > > > Key: AIRFLOW-4194 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4194 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Chao-Han Tsai >Assignee: Chao-Han Tsai >Priority: Major > Fix For: 2.0.0 > > > Reset dag_run state to failed if user terminate backfill. Otherwise the > dag_run state will stay in running state which consumes max_active_dagruns. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4194) set dag_run state to failed when user terminate backfill
[ https://issues.apache.org/jira/browse/AIRFLOW-4194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806300#comment-16806300 ] Tao Feng commented on AIRFLOW-4194: --- [~milton0825], I never finish the pr([https://github.com/apache/airflow/pull/3562]) with all the tests. > set dag_run state to failed when user terminate backfill > > > Key: AIRFLOW-4194 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4194 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Chao-Han Tsai >Assignee: Chao-Han Tsai >Priority: Major > > Currently when user terminate the backfill, we set the task_instance state to > failed. We should also set the dag_run state to failed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4190) Add a schedule delay monitoring DAG
[ https://issues.apache.org/jira/browse/AIRFLOW-4190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805679#comment-16805679 ] Tao Feng commented on AIRFLOW-4190: --- [~milton0825], not sure if it is a good idea to upstream this dag. Normally Airflow only include the example dags. > Add a schedule delay monitoring DAG > --- > > Key: AIRFLOW-4190 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4190 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Chao-Han Tsai >Assignee: Chao-Han Tsai >Priority: Major > > The DAG measures the delay between the scheduled DAG start time (e.g. > next_execution_date) and the wall clock time when first task executes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4160) Redirecting of "Trigger Dag" Button in DAG Page is not working
[ https://issues.apache.org/jira/browse/AIRFLOW-4160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-4160. --- Resolution: Fixed Fix Version/s: 1.10.3 > Redirecting of "Trigger Dag" Button in DAG Page is not working > -- > > Key: AIRFLOW-4160 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4160 > Project: Apache Airflow > Issue Type: Bug > Components: webapp >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > Fix For: 1.10.3 > > > "Trigger Dag" Button in DAG Page was added in > [https://github.com/apache/airflow/pull/4373] > but it was broken by [https://github.com/apache/airflow/pull/4643] > > This issue is found in the testing process of 1.10.3b1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-1557) backfill ignores configured number of slots in a pool
[
https://issues.apache.org/jira/browse/AIRFLOW-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Feng resolved AIRFLOW-1557.
---
Resolution: Fixed
Fix Version/s: 1.10.3
Thanks [~milton0825]. I think this pr should be included in 1.10.3.
> backfill ignores configured number of slots in a pool
> -
>
> Key: AIRFLOW-1557
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1557
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.8.1
>Reporter: Ash Berlin-Taylor
>Assignee: Chao-Han Tsai
>Priority: Major
> Fix For: 1.10.3
>
> Attachments: Backfill_bug_1.8.2.jpg, Screen Shot 2017-09-01 at
> 11.39.32.png
>
>
> I ran a backfill command with this tool:
> The backfill process appears to run as many tasks as possible, even when the
> pool it is running in should limit the numbers:
> {noformat}
> airflow backfill \
> -t fetch_dk_unfiltered \
> --pool brand_index_api \
> -s 2017-07-31 -e 2017-08-31 \
> -x \
> brand_index_fetcher
> {noformat}
> (Nothing else than the backfill is currently using this pool. I wasn't able
> to capture a screen shot of the task instances before the jobs completed.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2652) implement / enhance pythonOperator deepcopy
[ https://issues.apache.org/jira/browse/AIRFLOW-2652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-2652. --- Resolution: Fixed Fix Version/s: 1.10.3 > implement / enhance pythonOperator deepcopy > --- > > Key: AIRFLOW-2652 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2652 > Project: Apache Airflow > Issue Type: Bug >Reporter: Tao Feng >Assignee: Tao Feng >Priority: Major > Fix For: 1.10.3 > > > Currently if we backfill an airflow task which is pythonOperator, when > airflow does a deepcopy of callable from the operator we may encounter > unexpected issue. We should skip the callable deepcopy as it will not get > mutated. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2558) Clear TASK/DAG is clearing all executions
[
https://issues.apache.org/jira/browse/AIRFLOW-2558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Feng resolved AIRFLOW-2558.
---
Resolution: Fixed
Fix Version/s: (was: 1.10.0)
1.10.3
> Clear TASK/DAG is clearing all executions
> -
>
> Key: AIRFLOW-2558
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2558
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Marcos Bernardelli
>Assignee: Tao Feng
>Priority: Major
> Fix For: 1.10.3
>
>
> When I try to clear a DAG/TASK specific execution, the Airflow try to execute
> all the past executions:
> [Animeted
> GIF|https://gist.githubusercontent.com/bern4rdelli/34c1e57acd53c8c67417604202f3e0e6/raw/4bcb3d3c23f2a3bb7f7bfb3e977d935e5bb9f0ee/clear.gif]
> (I failed miserable trying to attache the animated GIF :()
>
> This behavior was changed here:
> [https://github.com/apache/incubator-airflow/pull/3444]
> The old version looks like this:
> {code:python}
> drs = session.query(DagModel).filter_by(dag_id=self.dag_id).all()
> {code}
> Then it's changed to:
> {code:python}
> drs = session.query(DagRun).filter_by(dag_id=self.dag_id).all()
> {code}
> This new query (using DagRun) get all the past executions, even when the
> "Past" button is not checked.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2537) clearing tasks shouldn't set backfill DAG runs to `running`
[ https://issues.apache.org/jira/browse/AIRFLOW-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-2537. --- Resolution: Fixed Fix Version/s: 1.10.3 > clearing tasks shouldn't set backfill DAG runs to `running` > --- > > Key: AIRFLOW-2537 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2537 > Project: Apache Airflow > Issue Type: Bug >Reporter: Maxime Beauchemin >Assignee: Tao Feng >Priority: Major > Fix For: 1.10.3 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (AIRFLOW-2558) Clear TASK/DAG is clearing all executions
[
https://issues.apache.org/jira/browse/AIRFLOW-2558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Feng reopened AIRFLOW-2558:
---
> Clear TASK/DAG is clearing all executions
> -
>
> Key: AIRFLOW-2558
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2558
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Marcos Bernardelli
>Assignee: Tao Feng
>Priority: Major
> Fix For: 1.10.0
>
>
> When I try to clear a DAG/TASK specific execution, the Airflow try to execute
> all the past executions:
> [Animeted
> GIF|https://gist.githubusercontent.com/bern4rdelli/34c1e57acd53c8c67417604202f3e0e6/raw/4bcb3d3c23f2a3bb7f7bfb3e977d935e5bb9f0ee/clear.gif]
> (I failed miserable trying to attache the animated GIF :()
>
> This behavior was changed here:
> [https://github.com/apache/incubator-airflow/pull/3444]
> The old version looks like this:
> {code:python}
> drs = session.query(DagModel).filter_by(dag_id=self.dag_id).all()
> {code}
> Then it's changed to:
> {code:python}
> drs = session.query(DagRun).filter_by(dag_id=self.dag_id).all()
> {code}
> This new query (using DagRun) get all the past executions, even when the
> "Past" button is not checked.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (AIRFLOW-2537) clearing tasks shouldn't set backfill DAG runs to `running`
[ https://issues.apache.org/jira/browse/AIRFLOW-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng reopened AIRFLOW-2537: --- > clearing tasks shouldn't set backfill DAG runs to `running` > --- > > Key: AIRFLOW-2537 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2537 > Project: Apache Airflow > Issue Type: Bug >Reporter: Maxime Beauchemin >Assignee: Tao Feng >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (AIRFLOW-2652) implement / enhance pythonOperator deepcopy
[ https://issues.apache.org/jira/browse/AIRFLOW-2652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng reopened AIRFLOW-2652: --- > implement / enhance pythonOperator deepcopy > --- > > Key: AIRFLOW-2652 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2652 > Project: Apache Airflow > Issue Type: Bug >Reporter: Tao Feng >Assignee: Tao Feng >Priority: Major > > Currently if we backfill an airflow task which is pythonOperator, when > airflow does a deepcopy of callable from the operator we may encounter > unexpected issue. We should skip the callable deepcopy as it will not get > mutated. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3982) DagRun state updater is incorrect when DagRun's tasks don't correspond to DAG's
[ https://issues.apache.org/jira/browse/AIRFLOW-3982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-3982. --- Resolution: Fixed Fix Version/s: 1.10.3 > DagRun state updater is incorrect when DagRun's tasks don't correspond to > DAG's > --- > > Key: AIRFLOW-3982 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3982 > Project: Apache Airflow > Issue Type: Bug >Reporter: Dima Kamalov >Assignee: Dima Kamalov >Priority: Minor > Fix For: 1.10.3 > > > To repro: > (1) create a Dag with a start date and a task, run it > (2) add a task in that DAG with a later start date > (3) rerun past run – tasks will get processed correctly but DagRun state > won't get updated -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-4139) DAG versioning
[ https://issues.apache.org/jira/browse/AIRFLOW-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799271#comment-16799271 ] Tao Feng edited comment on AIRFLOW-4139 at 3/22/19 7:07 PM: there is a long pr([https://github.com/apache/airflow/pull/4396)] from the user. I believe the graph_id is the version proposed by the AIP which the main goal to solve the dag shape change problem. I would suggest you follow up with author or mailing list to avoid any duplicate effort. The scope of his AIP and yours may not be totally aligned, but I feel there is certain overlap between these two which I suggest you to clarify with him or the mailing list first. was (Author: taofeng): there is a long pr([https://github.com/apache/airflow/pull/4396)] from the user. I believe the graph_id is the version proposed by the AIP which the main goal to solve the dag shape change problem. I would suggest you follow up with author or mailing list to avoid any duplicate effort. > DAG versioning > -- > > Key: AIRFLOW-4139 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4139 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Chao-Han Tsai >Assignee: Chao-Han Tsai >Priority: Major > > Currently, running DagRun will be impacted if we change the DAG file in the > middle of the run. After we have > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher > and https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-20+DAG+manifest > ready, we can start saving each version of the DAG file on the remote system > and the running tasks should refer to a specific version of DAG instead of > the latest DAG. > How is it different from > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-12+Persist+DAG+into+DB? > Please see > https://issues.apache.org/jira/browse/AIRFLOW-4139?focusedCommentId=16799264=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16799264 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4139) DAG versioning
[ https://issues.apache.org/jira/browse/AIRFLOW-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799271#comment-16799271 ] Tao Feng commented on AIRFLOW-4139: --- there is a long pr([https://github.com/apache/airflow/pull/4396)] from the user. I believe the graph_id is the version proposed by the AIP which the main goal to solve the dag shape change problem. I would suggest you follow up with author or mailing list to avoid any duplicate effort. > DAG versioning > -- > > Key: AIRFLOW-4139 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4139 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Chao-Han Tsai >Assignee: Chao-Han Tsai >Priority: Major > > Currently, running DagRun will be impacted if we change the DAG file in the > middle of the run. After we have > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher > and https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-20+DAG+manifest > ready, we can start saving each version of the DAG file on the remote system > and the running tasks should refer to a specific version of DAG instead of > the latest DAG. > How is it different from > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-12+Persist+DAG+into+DB? > Please see > https://issues.apache.org/jira/browse/AIRFLOW-4139?focusedCommentId=16799264=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16799264 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4145) Allow RBAC roles permissions , VM to be overridable
Tao Feng created AIRFLOW-4145: - Summary: Allow RBAC roles permissions , VM to be overridable Key: AIRFLOW-4145 URL: https://issues.apache.org/jira/browse/AIRFLOW-4145 Project: Apache Airflow Issue Type: Improvement Reporter: Tao Feng Assignee: Tao Feng -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4139) DAG versioning
[ https://issues.apache.org/jira/browse/AIRFLOW-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799193#comment-16799193 ] Tao Feng commented on AIRFLOW-4139: --- [~milton0825], I wonder what is the difference between this vs [https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-12+Persist+DAG+into+DB] as they all try to maintain the DAG version . > DAG versioning > -- > > Key: AIRFLOW-4139 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4139 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Chao-Han Tsai >Assignee: Chao-Han Tsai >Priority: Major > > Currently, running DagRun will be impacted if we change the DAG file. > Existing running DagRun should not be impacted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4139) DAG versioning
[ https://issues.apache.org/jira/browse/AIRFLOW-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799194#comment-16799194 ] Tao Feng commented on AIRFLOW-4139: --- For AIP12, there are couples of people that have been working on the feature with pr review already. > DAG versioning > -- > > Key: AIRFLOW-4139 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4139 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Chao-Han Tsai >Assignee: Chao-Han Tsai >Priority: Major > > Currently, running DagRun will be impacted if we change the DAG file. > Existing running DagRun should not be impacted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4057) airflow should handle invalid stats name
[ https://issues.apache.org/jira/browse/AIRFLOW-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-4057. --- Resolution: Fixed > airflow should handle invalid stats name > > > Key: AIRFLOW-4057 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4057 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Chao-Han Tsai >Assignee: Chao-Han Tsai >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4137) Create a doc how to configure FAB with OAUTH
[ https://issues.apache.org/jira/browse/AIRFLOW-4137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng updated AIRFLOW-4137: -- Summary: Create a doc how to configure FAB with OAUTH (was: Create a doc how to configure FAB with OATU) > Create a doc how to configure FAB with OAUTH > > > Key: AIRFLOW-4137 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4137 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Tao Feng >Assignee: Andrew Stahlman >Priority: Major > > Andrew, create this place holder ticket to keep track of the documentation > for the configuration. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4137) Create a doc how to configure FAB with OATU
Tao Feng created AIRFLOW-4137: - Summary: Create a doc how to configure FAB with OATU Key: AIRFLOW-4137 URL: https://issues.apache.org/jira/browse/AIRFLOW-4137 Project: Apache Airflow Issue Type: Improvement Reporter: Tao Feng Assignee: Andrew Stahlman Andrew, create this place holder ticket to keep track of the documentation for the configuration. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4106) Instrument starving tasks in pool
[ https://issues.apache.org/jira/browse/AIRFLOW-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-4106. --- Resolution: Fixed Fix Version/s: 1.10.3 > Instrument starving tasks in pool > - > > Key: AIRFLOW-4106 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4106 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Chao-Han Tsai >Assignee: Chao-Han Tsai >Priority: Major > Fix For: 1.10.3 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4122) Remove chain function in airflow.utils
[ https://issues.apache.org/jira/browse/AIRFLOW-4122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-4122. --- Resolution: Fixed > Remove chain function in airflow.utils > -- > > Key: AIRFLOW-4122 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4122 > Project: Apache Airflow > Issue Type: Improvement > Components: utils >Affects Versions: 1.10.2 >Reporter: zhongjiajie >Assignee: zhongjiajie >Priority: Major > Labels: patch > Fix For: 1.10.3 > > > I had submit [pr|[https://github.com/apache/airflow/pull/4779]] and it > related to Jira ticket . > In PR comment, I was suggest close this PR and submit new one to remove > function `chain` in `airflow.utils.heplers`, due to Airflower think use bit > operation like `>>` or `<<` visual and easier to explain, and have multiple > ways is confusion to users. BTW, this function was created 3 year ago and > never get updated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4107) Instrument executor
[ https://issues.apache.org/jira/browse/AIRFLOW-4107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-4107. --- Resolution: Fixed Fix Version/s: 1.10.3 > Instrument executor > --- > > Key: AIRFLOW-4107 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4107 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Chao-Han Tsai >Priority: Major > Fix For: 1.10.3 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4087) Calling SUDO to remove temp cfg file failed
[
https://issues.apache.org/jira/browse/AIRFLOW-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Feng resolved AIRFLOW-4087.
---
Resolution: Fixed
> Calling SUDO to remove temp cfg file failed
> ---
>
> Key: AIRFLOW-4087
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4087
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Junda Yang
>Priority: Major
>
> At Lyft we use a non-root user (named *airflow* in our case) to run airflow
> worker and we keep getting emails saying that the user is not in sudoers so
> the command to remove temp airflow.cfg failed.
> {code:java}
> Mar 13 21:22:33 : airflow : user NOT in sudoers ; TTY=unknown ;
> PWD=/srv/airflowinfra ; USER=root ; COMMAND=/bin/rm /tmp/tmpgzw8eaww
> {code}
> I found that the error was generated due to
> [https://github.com/apache/airflow/blob/5a303ebbc572cee7c9c30be84ebf625357360d4b/airflow/task_runner/base_task_runner.py#L155-#L160]
>
> Not having enough context, is there any reason airflow needs to run sudo to
> remove the file? If not, I think it should be safe to remove the sudo,
> because the temp file is owned by the same user who runs worker command.
> {code:java}
> jundayang@airflowinfraworker-production-iad-0184b3e28b386f0cf:/tmp$ ls -la
> tmpzyfuiw2v
> -rw--- 1 airflow airflow 7139 Mar 13 21:25 tmpzyfuiw2v
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4086) Docker build fails
[ https://issues.apache.org/jira/browse/AIRFLOW-4086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-4086. --- Resolution: Fixed > Docker build fails > --- > > Key: AIRFLOW-4086 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4086 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Jarek Potiuk >Priority: Major > Fix For: 1.10.3 > > > The current Dockerfile has an error that makes it fail (-o pipefail). Docker > by default uses sh and not bash, therefore set -o pipefail causes the Docker > to fail -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4073) Add supporting of templated file of Athena Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-4073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-4073. --- Resolution: Fixed > Add supporting of templated file of Athena Operator > --- > > Key: AIRFLOW-4073 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4073 > Project: Apache Airflow > Issue Type: Improvement > Components: aws >Reporter: Bryan Yang >Assignee: Bryan Yang >Priority: Minor > Original Estimate: 24h > Remaining Estimate: 24h > > Don't like other SQL operators(eg. hive, bigquery), AWS Athena operator > didn't support template_ext now. I'll add template_ext to the operator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4033) Record stats of task duration
[ https://issues.apache.org/jira/browse/AIRFLOW-4033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-4033. --- Resolution: Fixed Fix Version/s: 1.10.3 > Record stats of task duration > - > > Key: AIRFLOW-4033 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4033 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Chao-Han Tsai >Assignee: Chao-Han Tsai >Priority: Major > Fix For: 1.10.3 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4044) The documentation of `query_params` in `BigQueryOperator` is wrong
[ https://issues.apache.org/jira/browse/AIRFLOW-4044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-4044. --- Resolution: Fixed Fix Version/s: 1.10.3 > The documentation of `query_params` in `BigQueryOperator` is wrong > -- > > Key: AIRFLOW-4044 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4044 > Project: Apache Airflow > Issue Type: Bug > Components: Documentation >Affects Versions: 1.10.2 >Reporter: Hengfeng Li >Priority: Minor > Fix For: 1.10.3 > > > Currently, the doc > ([https://airflow.apache.org/code.html?highlight=query_params]) says: > * *query_params* (_dict_) - a dictionary containing query parameter types > and values, passed to BigQuery. > However, in BigQueryBaseCursor > ([https://github.com/apache/airflow/blob/1.10.2/airflow/contrib/hooks/bigquery_hook.py#L575-L577] > > [),|https://github.com/apache/airflow/blob/0c797a830e3370bd6e39f5fcfc128a8fd776912e/airflow/contrib/hooks/bigquery_hook.py#L694-L696),] > the doc indicates that this parameter should be a list of `dict`. Also, it > is unclear how this `query_params` look like and no examples are available. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4053) Fix KubePodOperator Xcom on Kube 1.13.0
[ https://issues.apache.org/jira/browse/AIRFLOW-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-4053. --- Resolution: Fixed Fix Version/s: 1.10.3 > Fix KubePodOperator Xcom on Kube 1.13.0 > --- > > Key: AIRFLOW-4053 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4053 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Reporter: Ash Berlin-Taylor >Priority: Major > Labels: kubernetes > Fix For: 1.10.3 > > > In attempting to debug another Kube problem locally using docker-desktop I > discovered that the fetching of XCom doesn't work on 1.13 - the events Kube > returns changed somewhere between 1.10 and 1.13 > (The problem was the side-car container was exiting with non 0, which caused > the operator to fail.) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4052) To allow filtering using "event" and "owner" in "Log" view
[ https://issues.apache.org/jira/browse/AIRFLOW-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-4052. --- Resolution: Fixed Fix Version/s: 1.10.3 > To allow filtering using "event" and "owner" in "Log" view > -- > > Key: AIRFLOW-4052 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4052 > Project: Apache Airflow > Issue Type: Improvement > Components: webapp >Affects Versions: 1.10.2 >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > Fix For: 1.10.3 > > > In the RBAC UI, users can check Logs. But they can only use "dag id", "task > id", "execution date", or "extra" to filter, while filtering using "event" > and "owner" will be very useful (to allow users to check specific events > happened, or check what a specific user did). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3885) Improve Travis buildtime
[ https://issues.apache.org/jira/browse/AIRFLOW-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787168#comment-16787168 ] Tao Feng commented on AIRFLOW-3885: --- [~ashb], you need this pr([https://github.com/apache/airflow/pull/4737)] as well. > Improve Travis buildtime > > > Key: AIRFLOW-3885 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3885 > Project: Apache Airflow > Issue Type: Bug > Components: travis >Affects Versions: 1.10.2 >Reporter: Drew Sonne >Assignee: Drew Sonne >Priority: Major > Fix For: 1.10.3 > > > * Remove the "install" action on the "pre-test" stage to avoid performing > lengthy Docker pulls to perform pre-checks > * Set nosetests to return on any test failures > ** Given the lengthy runtime of the airflow CI test suites, if any tests > fail, we should fail immediately and return the failed test locally. Users > can run the tests locally to get full lists of failed tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3892) Create RedisPubSubSensor
[ https://issues.apache.org/jira/browse/AIRFLOW-3892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-3892. --- Resolution: Fixed > Create RedisPubSubSensor > > > Key: AIRFLOW-3892 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3892 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib >Affects Versions: 1.10.2 >Reporter: Mans Singh >Assignee: Mans Singh >Priority: Minor > Labels: pubsub, redis, sensor > Fix For: 1.10.3 > > Original Estimate: 12h > Remaining Estimate: 12h > > Create a sensor that checks for redis message -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3258) Add ability to set Environment Variables for pods brought up by Kubernetes Executor
[ https://issues.apache.org/jira/browse/AIRFLOW-3258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-3258. --- Resolution: Fixed Fix Version/s: (was: 1.10.0) 1.10.3 > Add ability to set Environment Variables for pods brought up by Kubernetes > Executor > --- > > Key: AIRFLOW-3258 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3258 > Project: Apache Airflow > Issue Type: Improvement > Components: kubernetes >Affects Versions: 1.10.0 >Reporter: David Lum >Assignee: David Lum >Priority: Minor > Labels: kubernetes > Fix For: 1.10.3 > > > Currently the only way to add arbitrary environment variables to pods brought > up by the KubernetesExecutor is to add them using Kubernetes secrets, and > then specifying those secrets in the in the kubernetes_secrets section of the > the airflow.cfg. However, not all environment variables are secrets and need > to be encoded. The suggestion would be to add another section to the > airflow.cfg, kubernetes_environment_variables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3981) Make Airflow UI timezone aware
[ https://issues.apache.org/jira/browse/AIRFLOW-3981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16785079#comment-16785079 ] Tao Feng commented on AIRFLOW-3981: --- cc [~verdan] our Airflow UI expect [~tronbabylove] I asked your question trough FAB issues(https://github.com/dpgaspar/Flask-AppBuilder/issues/920). > Make Airflow UI timezone aware > -- > > Key: AIRFLOW-3981 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3981 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Tao Feng >Assignee: Andrew Stahlman >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4020) remove viewer DAG edit permission
[ https://issues.apache.org/jira/browse/AIRFLOW-4020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-4020. --- Resolution: Fixed Fix Version/s: 1.10.3 > remove viewer DAG edit permission > - > > Key: AIRFLOW-4020 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4020 > Project: Apache Airflow > Issue Type: Improvement >Affects Versions: 1.10.2 >Reporter: Tao Feng >Assignee: Tao Feng >Priority: Major > Fix For: 1.10.3 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4020) remove viewer DAG edit permission
Tao Feng created AIRFLOW-4020: - Summary: remove viewer DAG edit permission Key: AIRFLOW-4020 URL: https://issues.apache.org/jira/browse/AIRFLOW-4020 Project: Apache Airflow Issue Type: Improvement Affects Versions: 1.10.2 Reporter: Tao Feng Assignee: Tao Feng -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1557) backfill ignores configured number of slots in a pool
[
https://issues.apache.org/jira/browse/AIRFLOW-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784115#comment-16784115
]
Tao Feng commented on AIRFLOW-1557:
---
it shouldn't be that difficult to fix. My understanding is that we need to
incorporate this
logic([https://github.com/apache/airflow/blob/master/airflow/jobs.py#L1106-L1207)]
into backfillJob scheduler which two things need to take into consideration:
1. open slot setting based on pool; 2. task_concurrency_limit
definition([https://github.com/apache/airflow/blob/master/airflow/jobs.py#L1025)]
cc [~tronbabylove] [~milton0825] , did either of you want to take a stab?
> backfill ignores configured number of slots in a pool
> -
>
> Key: AIRFLOW-1557
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1557
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.8.1
>Reporter: Ash Berlin-Taylor
>Priority: Major
> Attachments: Backfill_bug_1.8.2.jpg, Screen Shot 2017-09-01 at
> 11.39.32.png
>
>
> I ran a backfill command with this tool:
> The backfill process appears to run as many tasks as possible, even when the
> pool it is running in should limit the numbers:
> {noformat}
> airflow backfill \
> -t fetch_dk_unfiltered \
> --pool brand_index_api \
> -s 2017-07-31 -e 2017-08-31 \
> -x \
> brand_index_fetcher
> {noformat}
> (Nothing else than the backfill is currently using this pool. I wasn't able
> to capture a screen shot of the task instances before the jobs completed.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-1249) Running tasks from backfills with unmet conditions are stuck running
[
https://issues.apache.org/jira/browse/AIRFLOW-1249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Feng updated AIRFLOW-1249:
--
Labels: backfill (was: )
> Running tasks from backfills with unmet conditions are stuck running
>
>
> Key: AIRFLOW-1249
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1249
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Dan Davydov
>Priority: Major
> Labels: backfill
>
> Running tasks from backfills with unmet conditions are stuck running. E.g.
> create a DAG and pause it, mark the dagrun for a specific date as failed in
> the UI, and then start a backfill for that date. Note that the task is stuck
> in the running or queued state with no start/end date set. I believe this
> occurs for the Celery executor but it may affect other/all executors.
> This can e.g. cause pools to fill up.
> This query cleans up the tasks as a workaround:
> {code}
> DELETE ti FROM task_instance ti
> JOIN dag_run dr
> ON ti.execution_date = dr.execution_date AND
>ti.dag_id = dr.dag_id
>JOIN dag dg
>ON dr.dag_id=dg.dag_id
> WHERE ISNULL(ti.start_date) and (ti.state="queued" or
> ti.state="running")
> and dr.state="failed" and (dg.is_paused="1" or dg.is_active="0");
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4011) Add Classmethod as an Airflow user
[
https://issues.apache.org/jira/browse/AIRFLOW-4011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Feng resolved AIRFLOW-4011.
---
Resolution: Fixed
Fix Version/s: 1.10.3
> Add Classmethod as an Airflow user
> --
>
> Key: AIRFLOW-4011
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4011
> Project: Apache Airflow
> Issue Type: Wish
> Components: docs
>Reporter: Sho Ito
>Priority: Trivial
> Labels: documentation
> Fix For: 1.10.3
>
>
> Add to "Who uses Apache Airflow?" in README.md:
> {code:none}
> 1. [Classmethod, Inc.](https://classmethod.jp/)
> [[@shoito](https://github.com/shoito)]{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4006) Make better use of Set in AirflowSecurityManager for performance & simplicity
[ https://issues.apache.org/jira/browse/AIRFLOW-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-4006. --- Resolution: Fixed Fix Version/s: 1.10.3 > Make better use of Set in AirflowSecurityManager for performance & simplicity > - > > Key: AIRFLOW-4006 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4006 > Project: Apache Airflow > Issue Type: Improvement > Components: webserver >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > Fix For: 1.10.3 > > > There are a few parts in AirflowSecurityManager in which we can improve > performance and simplicity by making better use of Set. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-3981) Make Airflow UI timezone aware
[ https://issues.apache.org/jira/browse/AIRFLOW-3981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng reassigned AIRFLOW-3981: - Assignee: Andrew Stahlman > Make Airflow UI timezone aware > -- > > Key: AIRFLOW-3981 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3981 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Tao Feng >Assignee: Andrew Stahlman >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4001) Update docs about how to run tests
[ https://issues.apache.org/jira/browse/AIRFLOW-4001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-4001. --- Resolution: Fixed > Update docs about how to run tests > -- > > Key: AIRFLOW-4001 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4001 > Project: Apache Airflow > Issue Type: Bug >Reporter: Chao-Han Tsai >Assignee: Chao-Han Tsai >Priority: Major > > The steps in [CONTRIBUTING.md | > https://github.com/apache/airflow/blob/master/CONTRIBUTING.md ] about how to > run tests are out-dated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3977) Incorrect example about the interaction between skipped tasks and trigger rules in documentation.
[ https://issues.apache.org/jira/browse/AIRFLOW-3977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-3977. --- Resolution: Fixed > Incorrect example about the interaction between skipped tasks and trigger > rules in documentation. > - > > Key: AIRFLOW-3977 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3977 > Project: Apache Airflow > Issue Type: Bug >Affects Versions: 1.8.2, 1.9.0, 1.10.0, 1.10.1, 1.10.2 >Reporter: cixuuz >Assignee: cixuuz >Priority: Major > Labels: documentaion > Fix For: 1.10.3 > > > Current LatestOnlyOperator will skip all downstream tasks blindly. > BranchingOperator could be a better example to show how trigger rules > interacted with skipped tasks in schedule level. > This fix can also resolve this ticket: > https://issues.apache.org/jira/browse/AIRFLOW-1784 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3975) Handle null values in attr renderers
[ https://issues.apache.org/jira/browse/AIRFLOW-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-3975. --- Resolution: Fixed > Handle null values in attr renderers > > > Key: AIRFLOW-3975 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3975 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Josh Carp >Assignee: Josh Carp >Priority: Trivial > > Some renderers in `attr_renderers` raise unhandled exceptions when given null > inputs. For example, the `python_callable` renderer raises an error if passed > `None`. Some operators allow null values for this attribute, such as > `TriggerDagRunOperator`. I think all renderers should handle null input by > returning the empty string and not raising an exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3983) Exclude node_modules from being linted by flake8
[
https://issues.apache.org/jira/browse/AIRFLOW-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Feng resolved AIRFLOW-3983.
---
Resolution: Fixed
> Exclude node_modules from being linted by flake8
>
>
> Key: AIRFLOW-3983
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3983
> Project: Apache Airflow
> Issue Type: Improvement
>Reporter: Felix Uellendall
>Assignee: Felix Uellendall
>Priority: Minor
>
> After installing node modules via {noformat}npm install{noformat}. Flake8
> (our current linting tool for python) will also lint these packages that we
> are not responsible for keeping clean.
> So it should be excluded for our flake8 linting in general.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3992) run-ci.sh should be re-runable
[
https://issues.apache.org/jira/browse/AIRFLOW-3992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Feng resolved AIRFLOW-3992.
---
Resolution: Fixed
> run-ci.sh should be re-runable
> --
>
> Key: AIRFLOW-3992
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3992
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Chao-Han Tsai
>Assignee: Chao-Han Tsai
>Priority: Major
>
> I am following the development setup in
> https://github.com/apache/airflow/blob/master/CONTRIBUTING.md
> and when I try to rerun:
> {code}
> /app/scripts/ci/run-ci.sh
> {code}
> in side the container it failed with:
> {code}
> + ln -s /home/airflow/.ssh/authorized_keys /home/airflow/.ssh/authorized_keys2
> ln: failed to create symbolic link '/home/airflow/.ssh/authorized_keys2':
> File exists
> ERROR: InvocationError for command '/app/scripts/ci/1-setup-env.sh' (exited
> with code 1)
> __ summary
> ___
> ERROR: py27-backend_sqlite-env_docker: commands failed
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3766) Kubernetes executor annotations
[ https://issues.apache.org/jira/browse/AIRFLOW-3766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-3766. --- Resolution: Fixed Assignee: Stijn De Haes > Kubernetes executor annotations > --- > > Key: AIRFLOW-3766 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3766 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Stijn De Haes >Assignee: Stijn De Haes >Priority: Major > > There is support for annotations on the kubernetes operator however for the > executor this does not work yet. We could add kubernetes_annotations which > could behave like kubernetes_node_selectors for example. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3870) SFTPOperator to push filepath into xcom
[ https://issues.apache.org/jira/browse/AIRFLOW-3870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-3870. --- Resolution: Fixed > SFTPOperator to push filepath into xcom > --- > > Key: AIRFLOW-3870 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3870 > Project: Apache Airflow > Issue Type: Improvement >Reporter: jack >Priority: Minor > > push filepath into xcom and update info messages. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3981) Make Airflow UI timezone aware
[ https://issues.apache.org/jira/browse/AIRFLOW-3981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng updated AIRFLOW-3981: -- Summary: Make Airflow UI timezone aware (was: Makr Airflow UI timezone aware) > Make Airflow UI timezone aware > -- > > Key: AIRFLOW-3981 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3981 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Tao Feng >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3981) Makr Airflow UI timezone aware
Tao Feng created AIRFLOW-3981: - Summary: Makr Airflow UI timezone aware Key: AIRFLOW-3981 URL: https://issues.apache.org/jira/browse/AIRFLOW-3981 Project: Apache Airflow Issue Type: Improvement Reporter: Tao Feng -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-2221) Fill up DagBag from remote locations
[ https://issues.apache.org/jira/browse/AIRFLOW-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng reassigned AIRFLOW-2221: - Assignee: Chao-Han Tsai (was: Diogo Franco) > Fill up DagBag from remote locations > > > Key: AIRFLOW-2221 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2221 > Project: Apache Airflow > Issue Type: New Feature > Components: configuration, core >Affects Versions: 2.0.0 >Reporter: Diogo Franco >Assignee: Chao-Han Tsai >Priority: Major > Fix For: 2.0.0 > > > The ability to fill up the DagBag from remote locations (HDFS, S3...) seems > to be deemed useful, e.g. facilitating deployment processes. > This JIRA is to propose an implementation of a *DagFetcher* abstraction on > the DagBag, where the collect_dags method can delegate the walking to a > *FileSystemDagFetcher*, *GitRepoDagFetcher*, *S3DagFetcher*, > *HDFSDagFetcher*, *GCSDagFetcher*, *ArtifactoryDagFetcher* or even > *TarballInS3DagFetcher*. > This was briefly discussed in [this mailing list > thread|https://lists.apache.org/thread.html/03ddcd3a42b7fd6e3dad9711e8adea37fc00391f6053762f73af5b6a@%3Cdev.airflow.apache.org%3E] > I'm happy to start work on this and provide an initial implementation for > review. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2221) Fill up DagBag from remote locations
[ https://issues.apache.org/jira/browse/AIRFLOW-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16781297#comment-16781297 ] Tao Feng commented on AIRFLOW-2221: --- [~milton0825], feel free to take it. But since it includes core change, we need to have an AIP(airflow improvement proposal) now. There is already one for this topic([https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+DagFetcher)] which you could talk to the author and see if you could take it over for discussion. > Fill up DagBag from remote locations > > > Key: AIRFLOW-2221 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2221 > Project: Apache Airflow > Issue Type: New Feature > Components: configuration, core >Affects Versions: 2.0.0 >Reporter: Diogo Franco >Assignee: Diogo Franco >Priority: Major > Fix For: 2.0.0 > > > The ability to fill up the DagBag from remote locations (HDFS, S3...) seems > to be deemed useful, e.g. facilitating deployment processes. > This JIRA is to propose an implementation of a *DagFetcher* abstraction on > the DagBag, where the collect_dags method can delegate the walking to a > *FileSystemDagFetcher*, *GitRepoDagFetcher*, *S3DagFetcher*, > *HDFSDagFetcher*, *GCSDagFetcher*, *ArtifactoryDagFetcher* or even > *TarballInS3DagFetcher*. > This was briefly discussed in [this mailing list > thread|https://lists.apache.org/thread.html/03ddcd3a42b7fd6e3dad9711e8adea37fc00391f6053762f73af5b6a@%3Cdev.airflow.apache.org%3E] > I'm happy to start work on this and provide an initial implementation for > review. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2767) Upgrade gunicorn to 19.5.0 or greater to avoid moderate-severity CVE
[ https://issues.apache.org/jira/browse/AIRFLOW-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-2767. --- Resolution: Fixed Assignee: (was: Siddharth Anand) > Upgrade gunicorn to 19.5.0 or greater to avoid moderate-severity CVE > > > Key: AIRFLOW-2767 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2767 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Siddharth Anand >Priority: Major > > Refer to the moderate-severity CVE in gunicorn 19.4.5 (apparently fixed in > 19.5.0) > [https://nvd.nist.gov/vuln/detail/CVE-2018-1000164] > Currently, apache airflow's setup.py allows 19.4.0 > -s -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3949) Users should only see the DAGs to which he/she has "can_dag_view"/"can_dag_edit" permission in the landing page
[ https://issues.apache.org/jira/browse/AIRFLOW-3949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778482#comment-16778482 ] Tao Feng commented on AIRFLOW-3949: --- [~XD-DENG] I guess the issue is that User initially has can_dag_read on all_dags. And since you remove the permission on all_dags, we don't resync the permissions for user role([https://github.com/apache/airflow/blob/master/airflow/www/security.py#L186)] which the User still has all_dag access. > Users should only see the DAGs to which he/she has > "can_dag_view"/"can_dag_edit" permission in the landing page > --- > > Key: AIRFLOW-3949 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3949 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Reporter: Xiaodong DENG >Assignee: Tao Feng >Priority: Major > > In the current master branch > ([https://github.com/apache/airflow/commit/bfa81b53597907ed58b2e01a69ba9fd52ce4a7b9)] > and 1.10.2, the DAG-level access control feature is already there. > According to Feng Tao, in his initial implementation, users aren't able to > see the DAGs to which he/she doesn't have access. But in the testing I have > done, seems I can still see all the DAGs as a role "User" after I have > removed the "can_dag_view on all_dags" and "can_dag_edit on all_dags" from > role "User". > > (The testing was done using the "built-in" sample DAGs only) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3949) Users should only see the DAGs to which he/she has "can_dag_view"/"can_dag_edit" permission in the landing page
[ https://issues.apache.org/jira/browse/AIRFLOW-3949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778483#comment-16778483 ] Tao Feng commented on AIRFLOW-3949: --- I don't think this is an issue in this case. > Users should only see the DAGs to which he/she has > "can_dag_view"/"can_dag_edit" permission in the landing page > --- > > Key: AIRFLOW-3949 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3949 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Reporter: Xiaodong DENG >Assignee: Tao Feng >Priority: Major > > In the current master branch > ([https://github.com/apache/airflow/commit/bfa81b53597907ed58b2e01a69ba9fd52ce4a7b9)] > and 1.10.2, the DAG-level access control feature is already there. > According to Feng Tao, in his initial implementation, users aren't able to > see the DAGs to which he/she doesn't have access. But in the testing I have > done, seems I can still see all the DAGs as a role "User" after I have > removed the "can_dag_view on all_dags" and "can_dag_edit on all_dags" from > role "User". > > (The testing was done using the "built-in" sample DAGs only) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3949) Users should only see the DAGs to which he/she has "can_dag_view"/"can_dag_edit" permission in the landing page
[ https://issues.apache.org/jira/browse/AIRFLOW-3949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-3949. --- Resolution: Not A Problem > Users should only see the DAGs to which he/she has > "can_dag_view"/"can_dag_edit" permission in the landing page > --- > > Key: AIRFLOW-3949 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3949 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Reporter: Xiaodong DENG >Assignee: Tao Feng >Priority: Major > > In the current master branch > ([https://github.com/apache/airflow/commit/bfa81b53597907ed58b2e01a69ba9fd52ce4a7b9)] > and 1.10.2, the DAG-level access control feature is already there. > According to Feng Tao, in his initial implementation, users aren't able to > see the DAGs to which he/she doesn't have access. But in the testing I have > done, seems I can still see all the DAGs as a role "User" after I have > removed the "can_dag_view on all_dags" and "can_dag_edit on all_dags" from > role "User". > > (The testing was done using the "built-in" sample DAGs only) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3950) Improve AirflowSecurityManager.update_admin_perm_view
[ https://issues.apache.org/jira/browse/AIRFLOW-3950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-3950. --- Resolution: Fixed > Improve AirflowSecurityManager.update_admin_perm_view > - > > Key: AIRFLOW-3950 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3950 > Project: Apache Airflow > Issue Type: Improvement > Components: webserver >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > > The implementation of `AirflowSecurityManager.update_admin_perm_view` can be > simplified using set. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3947) Add "No access" flash msg for DAG-level access control
[ https://issues.apache.org/jira/browse/AIRFLOW-3947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-3947. --- Resolution: Fixed > Add "No access" flash msg for DAG-level access control > -- > > Key: AIRFLOW-3947 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3947 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > > In FAB UI, when user clicks a page to which he/she doesn't have access, there > will be a "Access is Denied" flash message. > But for the DAG-level access control, when the user does that, he/she would > be redirected to the main page without any flash message. This may be > confusing to the user. > We should add proper flash warning message in the UI for this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3945) Row of NULLs inserted when not changing permission views
[ https://issues.apache.org/jira/browse/AIRFLOW-3945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-3945. --- Resolution: Fixed > Row of NULLs inserted when not changing permission views > > > Key: AIRFLOW-3945 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3945 > Project: Apache Airflow > Issue Type: Bug > Components: webserver >Affects Versions: 1.10.2 >Reporter: Diederik Perdok >Assignee: Diederik Perdok >Priority: Minor > Fix For: 1.10.3 > > > When creating new permission views in > AirflowSecurityManager.create_custom_dag_permission_view in file > airflow/www/security.py, the list of permission views to update might be > empty because everything is still up to date. In that case, a row in which > only the id is not NULL will still be inserted in table > ab_permission_view_role every time. > This is a minor bug when using PostgreSQL. When using a SQL Server backend, > this causes the webserver to crash as SQL Server disallows multiple NULLs in > a unique constraint (by default). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3926) Remove flask-admin usage
[ https://issues.apache.org/jira/browse/AIRFLOW-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-3926. --- Resolution: Fixed > Remove flask-admin usage > > > Key: AIRFLOW-3926 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3926 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Tao Feng >Assignee: Andrew Stahlman >Priority: Major > Fix For: 2.0.0 > > > Given we remove old UI which built up flask-admin, we should remove its usage > as well. > > AirflowViewPlugin - not referenced anywhere in the project: > https://github.com/apache/airflow/blob/master/airflow/__init__.py#L75 > Hive Metastore Browser: > https://github.com/apache/airflow/blame/master/airflow/contrib/plugins/metastore_browser/main.py#L48 > unit test for plugins, uses BaseView and Menu Link: > https://github.com/apache/airflow/blob/master/tests/plugins/test_plugin.py#L101 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3940) Modify metastore_browser plugin to use FAB
[ https://issues.apache.org/jira/browse/AIRFLOW-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-3940. --- Resolution: Fixed > Modify metastore_browser plugin to use FAB > -- > > Key: AIRFLOW-3940 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3940 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Tao Feng >Assignee: Andrew Stahlman >Priority: Major > > that plugin has been broken for a while. Let's just remove it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3940) Modify metastore_browser plugin to use FAB
[ https://issues.apache.org/jira/browse/AIRFLOW-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng updated AIRFLOW-3940: -- Summary: Modify metastore_browser plugin to use FAB (was: Remove metastore_browser plugin) > Modify metastore_browser plugin to use FAB > -- > > Key: AIRFLOW-3940 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3940 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Tao Feng >Assignee: Andrew Stahlman >Priority: Major > > that plugin has been broken for a while. Let's just remove it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
