[jira] [Resolved] (AIRFLOW-6949) SparkSubitOperator ignores explicit spark.kubernetes.namespace config option
[
https://issues.apache.org/jira/browse/AIRFLOW-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fokko Driesprong resolved AIRFLOW-6949.
---
Resolution: Fixed
> SparkSubitOperator ignores explicit spark.kubernetes.namespace config option
>
>
> Key: AIRFLOW-6949
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6949
> Project: Apache Airflow
> Issue Type: Bug
> Components: hooks
>Affects Versions: 1.10.9
>Reporter: Ash Berlin-Taylor
>Assignee: Ash Berlin-Taylor
>Priority: Minor
> Fix For: 1.10.10
>
>
> If a user explicitly passes {{spark.kubernetes.namespace}} in the config
> attribute, we should respect that over what is given in the connection.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6887) Do not check the state of fresh DAGRun
[ https://issues.apache.org/jira/browse/AIRFLOW-6887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-6887. --- Fix Version/s: 2.0.0 Assignee: Kamil Bregula Resolution: Fixed > Do not check the state of fresh DAGRun > -- > > Key: AIRFLOW-6887 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6887 > Project: Apache Airflow > Issue Type: New Feature > Components: scheduler >Affects Versions: 1.10.9 >Reporter: Kamil Bregula >Assignee: Kamil Bregula >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-6867) Decouple DagBag and TaskInstance
[ https://issues.apache.org/jira/browse/AIRFLOW-6867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong reassigned AIRFLOW-6867: - Assignee: Kamil Bregula > Decouple DagBag and TaskInstance > > > Key: AIRFLOW-6867 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6867 > Project: Apache Airflow > Issue Type: Bug > Components: DAG >Affects Versions: 1.10.9 >Reporter: Kamil Bregula >Assignee: Kamil Bregula >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6224) Fix multiple Alembic heads
[ https://issues.apache.org/jira/browse/AIRFLOW-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong updated AIRFLOW-6224: -- Fix Version/s: (was: 1.10.7) 2.0.0 > Fix multiple Alembic heads > -- > > Key: AIRFLOW-6224 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6224 > Project: Apache Airflow > Issue Type: Bug > Components: database >Affects Versions: 2.0.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (AIRFLOW-6224) Fix multiple Alembic heads
[ https://issues.apache.org/jira/browse/AIRFLOW-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-6224 started by Fokko Driesprong. - > Fix multiple Alembic heads > -- > > Key: AIRFLOW-6224 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6224 > Project: Apache Airflow > Issue Type: Bug > Components: database >Affects Versions: 2.0.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6224) Fix multiple Alembic heads
[ https://issues.apache.org/jira/browse/AIRFLOW-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong updated AIRFLOW-6224: -- Affects Version/s: (was: 1.10.6) 2.0.0 > Fix multiple Alembic heads > -- > > Key: AIRFLOW-6224 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6224 > Project: Apache Airflow > Issue Type: Bug > Components: database >Affects Versions: 2.0.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 1.10.7 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6224) Fix multiple Alembic heads
Fokko Driesprong created AIRFLOW-6224: - Summary: Fix multiple Alembic heads Key: AIRFLOW-6224 URL: https://issues.apache.org/jira/browse/AIRFLOW-6224 Project: Apache Airflow Issue Type: Bug Components: database Affects Versions: 1.10.6 Reporter: Fokko Driesprong Assignee: Fokko Driesprong Fix For: 1.10.7 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-6143) Inconsistent pylint hive changed merged to master
[ https://issues.apache.org/jira/browse/AIRFLOW-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong reassigned AIRFLOW-6143: - Assignee: Jarek Potiuk > Inconsistent pylint hive changed merged to master > - > > Key: AIRFLOW-6143 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6143 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Assignee: Jarek Potiuk >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6143) Inconsistent pylint hive changed merged to master
[ https://issues.apache.org/jira/browse/AIRFLOW-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-6143. --- Fix Version/s: 2.0.0 Resolution: Fixed > Inconsistent pylint hive changed merged to master > - > > Key: AIRFLOW-6143 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6143 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Assignee: Jarek Potiuk >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5804) Batch the xcom pull operation
[ https://issues.apache.org/jira/browse/AIRFLOW-5804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-5804. --- Resolution: Fixed > Batch the xcom pull operation > - > > Key: AIRFLOW-5804 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5804 > Project: Apache Airflow > Issue Type: Improvement > Components: xcom >Affects Versions: 1.10.5 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 2.0.0 > > > Right now the xcom_pull of the task_instance will call XCom.get_one many > times, therefore I'd like to change this to XCom.get_many. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-5929) Improve Scheduler Performance
[ https://issues.apache.org/jira/browse/AIRFLOW-5929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong reassigned AIRFLOW-5929: - Assignee: Ash Berlin-Taylor > Improve Scheduler Performance > - > > Key: AIRFLOW-5929 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5929 > Project: Apache Airflow > Issue Type: Epic > Components: scheduler >Affects Versions: 2.0.0 >Reporter: Ash Berlin-Taylor >Assignee: Ash Berlin-Taylor >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5806) Simplify the xcom table
[ https://issues.apache.org/jira/browse/AIRFLOW-5806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong updated AIRFLOW-5806: -- Summary: Simplify the xcom table (was: Drop id from xcom table) > Simplify the xcom table > --- > > Key: AIRFLOW-5806 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5806 > Project: Apache Airflow > Issue Type: Improvement > Components: xcom >Affects Versions: 1.10.5 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5806) Simplify the xcom table
[ https://issues.apache.org/jira/browse/AIRFLOW-5806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong updated AIRFLOW-5806: -- Description: The id column isn't used anywhere. > Simplify the xcom table > --- > > Key: AIRFLOW-5806 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5806 > Project: Apache Airflow > Issue Type: Improvement > Components: xcom >Affects Versions: 1.10.5 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 2.0.0 > > > The id column isn't used anywhere. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (AIRFLOW-5806) Drop id from xcom table
[ https://issues.apache.org/jira/browse/AIRFLOW-5806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-5806 started by Fokko Driesprong. - > Drop id from xcom table > --- > > Key: AIRFLOW-5806 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5806 > Project: Apache Airflow > Issue Type: Improvement > Components: xcom >Affects Versions: 1.10.5 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5806) Drop id from xcom table
Fokko Driesprong created AIRFLOW-5806: - Summary: Drop id from xcom table Key: AIRFLOW-5806 URL: https://issues.apache.org/jira/browse/AIRFLOW-5806 Project: Apache Airflow Issue Type: Improvement Components: xcom Affects Versions: 1.10.5 Reporter: Fokko Driesprong Fix For: 2.0.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5805) Make the xcom_push atomic
Fokko Driesprong created AIRFLOW-5805: - Summary: Make the xcom_push atomic Key: AIRFLOW-5805 URL: https://issues.apache.org/jira/browse/AIRFLOW-5805 Project: Apache Airflow Issue Type: Improvement Components: xcom Affects Versions: 1.10.5 Reporter: Fokko Driesprong Assignee: Fokko Driesprong Fix For: 2.0.0 There is a commit after the delete, we don't need this actually. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5804) Batch the xcom pull operation
Fokko Driesprong created AIRFLOW-5804: - Summary: Batch the xcom pull operation Key: AIRFLOW-5804 URL: https://issues.apache.org/jira/browse/AIRFLOW-5804 Project: Apache Airflow Issue Type: Improvement Components: xcom Affects Versions: 1.10.5 Reporter: Fokko Driesprong Assignee: Fokko Driesprong Fix For: 2.0.0 Right now the xcom_pull of the task_instance will call XCom.get_one many times, therefore I'd like to change this to XCom.get_many. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-4930) Druid hook sends json ingestion spec string using json= parameter of requests, not body=
[ https://issues.apache.org/jira/browse/AIRFLOW-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-4930. --- Fix Version/s: 1.10.7 2.0.0 Resolution: Fixed > Druid hook sends json ingestion spec string using json= parameter of > requests, not body= > > > Key: AIRFLOW-4930 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4930 > Project: Apache Airflow > Issue Type: Bug > Components: hooks >Affects Versions: 1.10.3 > Environment: druid 0.15.0 > requests 2.22.0 >Reporter: Hyeungshik Jung >Assignee: Hyeungshik Jung >Priority: Trivial > Labels: easyfix > Fix For: 2.0.0, 1.10.7 > > > Hi all, I recently tried to add druid ingestion task to my airflow DAG, > and found that druid hook for airflow is sending json string as `json=`, > which makes druid coordinator impossible to parse the request body. > [https://github.com/apache/airflow/blob/master/airflow/hooks/druid_hook.py#L71] > I changed it to send `json_index_spec` using `data=`, and my ingestion job > worked as expected. I'm not sure whether this is a bug or environment > specific issue. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5793) add test for multiple alembic revision heads
[ https://issues.apache.org/jira/browse/AIRFLOW-5793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-5793. --- Fix Version/s: 2.0.0 Resolution: Fixed > add test for multiple alembic revision heads > > > Key: AIRFLOW-5793 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5793 > Project: Apache Airflow > Issue Type: Improvement > Components: tests >Affects Versions: 1.10.5 >Reporter: Daniel Standish >Assignee: Daniel Standish >Priority: Major > Fix For: 2.0.0 > > > Depending on the timing of merges with migrations, we can end up with two > revision heads that need to be merged. > This adds a test to detect when multiple heads are present. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5771) Merge alembic heads
[ https://issues.apache.org/jira/browse/AIRFLOW-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-5771. --- Resolution: Fixed > Merge alembic heads > --- > > Key: AIRFLOW-5771 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5771 > Project: Apache Airflow > Issue Type: Bug > Components: database >Affects Versions: 2.0.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5792) Straighten out Alembic table migrations
Fokko Driesprong created AIRFLOW-5792: - Summary: Straighten out Alembic table migrations Key: AIRFLOW-5792 URL: https://issues.apache.org/jira/browse/AIRFLOW-5792 Project: Apache Airflow Issue Type: Improvement Components: database Affects Versions: 1.10.5 Reporter: Fokko Driesprong Assignee: Fokko Driesprong Fix For: 1.10.6 Right now we have 3 heads, and this makes future migrations more painful. By changing this, we only have one head as it should :-) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (AIRFLOW-5792) Straighten out Alembic table migrations
[ https://issues.apache.org/jira/browse/AIRFLOW-5792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-5792 started by Fokko Driesprong. - > Straighten out Alembic table migrations > --- > > Key: AIRFLOW-5792 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5792 > Project: Apache Airflow > Issue Type: Improvement > Components: database >Affects Versions: 1.10.5 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 1.10.6 > > > Right now we have 3 heads, and this makes future migrations more painful. By > changing this, we only have one head as it should :-) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5789) Make xcom_pull a single query
Fokko Driesprong created AIRFLOW-5789: - Summary: Make xcom_pull a single query Key: AIRFLOW-5789 URL: https://issues.apache.org/jira/browse/AIRFLOW-5789 Project: Apache Airflow Issue Type: Improvement Components: xcom Affects Versions: 1.10.5 Reporter: Fokko Driesprong Assignee: Fokko Driesprong Fix For: 2.0.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5771) Merge alembic heads
Fokko Driesprong created AIRFLOW-5771: - Summary: Merge alembic heads Key: AIRFLOW-5771 URL: https://issues.apache.org/jira/browse/AIRFLOW-5771 Project: Apache Airflow Issue Type: Bug Components: database Affects Versions: 2.0.0 Reporter: Fokko Driesprong Assignee: Fokko Driesprong Fix For: 2.0.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5767) Remove id column from xcom
Fokko Driesprong created AIRFLOW-5767: - Summary: Remove id column from xcom Key: AIRFLOW-5767 URL: https://issues.apache.org/jira/browse/AIRFLOW-5767 Project: Apache Airflow Issue Type: Bug Components: database Affects Versions: 2.0.0 Reporter: Fokko Driesprong Assignee: Fokko Driesprong Fix For: 2.0.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5474) Add Basic Auth to Druid hook
[ https://issues.apache.org/jira/browse/AIRFLOW-5474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-5474. --- Fix Version/s: 2.0.0 Resolution: Fixed > Add Basic Auth to Druid hook > > > Key: AIRFLOW-5474 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5474 > Project: Apache Airflow > Issue Type: Improvement > Components: hooks >Affects Versions: 1.10.5 >Reporter: Adam Welsh >Assignee: Adam Welsh >Priority: Minor > Fix For: 2.0.0 > > > Use login and password from druid ingestion connection to add Basic HTTP auth > to druid hook. If login and/or password is None then ensure hook still works. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5701) Don't clear xcom explicitly before execution
[ https://issues.apache.org/jira/browse/AIRFLOW-5701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-5701. --- Resolution: Fixed > Don't clear xcom explicitly before execution > > > Key: AIRFLOW-5701 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5701 > Project: Apache Airflow > Issue Type: Bug > Components: xcom >Affects Versions: 1.10.5 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5701) Dont clear xcom explicitly before execution
Fokko Driesprong created AIRFLOW-5701: - Summary: Dont clear xcom explicitly before execution Key: AIRFLOW-5701 URL: https://issues.apache.org/jira/browse/AIRFLOW-5701 Project: Apache Airflow Issue Type: Bug Components: xcom Affects Versions: 1.10.5 Reporter: Fokko Driesprong Assignee: Fokko Driesprong Fix For: 2.0.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5701) Don't clear xcom explicitly before execution
[ https://issues.apache.org/jira/browse/AIRFLOW-5701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong updated AIRFLOW-5701: -- Summary: Don't clear xcom explicitly before execution (was: Dont clear xcom explicitly before execution) > Don't clear xcom explicitly before execution > > > Key: AIRFLOW-5701 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5701 > Project: Apache Airflow > Issue Type: Bug > Components: xcom >Affects Versions: 1.10.5 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5688) Merge alembic migrations
Fokko Driesprong created AIRFLOW-5688: - Summary: Merge alembic migrations Key: AIRFLOW-5688 URL: https://issues.apache.org/jira/browse/AIRFLOW-5688 Project: Apache Airflow Issue Type: Bug Components: database Affects Versions: 2.0.0 Reporter: Fokko Driesprong Assignee: Fokko Driesprong Fix For: 2.0.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5586) Improve error messaging for role creation and updates
[ https://issues.apache.org/jira/browse/AIRFLOW-5586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-5586. --- Fix Version/s: 2.0.0 Resolution: Fixed > Improve error messaging for role creation and updates > - > > Key: AIRFLOW-5586 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5586 > Project: Apache Airflow > Issue Type: Improvement > Components: cli >Affects Versions: 1.10.5 >Reporter: Leah >Assignee: Leah >Priority: Minor > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-5430) airflow initdb is broken on py2 and py3.5 due to transitive dependency
[ https://issues.apache.org/jira/browse/AIRFLOW-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong reassigned AIRFLOW-5430: - Assignee: Max Gasner > airflow initdb is broken on py2 and py3.5 due to transitive dependency > --- > > Key: AIRFLOW-5430 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5430 > Project: Apache Airflow > Issue Type: Bug > Components: cli, core, database, dependencies >Affects Versions: 1.10.0, 1.10.1, 1.10.2, 1.10.3, 1.10.4, 1.10.5 >Reporter: Max Gasner >Assignee: Max Gasner >Priority: Major > Fix For: 1.10.6 > > > marshmallow-sqlalchemy broke py2 and py3.5 compatibility in their 0.19.0 > release. > > Flask-AppBuilder's transitive dependency on marshmallow-sqlalchemy breaks all > new installs of Airflow 1.10.0 and higher on py2 and py3.5 with one of the > following syntax errors when running airflow initdb: > > Traceback (most recent call last): > File "/workdir/python_modules/dagster-airflow/.tox/py35/bin/airflow", line > 22, in > from airflow.bin.cli import CLIFactory > File > "/workdir/python_modules/dagster-airflow/.tox/py35/lib/python3.5/site-packages/airflow/bin/cli.py", > line 68, in > from airflow.www_rbac.app import cached_app as cached_app_rbac > File > "/workdir/python_modules/dagster-airflow/.tox/py35/lib/python3.5/site-packages/airflow/www_rbac/app.py", > line 26, in > from flask_appbuilder import AppBuilder, SQLA > File > "/workdir/python_modules/dagster-airflow/.tox/py35/lib/python3.5/site-packages/flask_appbuilder/__init__.py", > line 5, in > from .base import AppBuilder > File > "/workdir/python_modules/dagster-airflow/.tox/py35/lib/python3.5/site-packages/flask_appbuilder/base.py", > line 5, in > from .api.manager import OpenApiManager > File > "/workdir/python_modules/dagster-airflow/.tox/py35/lib/python3.5/site-packages/flask_appbuilder/api/__init__.py", > line 11, in > from marshmallow_sqlalchemy.fields import Related, RelatedList > File > "/workdir/python_modules/dagster-airflow/.tox/py35/lib/python3.5/site-packages/marshmallow_sqlalchemy/__init__.py", > line 1, in > from .schema import TableSchemaOpts, ModelSchemaOpts, TableSchema, > ModelSchema > File > "/workdir/python_modules/dagster-airflow/.tox/py35/lib/python3.5/site-packages/marshmallow_sqlalchemy/schema.py", > line 3, in > from .convert import ModelConverter > File > "/workdir/python_modules/dagster-airflow/.tox/py35/lib/python3.5/site-packages/marshmallow_sqlalchemy/convert.py", > line 80 > ): > ^ > SyntaxError: invalid syntax > > Traceback (most recent call last): > File "/workdir/python_modules/dagster-airflow/.tox/py27/bin/airflow", line > 22, in > from airflow.bin.cli import CLIFactory > File > "/workdir/python_modules/dagster-airflow/.tox/py27/lib/python2.7/site-packages/airflow/bin/cli.py", > line 68, in > from airflow.www_rbac.app import cached_app as cached_app_rbac > File > "/workdir/python_modules/dagster-airflow/.tox/py27/lib/python2.7/site-packages/airflow/www_rbac/app.py", > line 26, in > from flask_appbuilder import AppBuilder, SQLA > File > "/workdir/python_modules/dagster-airflow/.tox/py27/lib/python2.7/site-packages/flask_appbuilder/__init__.py", > line 5, in > from .base import AppBuilder > File > "/workdir/python_modules/dagster-airflow/.tox/py27/lib/python2.7/site-packages/flask_appbuilder/base.py", > line 5, in > from .api.manager import OpenApiManager > File > "/workdir/python_modules/dagster-airflow/.tox/py27/lib/python2.7/site-packages/flask_appbuilder/api/__init__.py", > line 11, in > from marshmallow_sqlalchemy.fields import Related, RelatedList > File > "/workdir/python_modules/dagster-airflow/.tox/py27/lib/python2.7/site-packages/marshmallow_sqlalchemy/__init__.py", > line 1, in > from .schema import TableSchemaOpts, ModelSchemaOpts, TableSchema, > ModelSchema > File > "/workdir/python_modules/dagster-airflow/.tox/py27/lib/python2.7/site-packages/marshmallow_sqlalchemy/schema.py", > line 101 > class TableSchema(ma.Schema, metaclass=TableSchemaMeta): > ^ > SyntaxError: invalid syntax -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (AIRFLOW-5451) Spark Submit Hook don't set namespace if default
Fokko Driesprong created AIRFLOW-5451: - Summary: Spark Submit Hook don't set namespace if default Key: AIRFLOW-5451 URL: https://issues.apache.org/jira/browse/AIRFLOW-5451 Project: Apache Airflow Issue Type: Task Components: hooks Affects Versions: 1.10.5 Reporter: Fokko Driesprong Assignee: Fokko Driesprong Fix For: 2.0.0 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Resolved] (AIRFLOW-5402) Remove deprecated logger
[ https://issues.apache.org/jira/browse/AIRFLOW-5402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-5402. --- Resolution: Fixed > Remove deprecated logger > > > Key: AIRFLOW-5402 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5402 > Project: Apache Airflow > Issue Type: Task > Components: core >Affects Versions: 1.10.4 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Resolved] (AIRFLOW-5208) Passed **kwargs to push_by_returning
[ https://issues.apache.org/jira/browse/AIRFLOW-5208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-5208. --- Assignee: Gaurav Prachchhak Resolution: Fixed > Passed **kwargs to push_by_returning > > > Key: AIRFLOW-5208 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5208 > Project: Apache Airflow > Issue Type: Improvement > Components: examples >Affects Versions: 1.10.4 > Environment: Linux >Reporter: Gaurav Prachchhak >Assignee: Gaurav Prachchhak >Priority: Minor > Attachments: Screen Shot 2019-08-13 at 11.20.14 AM.png > > > Without **kwargs push_by_returning was giving error, so added that parameter. > [https://github.com/apache/airflow/blob/master/airflow/example_dags/example_xcom.py] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (AIRFLOW-5208) Passed **kwargs to push_by_returning
[ https://issues.apache.org/jira/browse/AIRFLOW-5208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong updated AIRFLOW-5208: -- Fix Version/s: 2.0.0 > Passed **kwargs to push_by_returning > > > Key: AIRFLOW-5208 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5208 > Project: Apache Airflow > Issue Type: Improvement > Components: examples >Affects Versions: 1.10.4 > Environment: Linux >Reporter: Gaurav Prachchhak >Assignee: Gaurav Prachchhak >Priority: Minor > Fix For: 2.0.0 > > Attachments: Screen Shot 2019-08-13 at 11.20.14 AM.png > > > Without **kwargs push_by_returning was giving error, so added that parameter. > [https://github.com/apache/airflow/blob/master/airflow/example_dags/example_xcom.py] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work started] (AIRFLOW-5402) Remove deprecated logger
[ https://issues.apache.org/jira/browse/AIRFLOW-5402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-5402 started by Fokko Driesprong. - > Remove deprecated logger > > > Key: AIRFLOW-5402 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5402 > Project: Apache Airflow > Issue Type: Task > Components: core >Affects Versions: 1.10.4 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (AIRFLOW-5402) Remove deprecated logger
Fokko Driesprong created AIRFLOW-5402: - Summary: Remove deprecated logger Key: AIRFLOW-5402 URL: https://issues.apache.org/jira/browse/AIRFLOW-5402 Project: Apache Airflow Issue Type: Task Components: core Affects Versions: 1.10.4 Reporter: Fokko Driesprong Assignee: Fokko Driesprong Fix For: 2.0.0 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (AIRFLOW-5390) Remove provide_context
Fokko Driesprong created AIRFLOW-5390: - Summary: Remove provide_context Key: AIRFLOW-5390 URL: https://issues.apache.org/jira/browse/AIRFLOW-5390 Project: Apache Airflow Issue Type: Task Components: core Affects Versions: 1.10.4 Reporter: Fokko Driesprong Assignee: Fokko Driesprong Fix For: 2.0.0 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Resolved] (AIRFLOW-5269) Reuse existing session of '/health' endpoint to get most recent Scheduler Job
[ https://issues.apache.org/jira/browse/AIRFLOW-5269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-5269. --- Resolution: Fixed > Reuse existing session of '/health' endpoint to get most recent Scheduler Job > - > > Key: AIRFLOW-5269 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5269 > Project: Apache Airflow > Issue Type: Improvement > Components: core, database, scheduler >Affects Versions: 1.10.4, 2.0.0 >Reporter: Kaxil Naik >Assignee: Kaxil Naik >Priority: Minor > Fix For: 2.0.0, 1.10.5 > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (AIRFLOW-5284) Replace warn by warning
Fokko Driesprong created AIRFLOW-5284: - Summary: Replace warn by warning Key: AIRFLOW-5284 URL: https://issues.apache.org/jira/browse/AIRFLOW-5284 Project: Apache Airflow Issue Type: Task Components: scheduler Affects Versions: 1.10.4 Reporter: Fokko Driesprong Assignee: Fokko Driesprong Got a warning on using a deprecated method: dag_processing.py:689: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (AIRFLOW-5268) Rectify DAG naming conventions
[ https://issues.apache.org/jira/browse/AIRFLOW-5268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong reassigned AIRFLOW-5268: - Assignee: Bas Harenslak > Rectify DAG naming conventions > -- > > Key: AIRFLOW-5268 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5268 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Resolved] (AIRFLOW-5268) Rectify DAG naming conventions
[ https://issues.apache.org/jira/browse/AIRFLOW-5268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-5268. --- Fix Version/s: 2.0.0 Resolution: Fixed > Rectify DAG naming conventions > -- > > Key: AIRFLOW-5268 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5268 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Resolved] (AIRFLOW-5247) Getting all dependencies from NPM can be moved up in Dockerfile
[ https://issues.apache.org/jira/browse/AIRFLOW-5247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-5247. --- Fix Version/s: 2.0.0 Resolution: Fixed > Getting all dependencies from NPM can be moved up in Dockerfile > --- > > Key: AIRFLOW-5247 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5247 > Project: Apache Airflow > Issue Type: Improvement > Components: ci >Affects Versions: 1.10.4, 2.0.0 >Reporter: Jarek Potiuk >Assignee: Jarek Potiuk >Priority: Major > Fix For: 2.0.0 > > > Retrieving all NPM dependencies can happen before updating apt-get > dependencies. It takes long time and it does not change that often so it is > quite OK to move it before adding setup.py in dockerfile -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work started] (AIRFLOW-4192) Remove duplicate and obsolete task context variables
[ https://issues.apache.org/jira/browse/AIRFLOW-4192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-4192 started by Fokko Driesprong. - > Remove duplicate and obsolete task context variables > > > Key: AIRFLOW-4192 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4192 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Bas Harenslak >Assignee: Fokko Driesprong >Priority: Major > > More info in the PR: https://github.com/apache/airflow/pull/5010 -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (AIRFLOW-4192) Remove duplicate and obsolete task context variables
[ https://issues.apache.org/jira/browse/AIRFLOW-4192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong reassigned AIRFLOW-4192: - Assignee: Bas Harenslak > Remove duplicate and obsolete task context variables > > > Key: AIRFLOW-4192 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4192 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > > More info in the PR: https://github.com/apache/airflow/pull/5010 -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (AIRFLOW-5111) Remove apt-get upgrade from the Dockerfile
Fokko Driesprong created AIRFLOW-5111: - Summary: Remove apt-get upgrade from the Dockerfile Key: AIRFLOW-5111 URL: https://issues.apache.org/jira/browse/AIRFLOW-5111 Project: Apache Airflow Issue Type: Task Components: ci Affects Versions: 1.10.5 Reporter: Fokko Driesprong Assignee: Fokko Driesprong -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (AIRFLOW-3533) Remove the num_runs option
[ https://issues.apache.org/jira/browse/AIRFLOW-3533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong updated AIRFLOW-3533: -- Fix Version/s: 2.0.0 > Remove the num_runs option > -- > > Key: AIRFLOW-3533 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3533 > Project: Apache Airflow > Issue Type: Task >Reporter: Fokko Driesprong >Priority: Major > Fix For: 2.0.0 > > > This is an old option to let the scheduler periodically restart. But this > should not be necessary. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (AIRFLOW-4959) Add hql template support for DataProcHiveOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-4959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-4959. --- Resolution: Fixed > Add hql template support for DataProcHiveOperator > - > > Key: AIRFLOW-4959 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4959 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.3 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (AIRFLOW-4959) Add hql template support for DataProcHiveOperator
Fokko Driesprong created AIRFLOW-4959: - Summary: Add hql template support for DataProcHiveOperator Key: AIRFLOW-4959 URL: https://issues.apache.org/jira/browse/AIRFLOW-4959 Project: Apache Airflow Issue Type: Improvement Components: gcp Affects Versions: 1.10.3 Reporter: Fokko Driesprong Assignee: Fokko Driesprong Fix For: 2.0.0 -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (AIRFLOW-4942) Drop six.next
[ https://issues.apache.org/jira/browse/AIRFLOW-4942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-4942. --- Resolution: Fixed Fix Version/s: 2.0.0 > Drop six.next > - > > Key: AIRFLOW-4942 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4942 > Project: Apache Airflow > Issue Type: Sub-task > Components: core >Affects Versions: 1.10.3 >Reporter: Kamil >Assignee: Kamil >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (AIRFLOW-4906) Improve debugging for the SparkSubmitHook
Fokko Driesprong created AIRFLOW-4906: - Summary: Improve debugging for the SparkSubmitHook Key: AIRFLOW-4906 URL: https://issues.apache.org/jira/browse/AIRFLOW-4906 Project: Apache Airflow Issue Type: Improvement Components: hooks Affects Versions: 1.10.3 Reporter: Fokko Driesprong Assignee: Fokko Driesprong Fix For: 2.0.0 Currently, the output of the spark-submit command is not being sent to the logs. This makes debugging of the k8s jobs rather hard. For example, if you make a typo, you only will get the exit code which is non-descriptive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4510) Timezone set incorrectly if multiple DAGs defined in the same file
[
https://issues.apache.org/jira/browse/AIRFLOW-4510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fokko Driesprong resolved AIRFLOW-4510.
---
Resolution: Fixed
Fix Version/s: 2.0.0
> Timezone set incorrectly if multiple DAGs defined in the same file
> --
>
> Key: AIRFLOW-4510
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4510
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG
>Affects Versions: 1.10.3
>Reporter: Abhishek Ray
>Assignee: Abhishek Ray
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: Screen Shot 2019-05-13 at 2.41.54 PM.png, Screen Shot
> 2019-05-13 at 6.45.25 PM.png
>
>
> If multiple DAGs are defined in the same file and they share the same
> default_args, then the subsequent DAGs have an incorrect timezone.
>
> Steps to reproduce:
>
> Set the default_timezone to be non-UTC in airflow.cfg
>
> {noformat}
> default_timezone = America/New_York{noformat}
>
> DAG definition which has multiple DAGs in the same file:
>
>
> {code:java}
> from airflow import DAG
> from airflow.operators.bash_operator import BashOperator
> from datetime import datetime, timedelta
> default_args = {
> 'owner': 'airflow',
> 'depends_on_past': False,
> 'start_date': datetime(2019, 5, 11),
> }
> def make_dynamic_dag(schedule_interval, dag_name):
> dag = DAG(f"tutorial_{dag_name}", default_args=default_args,
> schedule_interval=schedule_interval)
> t1 = BashOperator(task_id='print_date', bash_command='date', dag=dag)
> return dag
> test_dag_1 = make_dynamic_dag("00 15 * * *", “1”)
> test_dag_2 = make_dynamic_dag("00 18 * * *", “2”)
> {code}
>
>
> test_dag_1 is expected to run at 15:00 EST or 19:00 UTC and test_dag_2 is
> expected to run at 18:00 EST or 22:00 UTC.
>
> However, test_dag_2 runs at 18:00 UTC which seems to point at it losing
> timezone information:
> !Screen Shot 2019-05-13 at 2.41.54 PM.png!
>
> I added some logging in the Airflow code around the default_args
> initialization and it confirmed the hypothesis that the default_args were
> being mutated:
>
> {noformat}
> [2019-05-13 18:40:10,409] {__init__.py:3045} INFO - default_args for DAG
> tutorial_1: {'owner': 'airflow', 'start_date': datetime.datetime(2019, 5, 11,
> 0, 0)}
> [2019-05-13 18:40:10,410] {__init__.py:3045} INFO - default_args for DAG
> tutorial_2: {'owner': 'airflow', 'start_date': [2019-05-11T04:00:00+00:00]>}
> {noformat}
>
>
> As a simple fix, I changed the DAG definition to:
> {noformat}
> dag = DAG(f"tutorial_{dag_name}", default_args=default_args,
> schedule_interval=schedule_interval){noformat}
> and this seems to fix the problem:
>
> {noformat}
> [2019-05-13 18:44:44,674] {__init__.py:3045} INFO - default_args for DAG
> tutorial_1: {'owner': 'airflow', 'start_date': datetime.datetime(2019, 5, 11,
> 0, 0)}
> [2019-05-13 18:44:44,676] {__init__.py:3045} INFO - default_args for DAG
> tutorial_2: {'owner': 'airflow', 'start_date': datetime.datetime(2019, 5, 11,
> 0, 0)}
> {noformat}
>
> !Screen Shot 2019-05-13 at 6.45.25 PM.png!
> I want to add a fix to create a deep-copy of default_args here:
> [https://github.com/apache/airflow/blob/master/airflow/models/dag.py#L197]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3517) Set expire_on_commit to True
[ https://issues.apache.org/jira/browse/AIRFLOW-3517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong updated AIRFLOW-3517: -- Description: Due to expire_on_commit=False the connections to the database are open much longer than needed, and therefore we have a lot of open connections to the database. (was: Due to expire_on_commit=True the connections to the database are open much longer than needed, and therefore we have a lot of open connections to the database.) > Set expire_on_commit to True > > > Key: AIRFLOW-3517 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3517 > Project: Apache Airflow > Issue Type: Task > Components: core >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 2.0.0 > > > Due to expire_on_commit=False the connections to the database are open much > longer than needed, and therefore we have a lot of open connections to the > database. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3517) Set expire_on_commit to True
[ https://issues.apache.org/jira/browse/AIRFLOW-3517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16867993#comment-16867993 ] Fokko Driesprong commented on AIRFLOW-3517: --- Great catch, we should set it to True. I haven't had the time to fix this. This is a huge ticket and we should do it in steps I guess. Currently, I don't have the time to move this forward, unfortunately. > Set expire_on_commit to True > > > Key: AIRFLOW-3517 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3517 > Project: Apache Airflow > Issue Type: Task > Components: core >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 2.0.0 > > > Due to expire_on_commit=True the connections to the database are open much > longer than needed, and therefore we have a lot of open connections to the > database. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3517) Set expire_on_commit to True
[ https://issues.apache.org/jira/browse/AIRFLOW-3517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong updated AIRFLOW-3517: -- Summary: Set expire_on_commit to True (was: Set expire_on_commit to False) > Set expire_on_commit to True > > > Key: AIRFLOW-3517 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3517 > Project: Apache Airflow > Issue Type: Task > Components: core >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 2.0.0 > > > Due to expire_on_commit=True the connections to the database are open much > longer than needed, and therefore we have a lot of open connections to the > database. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4781) Added the ability to specify ports in kubernetesOperator
[
https://issues.apache.org/jira/browse/AIRFLOW-4781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fokko Driesprong updated AIRFLOW-4781:
--
Fix Version/s: (was: 1.10.4)
> Added the ability to specify ports in kubernetesOperator
>
>
> Key: AIRFLOW-4781
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4781
> Project: Apache Airflow
> Issue Type: New Feature
> Components: operators
>Affects Versions: 1.10.3
>Reporter: Ron van Weverwijk
>Assignee: Ron van Weverwijk
>Priority: Minor
> Fix For: 2.0.0
>
>
> In kubernetes you have the ability to specify which ports to open to the
> container.
>
> {code:java}
> containers:
> - name: task-pv-container
> image: nginx
> ports:
> - containerPort: 80
> name: "http-server"
> {code}
> In this issue we want to add that support to the kubernetesOperator and
> `PodRequestFactory`
> With the support in the PodRequestFactory we can add functionality to build
> an operator that start a container in kubernetes and interact over the opened
> port.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4781) Added the ability to specify ports in kubernetesOperator
[
https://issues.apache.org/jira/browse/AIRFLOW-4781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fokko Driesprong resolved AIRFLOW-4781.
---
Resolution: Fixed
> Added the ability to specify ports in kubernetesOperator
>
>
> Key: AIRFLOW-4781
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4781
> Project: Apache Airflow
> Issue Type: New Feature
> Components: operators
>Affects Versions: 1.10.3
>Reporter: Ron van Weverwijk
>Assignee: Ron van Weverwijk
>Priority: Minor
> Fix For: 2.0.0
>
>
> In kubernetes you have the ability to specify which ports to open to the
> container.
>
> {code:java}
> containers:
> - name: task-pv-container
> image: nginx
> ports:
> - containerPort: 80
> name: "http-server"
> {code}
> In this issue we want to add that support to the kubernetesOperator and
> `PodRequestFactory`
> With the support in the PodRequestFactory we can add functionality to build
> an operator that start a container in kubernetes and interact over the opened
> port.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4423) Improve time handling in MySqlToGoogleCloudStorageOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-4423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong updated AIRFLOW-4423: -- Fix Version/s: (was: 1.10.4) > Improve time handling in MySqlToGoogleCloudStorageOperator > -- > > Key: AIRFLOW-4423 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4423 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.3 >Reporter: Josh Carp >Assignee: Josh Carp >Priority: Trivial > Labels: gcs, mysql > Fix For: 2.0.0 > > > * `MySqlToGoogleCloudStorageOperator` should handle `TIME` columns > (represented as `datetime.timedelta`) correctly > * `MySqlToGoogleCloudStorageOperator` should return `DATETIME` and > `TIMESTAMP` columns in UTC -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4423) Improve time handling in MySqlToGoogleCloudStorageOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-4423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-4423. --- Resolution: Fixed > Improve time handling in MySqlToGoogleCloudStorageOperator > -- > > Key: AIRFLOW-4423 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4423 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.3 >Reporter: Josh Carp >Assignee: Josh Carp >Priority: Trivial > Labels: gcs, mysql > Fix For: 1.10.4, 2.0.0 > > > * `MySqlToGoogleCloudStorageOperator` should handle `TIME` columns > (represented as `datetime.timedelta`) correctly > * `MySqlToGoogleCloudStorageOperator` should return `DATETIME` and > `TIMESTAMP` columns in UTC -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4423) Improve time handling in MySqlToGoogleCloudStorageOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-4423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong updated AIRFLOW-4423: -- Fix Version/s: 2.0.0 > Improve time handling in MySqlToGoogleCloudStorageOperator > -- > > Key: AIRFLOW-4423 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4423 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.3 >Reporter: Josh Carp >Assignee: Josh Carp >Priority: Trivial > Labels: gcs, mysql > Fix For: 1.10.4, 2.0.0 > > > * `MySqlToGoogleCloudStorageOperator` should handle `TIME` columns > (represented as `datetime.timedelta`) correctly > * `MySqlToGoogleCloudStorageOperator` should return `DATETIME` and > `TIMESTAMP` columns in UTC -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-4210) Update the CI image to Python3
[ https://issues.apache.org/jira/browse/AIRFLOW-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong closed AIRFLOW-4210. - Resolution: Won't Fix Good point, I think we can close this one, thanks [~zhongjiajie] > Update the CI image to Python3 > -- > > Key: AIRFLOW-4210 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4210 > Project: Apache Airflow > Issue Type: Sub-task > Components: core >Reporter: Fokko Driesprong >Assignee: zhongjiajie >Priority: Major > > Currently, we build from plain Debian, but we could now directly build from > Python3 image: https://github.com/apache/airflow-ci/blob/master/Dockerfile -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4586) Task getting stuck in Queued State
[ https://issues.apache.org/jira/browse/AIRFLOW-4586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16865675#comment-16865675 ] Fokko Driesprong commented on AIRFLOW-4586: --- Which executor are you running? The tasks in queue state are dispatched by the scheduler and this means that all dependencies are met. In this case, you should check if the tasks are being picked up by the workers. For example, check if they are being consumed from the queue. > Task getting stuck in Queued State > -- > > Key: AIRFLOW-4586 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4586 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Affects Versions: 1.10.0 >Reporter: raman >Assignee: raman >Priority: Major > > We are observing intermittently that Tasks get stuck in queued state and > never get executed by Airflow. On debugging it we found that one of the > queued dependency was not met due to which task did not move from queued to > running state. So task remained in queued state. (are_dependencies_met > function returned false for QUEUE_DEPS inside > _check_and_change_state_before_execution). > By looking into scheduler code it seems that scheduler does not reschedule > the queued state tasks due to which task never got added to executor queue > again and remained stuck in queued state. There is a logic inside > _check_and_change_state_before_execution function to move the task from > queued to None state(which gets picked by scheduler for rescheduling) if > RUN_DEPS are not met but this logic seems to be missing for QUEUE_DEPS. It > seems that task should be moved to None state even if QUEUE_DEPS are not met. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4777) Simplify the Python require in setup.py
[ https://issues.apache.org/jira/browse/AIRFLOW-4777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-4777. --- Resolution: Fixed > Simplify the Python require in setup.py > --- > > Key: AIRFLOW-4777 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4777 > Project: Apache Airflow > Issue Type: Improvement > Components: build >Affects Versions: 2.0.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-4781) Added the ability to specify ports in kubernetesOperator
[
https://issues.apache.org/jira/browse/AIRFLOW-4781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fokko Driesprong reassigned AIRFLOW-4781:
-
Assignee: Ron van Weverwijk
> Added the ability to specify ports in kubernetesOperator
>
>
> Key: AIRFLOW-4781
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4781
> Project: Apache Airflow
> Issue Type: New Feature
> Components: operators
>Affects Versions: 1.10.3
>Reporter: Ron van Weverwijk
>Assignee: Ron van Weverwijk
>Priority: Minor
> Fix For: 1.10.4, 2.0.0
>
>
> In kubernetes you have the ability to specify which ports to open to the
> container.
>
> {code:java}
> containers:
> - name: task-pv-container
> image: nginx
> ports:
> - containerPort: 80
> name: "http-server"
> {code}
> In this issue we want to add that support to the kubernetesOperator and
> `PodRequestFactory`
> With the support in the PodRequestFactory we can add functionality to build
> an operator that start a container in kubernetes and interact over the opened
> port.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-4777) Simplify the Python require in setup.py
Fokko Driesprong created AIRFLOW-4777: - Summary: Simplify the Python require in setup.py Key: AIRFLOW-4777 URL: https://issues.apache.org/jira/browse/AIRFLOW-4777 Project: Apache Airflow Issue Type: Improvement Components: build Affects Versions: 2.0.0 Reporter: Fokko Driesprong Assignee: Fokko Driesprong Fix For: 2.0.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (AIRFLOW-986) HiveCliHook silently ignores 'proxy_user' in connection's extra parameters
[ https://issues.apache.org/jira/browse/AIRFLOW-986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-986 started by Fokko Driesprong. > HiveCliHook silently ignores 'proxy_user' in connection's extra parameters > -- > > Key: AIRFLOW-986 > URL: https://issues.apache.org/jira/browse/AIRFLOW-986 > Project: Apache Airflow > Issue Type: Bug > Components: hooks >Affects Versions: 1.10.3 >Reporter: Yi Wei >Assignee: Fokko Driesprong >Priority: Minor > Labels: hive, hive-hooks > Fix For: 1.10.4, 2.0.0 > > > HiveCliHook just ignores the value of key 'proxy_user' in a connection's > extra parameters JSON object. There's two exceptions, if a user specify > 'proxy_user' to be 'login' or 'owner', HiveCliHook will append > 'hive.server2.proxy.user' to JDBC url, otherwise the proxy_user value is > always ignored. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-986) HiveCliHook silently ignores 'proxy_user' in connection's extra parameters
[ https://issues.apache.org/jira/browse/AIRFLOW-986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-986. -- Resolution: Fixed > HiveCliHook silently ignores 'proxy_user' in connection's extra parameters > -- > > Key: AIRFLOW-986 > URL: https://issues.apache.org/jira/browse/AIRFLOW-986 > Project: Apache Airflow > Issue Type: Bug > Components: hooks >Affects Versions: 1.10.3 >Reporter: Yi Wei >Assignee: omerjog >Priority: Minor > Labels: hive, hive-hooks > Fix For: 1.10.4, 2.0.0 > > > HiveCliHook just ignores the value of key 'proxy_user' in a connection's > extra parameters JSON object. There's two exceptions, if a user specify > 'proxy_user' to be 'login' or 'owner', HiveCliHook will append > 'hive.server2.proxy.user' to JDBC url, otherwise the proxy_user value is > always ignored. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-986) HiveCliHook silently ignores 'proxy_user' in connection's extra parameters
[ https://issues.apache.org/jira/browse/AIRFLOW-986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong reassigned AIRFLOW-986: Assignee: omerjog (was: Fokko Driesprong) > HiveCliHook silently ignores 'proxy_user' in connection's extra parameters > -- > > Key: AIRFLOW-986 > URL: https://issues.apache.org/jira/browse/AIRFLOW-986 > Project: Apache Airflow > Issue Type: Bug > Components: hooks >Affects Versions: 1.10.3 >Reporter: Yi Wei >Assignee: omerjog >Priority: Minor > Labels: hive, hive-hooks > Fix For: 1.10.4, 2.0.0 > > > HiveCliHook just ignores the value of key 'proxy_user' in a connection's > extra parameters JSON object. There's two exceptions, if a user specify > 'proxy_user' to be 'login' or 'owner', HiveCliHook will append > 'hive.server2.proxy.user' to JDBC url, otherwise the proxy_user value is > always ignored. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-986) HiveCliHook silently ignores 'proxy_user' in connection's extra parameters
[ https://issues.apache.org/jira/browse/AIRFLOW-986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong updated AIRFLOW-986: - Fix Version/s: 2.0.0 1.10.4 > HiveCliHook silently ignores 'proxy_user' in connection's extra parameters > -- > > Key: AIRFLOW-986 > URL: https://issues.apache.org/jira/browse/AIRFLOW-986 > Project: Apache Airflow > Issue Type: Bug > Components: hooks >Affects Versions: 1.10.3 >Reporter: Yi Wei >Assignee: Yi Wei >Priority: Minor > Labels: hive, hive-hooks > Fix For: 1.10.4, 2.0.0 > > > HiveCliHook just ignores the value of key 'proxy_user' in a connection's > extra parameters JSON object. There's two exceptions, if a user specify > 'proxy_user' to be 'login' or 'owner', HiveCliHook will append > 'hive.server2.proxy.user' to JDBC url, otherwise the proxy_user value is > always ignored. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-986) HiveCliHook silently ignores 'proxy_user' in connection's extra parameters
[ https://issues.apache.org/jira/browse/AIRFLOW-986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong updated AIRFLOW-986: - Affects Version/s: (was: 2.0.0) (was: 1.8.0) (was: 1.7.1.3) (was: 1.7.1.2) (was: 1.6.2) (was: 1.7.0) (was: 1.7.1) 1.10.3 > HiveCliHook silently ignores 'proxy_user' in connection's extra parameters > -- > > Key: AIRFLOW-986 > URL: https://issues.apache.org/jira/browse/AIRFLOW-986 > Project: Apache Airflow > Issue Type: Bug > Components: hooks >Affects Versions: 1.10.3 >Reporter: Yi Wei >Assignee: Yi Wei >Priority: Minor > Labels: hive, hive-hooks > > HiveCliHook just ignores the value of key 'proxy_user' in a connection's > extra parameters JSON object. There's two exceptions, if a user specify > 'proxy_user' to be 'login' or 'owner', HiveCliHook will append > 'hive.server2.proxy.user' to JDBC url, otherwise the proxy_user value is > always ignored. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4442) fix hive_tblproperties in HiveToDruidTransfer
[
https://issues.apache.org/jira/browse/AIRFLOW-4442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fokko Driesprong resolved AIRFLOW-4442.
---
Resolution: Fixed
> fix hive_tblproperties in HiveToDruidTransfer
> -
>
> Key: AIRFLOW-4442
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4442
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.10.3
>Reporter: Roster
>Assignee: Fokko Driesprong
>Priority: Minor
> Fix For: 1.10.4, 2.0.0
>
>
> The param is set to None by default which is wrong as later in the code it
> tries to do:
> {code:java}
> hive_tblproperties.items(){code}
> I will submit a PR for this
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4442) fix hive_tblproperties in HiveToDruidTransfer
[
https://issues.apache.org/jira/browse/AIRFLOW-4442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fokko Driesprong updated AIRFLOW-4442:
--
Affects Version/s: 1.10.3
> fix hive_tblproperties in HiveToDruidTransfer
> -
>
> Key: AIRFLOW-4442
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4442
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.10.3
>Reporter: Roster
>Priority: Minor
> Fix For: 1.10.4
>
>
> The param is set to None by default which is wrong as later in the code it
> tries to do:
> {code:java}
> hive_tblproperties.items(){code}
> I will submit a PR for this
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-4442) fix hive_tblproperties in HiveToDruidTransfer
[
https://issues.apache.org/jira/browse/AIRFLOW-4442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fokko Driesprong reassigned AIRFLOW-4442:
-
Assignee: Roster
> fix hive_tblproperties in HiveToDruidTransfer
> -
>
> Key: AIRFLOW-4442
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4442
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.10.3
>Reporter: Roster
>Assignee: Roster
>Priority: Minor
> Fix For: 1.10.4, 2.0.0
>
>
> The param is set to None by default which is wrong as later in the code it
> tries to do:
> {code:java}
> hive_tblproperties.items(){code}
> I will submit a PR for this
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (AIRFLOW-4442) fix hive_tblproperties in HiveToDruidTransfer
[
https://issues.apache.org/jira/browse/AIRFLOW-4442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on AIRFLOW-4442 started by Fokko Driesprong.
-
> fix hive_tblproperties in HiveToDruidTransfer
> -
>
> Key: AIRFLOW-4442
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4442
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.10.3
>Reporter: Roster
>Assignee: Fokko Driesprong
>Priority: Minor
> Fix For: 1.10.4, 2.0.0
>
>
> The param is set to None by default which is wrong as later in the code it
> tries to do:
> {code:java}
> hive_tblproperties.items(){code}
> I will submit a PR for this
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4442) fix hive_tblproperties in HiveToDruidTransfer
[
https://issues.apache.org/jira/browse/AIRFLOW-4442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fokko Driesprong updated AIRFLOW-4442:
--
Fix Version/s: 2.0.0
> fix hive_tblproperties in HiveToDruidTransfer
> -
>
> Key: AIRFLOW-4442
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4442
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.10.3
>Reporter: Roster
>Priority: Minor
> Fix For: 1.10.4, 2.0.0
>
>
> The param is set to None by default which is wrong as later in the code it
> tries to do:
> {code:java}
> hive_tblproperties.items(){code}
> I will submit a PR for this
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4370) Testing: document “best practices” on dealing with Apache Airflow DAGs and operators
[ https://issues.apache.org/jira/browse/AIRFLOW-4370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong updated AIRFLOW-4370: -- Description: h4. Expected deliverables * Introduction in testing workflows. * A page for Designing and Testing DAGs * Tips and working examples on “good practices” for designing DAGs * Descriptions on how to perform dry-runs of DAGs * Descriptions on how to write unit tests for DAGs * Snippets with working examples of DAGs and tests for them. Use [PlantUML|https://github.com/plantuml/plantuml] diagrams to compliment all the new documentation * How to develop operators that are testable? h4. Related resources [1] [https://github.com/jghoman/awesome-apache-airflow] [2] [https://airflow.apache.org/scheduler.html] [3] [https://github.com/PolideaInternal/airflow/blob/simplified-development-workflow/CONTRIBUTING.md] [4] [Airflow Breeze|https://github.com/PolideaInternal/airflow/blob/simplified-development-workflow/BREEZE.rst] [5] [https://github.com/godatadriven/whirl] was: h4. Expected deliverables * Introduction in testing workflows. * A page for Designing and Testing DAGs * Tips and working examples on “good practices” for designing DAGs * Descriptions on how to perform dry-runs of DAGs * Descriptions on how to write unit tests for DAGs * Snippets with working examples of DAGs and tests for them. Use [PlantUML|https://github.com/plantuml/plantuml] diagrams to compliment all the new documentation * How to develop operators that are testable? h4. Related resources [1] [https://github.com/jghoman/awesome-apache-airflow] [2] [https://airflow.apache.org/scheduler.html] [3] [https://github.com/PolideaInternal/airflow/blob/simplified-development-workflow/CONTRIBUTING.md] [4][Airflow Breeze|https://github.com/PolideaInternal/airflow/blob/simplified-development-workflow/BREEZE.rst] > Testing: document “best practices” on dealing with Apache Airflow DAGs and > operators > > > Key: AIRFLOW-4370 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4370 > Project: Apache Airflow > Issue Type: Improvement > Components: docs >Reporter: Aizhamal Nurmamat kyzy >Priority: Major > Labels: gsod2019 > > h4. Expected deliverables > * Introduction in testing workflows. > * A page for Designing and Testing DAGs > * Tips and working examples on “good practices” for designing DAGs > * Descriptions on how to perform dry-runs of DAGs > * Descriptions on how to write unit tests for DAGs > * Snippets with working examples of DAGs and tests for them. Use > [PlantUML|https://github.com/plantuml/plantuml] diagrams to compliment all > the new documentation > * How to develop operators that are testable? > h4. Related resources > [1] [https://github.com/jghoman/awesome-apache-airflow] > [2] [https://airflow.apache.org/scheduler.html] > [3] > [https://github.com/PolideaInternal/airflow/blob/simplified-development-workflow/CONTRIBUTING.md] > [4] [Airflow > Breeze|https://github.com/PolideaInternal/airflow/blob/simplified-development-workflow/BREEZE.rst] > [5] [https://github.com/godatadriven/whirl] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4092) Create gRPC Operator to make gRPC calls
[ https://issues.apache.org/jira/browse/AIRFLOW-4092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong updated AIRFLOW-4092: -- Fix Version/s: 2.0.0 > Create gRPC Operator to make gRPC calls > --- > > Key: AIRFLOW-4092 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4092 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Zhiwei Zhao >Assignee: Zhiwei Zhao >Priority: Minor > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4092) Create gRPC Operator to make gRPC calls
[ https://issues.apache.org/jira/browse/AIRFLOW-4092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-4092. --- Resolution: Fixed > Create gRPC Operator to make gRPC calls > --- > > Key: AIRFLOW-4092 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4092 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Zhiwei Zhao >Assignee: Zhiwei Zhao >Priority: Minor > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-3746) DockerOperator tasks in Airflow celery worker are stuck in "Running" state
[
https://issues.apache.org/jira/browse/AIRFLOW-3746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fokko Driesprong reassigned AIRFLOW-3746:
-
Assignee: Ashwini
> DockerOperator tasks in Airflow celery worker are stuck in "Running" state
> ---
>
> Key: AIRFLOW-3746
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3746
> Project: Apache Airflow
> Issue Type: Bug
> Components: celery
>Reporter: Ashwini
>Assignee: Ashwini
>Priority: Major
> Fix For: 2.0.0
>
>
> With the following DAG and task and using the celery executor, the task runs
> but never completes.
> *from* *airflow* *import* DAG
> *from* *airflow.operators.bash_operator* *import* BashOperator
> *from* *airflow.operators.docker_operator* *import* DockerOperator
> *from* *datetime* *import* datetime, timedelta
>
>
> default_args = {
> "owner": "airflow",
> "depends_on_past": False,
> "start_date": datetime(2018, 12, 31),
> "email": ["airf...@airflow.com"],
> "email_on_failure": False,
> "email_on_retry": False,
> "retries": 1,
> "retry_delay": timedelta(minutes=5),
> }
>
> dag = DAG("celery_test", default_args=default_args,
> schedule_interval=timedelta(1))
> DockerOperator(task_id ="test_docker", image =
> "gitlab-registry.nordstrom.com/merchprice/airflow:hello_python", retries=0,
> xcom_all=True , dag = dag)
>
> t2.set_upstream(t1)
>
> This is verison of airfow and celery and using
> python 3.6.
> apache-airflow 1.10.1
> celery 4.1.1
> docker 3.7.0
>
> --
> Here is the logs:
> *** Log file does not exist:
> /home/x9eu/airflow/logs/celery_test/test_docker/2019-01-16T00:00:00+00:00/1.log
> *** Fetching from:
> http://test.com:8793/log/celery_test/test_docker/2019-01-16T00:00:00+00:00/1.log
>
> [2019-01-21 20:49:26,260] \{models.py:1361} INFO - Dependencies all met for
>
> [2019-01-21 20:49:26,742] \{models.py:1361} INFO - Dependencies all met for
>
> [2019-01-21 20:49:26,742] \{models.py:1573} INFO -
>
> Starting attempt 1 of 1
>
>
> [2019-01-21 20:49:26,925] \{models.py:1595} INFO - Executing
> on 2019-01-16T00:00:00+00:00
> [2019-01-21 20:49:26,925] \{base_task_runner.py:118} INFO - Running: ['bash',
> '-c', 'airflow run celery_test test_docker 2019-01-16T00:00:00+00:00 --pickle
> 20 --job_id 59 --raw --cfg_path /tmp/tmps0u9a_e0']
> [2019-01-21 20:49:27,524] \{base_task_runner.py:101} INFO - Job 59: Subtask
> test_docker [2019-01-21 20:49:27,523] \{settings.py:174} INFO -
> setting.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800
> [2019-01-21 20:49:28,187] \{base_task_runner.py:101} INFO - Job 59: Subtask
> test_docker [2019-01-21 20:49:28,185] \{__init__.py:51} INFO - Using executor
> CeleryExecutor
> [2019-01-21 20:49:29,544] \{base_task_runner.py:101} INFO - Job 59: Subtask
> test_docker [2019-01-21 20:49:29,542] \{cli.py:470} INFO - Loading pickle id
> 20
> [2019-01-21 20:49:31,140] \{base_task_runner.py:101} INFO - Job 59: Subtask
> test_docker [2019-01-21 20:49:31,137] \{cli.py:484} INFO - Running
>
> on host test.com
> [2019-01-21 20:49:32,603] \{docker_operator.py:182} INFO - Starting docker
> container from image registry/airflow:hello_python
> [2019-01-21 20:49:48,770] \{docker_operator.py:228} INFO - Hello, %d 0
> Hello, %d 1
> Hello, %d 2
> Hello, %d 3
> Hello, %d 4
> Hello, %d 5
> Hello, %d 6
> Hello, %d 7
> Hello, %d 8
> Hello, %d 9
> Hello, %d 10
> Hello, %d 11
> Hello, %d 12
> Hello, %d 13
> Hello, %d 14
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3746) DockerOperator tasks in Airflow celery worker are stuck in "Running" state
[
https://issues.apache.org/jira/browse/AIRFLOW-3746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837183#comment-16837183
]
Fokko Driesprong commented on AIRFLOW-3746:
---
Targeted for Airflow 2.0 since it is not Python2 compatible.
> DockerOperator tasks in Airflow celery worker are stuck in "Running" state
> ---
>
> Key: AIRFLOW-3746
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3746
> Project: Apache Airflow
> Issue Type: Bug
> Components: celery
>Reporter: Ashwini
>Assignee: Ashwini
>Priority: Major
> Fix For: 2.0.0
>
>
> With the following DAG and task and using the celery executor, the task runs
> but never completes.
> *from* *airflow* *import* DAG
> *from* *airflow.operators.bash_operator* *import* BashOperator
> *from* *airflow.operators.docker_operator* *import* DockerOperator
> *from* *datetime* *import* datetime, timedelta
>
>
> default_args = {
> "owner": "airflow",
> "depends_on_past": False,
> "start_date": datetime(2018, 12, 31),
> "email": ["airf...@airflow.com"],
> "email_on_failure": False,
> "email_on_retry": False,
> "retries": 1,
> "retry_delay": timedelta(minutes=5),
> }
>
> dag = DAG("celery_test", default_args=default_args,
> schedule_interval=timedelta(1))
> DockerOperator(task_id ="test_docker", image =
> "gitlab-registry.nordstrom.com/merchprice/airflow:hello_python", retries=0,
> xcom_all=True , dag = dag)
>
> t2.set_upstream(t1)
>
> This is verison of airfow and celery and using
> python 3.6.
> apache-airflow 1.10.1
> celery 4.1.1
> docker 3.7.0
>
> --
> Here is the logs:
> *** Log file does not exist:
> /home/x9eu/airflow/logs/celery_test/test_docker/2019-01-16T00:00:00+00:00/1.log
> *** Fetching from:
> http://test.com:8793/log/celery_test/test_docker/2019-01-16T00:00:00+00:00/1.log
>
> [2019-01-21 20:49:26,260] \{models.py:1361} INFO - Dependencies all met for
>
> [2019-01-21 20:49:26,742] \{models.py:1361} INFO - Dependencies all met for
>
> [2019-01-21 20:49:26,742] \{models.py:1573} INFO -
>
> Starting attempt 1 of 1
>
>
> [2019-01-21 20:49:26,925] \{models.py:1595} INFO - Executing
> on 2019-01-16T00:00:00+00:00
> [2019-01-21 20:49:26,925] \{base_task_runner.py:118} INFO - Running: ['bash',
> '-c', 'airflow run celery_test test_docker 2019-01-16T00:00:00+00:00 --pickle
> 20 --job_id 59 --raw --cfg_path /tmp/tmps0u9a_e0']
> [2019-01-21 20:49:27,524] \{base_task_runner.py:101} INFO - Job 59: Subtask
> test_docker [2019-01-21 20:49:27,523] \{settings.py:174} INFO -
> setting.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800
> [2019-01-21 20:49:28,187] \{base_task_runner.py:101} INFO - Job 59: Subtask
> test_docker [2019-01-21 20:49:28,185] \{__init__.py:51} INFO - Using executor
> CeleryExecutor
> [2019-01-21 20:49:29,544] \{base_task_runner.py:101} INFO - Job 59: Subtask
> test_docker [2019-01-21 20:49:29,542] \{cli.py:470} INFO - Loading pickle id
> 20
> [2019-01-21 20:49:31,140] \{base_task_runner.py:101} INFO - Job 59: Subtask
> test_docker [2019-01-21 20:49:31,137] \{cli.py:484} INFO - Running
>
> on host test.com
> [2019-01-21 20:49:32,603] \{docker_operator.py:182} INFO - Starting docker
> container from image registry/airflow:hello_python
> [2019-01-21 20:49:48,770] \{docker_operator.py:228} INFO - Hello, %d 0
> Hello, %d 1
> Hello, %d 2
> Hello, %d 3
> Hello, %d 4
> Hello, %d 5
> Hello, %d 6
> Hello, %d 7
> Hello, %d 8
> Hello, %d 9
> Hello, %d 10
> Hello, %d 11
> Hello, %d 12
> Hello, %d 13
> Hello, %d 14
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3746) DockerOperator tasks in Airflow celery worker are stuck in "Running" state
[
https://issues.apache.org/jira/browse/AIRFLOW-3746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fokko Driesprong updated AIRFLOW-3746:
--
Fix Version/s: 2.0.0
> DockerOperator tasks in Airflow celery worker are stuck in "Running" state
> ---
>
> Key: AIRFLOW-3746
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3746
> Project: Apache Airflow
> Issue Type: Bug
> Components: celery
>Reporter: Ashwini
>Priority: Major
> Fix For: 2.0.0
>
>
> With the following DAG and task and using the celery executor, the task runs
> but never completes.
> *from* *airflow* *import* DAG
> *from* *airflow.operators.bash_operator* *import* BashOperator
> *from* *airflow.operators.docker_operator* *import* DockerOperator
> *from* *datetime* *import* datetime, timedelta
>
>
> default_args = {
> "owner": "airflow",
> "depends_on_past": False,
> "start_date": datetime(2018, 12, 31),
> "email": ["airf...@airflow.com"],
> "email_on_failure": False,
> "email_on_retry": False,
> "retries": 1,
> "retry_delay": timedelta(minutes=5),
> }
>
> dag = DAG("celery_test", default_args=default_args,
> schedule_interval=timedelta(1))
> DockerOperator(task_id ="test_docker", image =
> "gitlab-registry.nordstrom.com/merchprice/airflow:hello_python", retries=0,
> xcom_all=True , dag = dag)
>
> t2.set_upstream(t1)
>
> This is verison of airfow and celery and using
> python 3.6.
> apache-airflow 1.10.1
> celery 4.1.1
> docker 3.7.0
>
> --
> Here is the logs:
> *** Log file does not exist:
> /home/x9eu/airflow/logs/celery_test/test_docker/2019-01-16T00:00:00+00:00/1.log
> *** Fetching from:
> http://test.com:8793/log/celery_test/test_docker/2019-01-16T00:00:00+00:00/1.log
>
> [2019-01-21 20:49:26,260] \{models.py:1361} INFO - Dependencies all met for
>
> [2019-01-21 20:49:26,742] \{models.py:1361} INFO - Dependencies all met for
>
> [2019-01-21 20:49:26,742] \{models.py:1573} INFO -
>
> Starting attempt 1 of 1
>
>
> [2019-01-21 20:49:26,925] \{models.py:1595} INFO - Executing
> on 2019-01-16T00:00:00+00:00
> [2019-01-21 20:49:26,925] \{base_task_runner.py:118} INFO - Running: ['bash',
> '-c', 'airflow run celery_test test_docker 2019-01-16T00:00:00+00:00 --pickle
> 20 --job_id 59 --raw --cfg_path /tmp/tmps0u9a_e0']
> [2019-01-21 20:49:27,524] \{base_task_runner.py:101} INFO - Job 59: Subtask
> test_docker [2019-01-21 20:49:27,523] \{settings.py:174} INFO -
> setting.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800
> [2019-01-21 20:49:28,187] \{base_task_runner.py:101} INFO - Job 59: Subtask
> test_docker [2019-01-21 20:49:28,185] \{__init__.py:51} INFO - Using executor
> CeleryExecutor
> [2019-01-21 20:49:29,544] \{base_task_runner.py:101} INFO - Job 59: Subtask
> test_docker [2019-01-21 20:49:29,542] \{cli.py:470} INFO - Loading pickle id
> 20
> [2019-01-21 20:49:31,140] \{base_task_runner.py:101} INFO - Job 59: Subtask
> test_docker [2019-01-21 20:49:31,137] \{cli.py:484} INFO - Running
>
> on host test.com
> [2019-01-21 20:49:32,603] \{docker_operator.py:182} INFO - Starting docker
> container from image registry/airflow:hello_python
> [2019-01-21 20:49:48,770] \{docker_operator.py:228} INFO - Hello, %d 0
> Hello, %d 1
> Hello, %d 2
> Hello, %d 3
> Hello, %d 4
> Hello, %d 5
> Hello, %d 6
> Hello, %d 7
> Hello, %d 8
> Hello, %d 9
> Hello, %d 10
> Hello, %d 11
> Hello, %d 12
> Hello, %d 13
> Hello, %d 14
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3888) HA for metastore connection
[ https://issues.apache.org/jira/browse/AIRFLOW-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong updated AIRFLOW-3888: -- Fix Version/s: (was: 1.10.1) 2.0.0 > HA for metastore connection > --- > > Key: AIRFLOW-3888 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3888 > Project: Apache Airflow > Issue Type: Improvement > Components: hive_hooks >Affects Versions: 1.10.1 > Environment: Linux Centos 7 >Reporter: Jakub Guzik >Priority: Major > Labels: high-availability, hive, metastore, patch > Fix For: 2.0.0 > > Attachments: hive_hook.py > > Original Estimate: 24h > Remaining Estimate: 24h > > Creating a connection to a metasotor with two hosts for high avitablity (eg > connection 1, connection 2) is not possible because the entire value entered > is taken. For our needs, it is necessary to go through subsequent hosts and > connect to the first working. > This change allows you to check and then connect to a working metastor. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3888) HA for metastore connection
[ https://issues.apache.org/jira/browse/AIRFLOW-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong updated AIRFLOW-3888: -- Affects Version/s: (was: 1.10.1) 1.10.3 > HA for metastore connection > --- > > Key: AIRFLOW-3888 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3888 > Project: Apache Airflow > Issue Type: Improvement > Components: hive_hooks >Affects Versions: 1.10.3 > Environment: Linux Centos 7 >Reporter: Jakub Guzik >Priority: Major > Labels: high-availability, hive, metastore, patch > Fix For: 2.0.0 > > Attachments: hive_hook.py > > Original Estimate: 24h > Remaining Estimate: 24h > > Creating a connection to a metasotor with two hosts for high avitablity (eg > connection 1, connection 2) is not possible because the entire value entered > is taken. For our needs, it is necessary to go through subsequent hosts and > connect to the first working. > This change allows you to check and then connect to a working metastor. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3888) HA for metastore connection
[ https://issues.apache.org/jira/browse/AIRFLOW-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-3888. --- Resolution: Fixed > HA for metastore connection > --- > > Key: AIRFLOW-3888 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3888 > Project: Apache Airflow > Issue Type: Improvement > Components: hive_hooks >Affects Versions: 1.10.3 > Environment: Linux Centos 7 >Reporter: Jakub Guzik >Priority: Major > Labels: high-availability, hive, metastore, patch > Fix For: 2.0.0 > > Attachments: hive_hook.py > > Original Estimate: 24h > Remaining Estimate: 24h > > Creating a connection to a metasotor with two hosts for high avitablity (eg > connection 1, connection 2) is not possible because the entire value entered > is taken. For our needs, it is necessary to go through subsequent hosts and > connect to the first working. > This change allows you to check and then connect to a working metastor. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4471) Dataproc operator templated fields improvements
[ https://issues.apache.org/jira/browse/AIRFLOW-4471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-4471. --- Resolution: Fixed > Dataproc operator templated fields improvements > --- > > Key: AIRFLOW-4471 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4471 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.3 >Reporter: Martijn van de Grift >Assignee: Martijn van de Grift >Priority: Major > Fix For: 2.0.0 > > > This ticket is about: > * Fixing bugs in documentation. > * Adding 'dataproc_jars' as templated field for missing > DataProcXOperators > * Renaming 'dataproc__jars' to dataproc_jars to remove duplication. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4471) Dataproc operator templated fields improvements
[ https://issues.apache.org/jira/browse/AIRFLOW-4471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong updated AIRFLOW-4471: -- Fix Version/s: 2.0.0 > Dataproc operator templated fields improvements > --- > > Key: AIRFLOW-4471 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4471 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Martijn van de Grift >Assignee: Martijn van de Grift >Priority: Major > Fix For: 2.0.0 > > > This ticket is about: > * Fixing bugs in documentation. > * Adding 'dataproc_jars' as templated field for missing > DataProcXOperators > * Renaming 'dataproc__jars' to dataproc_jars to remove duplication. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4471) Dataproc operator templated fields improvements
[ https://issues.apache.org/jira/browse/AIRFLOW-4471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong updated AIRFLOW-4471: -- Component/s: gcp > Dataproc operator templated fields improvements > --- > > Key: AIRFLOW-4471 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4471 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.3 >Reporter: Martijn van de Grift >Assignee: Martijn van de Grift >Priority: Major > Fix For: 2.0.0 > > > This ticket is about: > * Fixing bugs in documentation. > * Adding 'dataproc_jars' as templated field for missing > DataProcXOperators > * Renaming 'dataproc__jars' to dataproc_jars to remove duplication. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4471) Dataproc operator templated fields improvements
[ https://issues.apache.org/jira/browse/AIRFLOW-4471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong updated AIRFLOW-4471: -- Affects Version/s: 1.10.3 > Dataproc operator templated fields improvements > --- > > Key: AIRFLOW-4471 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4471 > Project: Apache Airflow > Issue Type: Improvement >Affects Versions: 1.10.3 >Reporter: Martijn van de Grift >Assignee: Martijn van de Grift >Priority: Major > Fix For: 2.0.0 > > > This ticket is about: > * Fixing bugs in documentation. > * Adding 'dataproc_jars' as templated field for missing > DataProcXOperators > * Renaming 'dataproc__jars' to dataproc_jars to remove duplication. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4136) Fix overwrite of key_file by constructor in SSHHook
[
https://issues.apache.org/jira/browse/AIRFLOW-4136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fokko Driesprong updated AIRFLOW-4136:
--
Summary: Fix overwrite of key_file by constructor in SSHHook (was: SSHHook)
> Fix overwrite of key_file by constructor in SSHHook
> ---
>
> Key: AIRFLOW-4136
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4136
> Project: Apache Airflow
> Issue Type: Wish
> Components: hooks
>Affects Versions: 1.10.2
>Reporter: Bart Eijk
>Priority: Minor
> Fix For: 2.0.0
>
>
> When instantiating the SSHHook with a ssh_conn_id (with an empty
> 'extra-field) and key_file, the key_file argument is ignored because it is
> overwritten by None in the constructor of the SSHHook.
> Code that is causing this behaviour is in the __init__ method of SSHHook
> (added comments for description of what actually happens)
> {code}
> if conn.extra is not None: # conn.extra is an empty string in
> our case: ''
> extra_options = conn.extra_dejson # returns an empty dict
> self.key_file = extra_options.get("key_file") # sets
> self.key_file to None, overriding user-provided key_file
> {code}
> A fix could either be to check if conn.extra is not None or not '' or to wrap
> the setting of the key_file with:
> {code}
> if "key_file" in extra_options:
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4136) SSHHook
[
https://issues.apache.org/jira/browse/AIRFLOW-4136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fokko Driesprong resolved AIRFLOW-4136.
---
Resolution: Fixed
Fix Version/s: 2.0.0
> SSHHook
> ---
>
> Key: AIRFLOW-4136
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4136
> Project: Apache Airflow
> Issue Type: Wish
> Components: hooks
>Affects Versions: 1.10.2
>Reporter: Bart Eijk
>Priority: Minor
> Fix For: 2.0.0
>
>
> When instantiating the SSHHook with a ssh_conn_id (with an empty
> 'extra-field) and key_file, the key_file argument is ignored because it is
> overwritten by None in the constructor of the SSHHook.
> Code that is causing this behaviour is in the __init__ method of SSHHook
> (added comments for description of what actually happens)
> {code}
> if conn.extra is not None: # conn.extra is an empty string in
> our case: ''
> extra_options = conn.extra_dejson # returns an empty dict
> self.key_file = extra_options.get("key_file") # sets
> self.key_file to None, overriding user-provided key_file
> {code}
> A fix could either be to check if conn.extra is not None or not '' or to wrap
> the setting of the key_file with:
> {code}
> if "key_file" in extra_options:
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4201) Replace unicode strings by "normal" strings
[ https://issues.apache.org/jira/browse/AIRFLOW-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-4201. --- Resolution: Fixed > Replace unicode strings by "normal" strings > --- > > Key: AIRFLOW-4201 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4201 > Project: Apache Airflow > Issue Type: Sub-task >Reporter: Fokko Driesprong >Assignee: zhongjiajie >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4311) LocalExecutor sleeps unnecessary after executing tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-4311. --- Resolution: Fixed Fix Version/s: 2.0.0 1.10.4 > LocalExecutor sleeps unnecessary after executing tasks > -- > > Key: AIRFLOW-4311 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4311 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > Fix For: 1.10.4, 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-4311) LocalExecutor sleeps unnecessary after executing tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong reassigned AIRFLOW-4311: - Assignee: Bas Harenslak > LocalExecutor sleeps unnecessary after executing tasks > -- > > Key: AIRFLOW-4311 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4311 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3672) Add support for Mongo DB DNS Seedlist Connection Format
[ https://issues.apache.org/jira/browse/AIRFLOW-3672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-3672. --- Resolution: Fixed Fix Version/s: 1.10.4 > Add support for Mongo DB DNS Seedlist Connection Format > --- > > Key: AIRFLOW-3672 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3672 > Project: Apache Airflow > Issue Type: New Feature > Components: hooks >Affects Versions: 1.10.0 >Reporter: John Cheng >Assignee: John Cheng >Priority: Minor > Fix For: 1.10.4 > > > Starting with version 3.6, PyMongo supports mongodb+srv:// URIs. The URI must > include one, and only one, hostname. The hostname will be resolved to one or > more DNS SRV records which will be used as the seed list for connecting to > the MongoDB deployment. > https://docs.mongodb.com/manual/reference/connection-string/index.html#dns-seedlist-connection-format > http://api.mongodb.com/python/current/api/pymongo/mongo_client.html#pymongo.mongo_client.MongoClient > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4313) Remove the Mesos executor
Fokko Driesprong created AIRFLOW-4313: - Summary: Remove the Mesos executor Key: AIRFLOW-4313 URL: https://issues.apache.org/jira/browse/AIRFLOW-4313 Project: Apache Airflow Issue Type: Improvement Reporter: Fokko Driesprong -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4266) Add mypy to setup devel dependencies
[ https://issues.apache.org/jira/browse/AIRFLOW-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-4266. --- Resolution: Fixed Fix Version/s: 2.0.0 > Add mypy to setup devel dependencies > > > Key: AIRFLOW-4266 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4266 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3993) Add tests for SalesforceHook
[ https://issues.apache.org/jira/browse/AIRFLOW-3993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-3993. --- Resolution: Fixed Fix Version/s: 2.0.0 > Add tests for SalesforceHook > > > Key: AIRFLOW-3993 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3993 > Project: Apache Airflow > Issue Type: Test >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4293) "Down" Migration introduced in AIRFLOW-3584 doesn't apply
[ https://issues.apache.org/jira/browse/AIRFLOW-4293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-4293. --- Resolution: Fixed > "Down" Migration introduced in AIRFLOW-3584 doesn't apply > - > > Key: AIRFLOW-4293 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4293 > Project: Apache Airflow > Issue Type: Bug >Reporter: Ash Berlin-Taylor >Assignee: Fokko Driesprong >Priority: Minor > Fix For: 2.0.0, 1.10.4 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4293) "Down" Migration introduced in AIRFLOW-3584 doesn't apply
[ https://issues.apache.org/jira/browse/AIRFLOW-4293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong updated AIRFLOW-4293: -- Fix Version/s: 2.0.0 > "Down" Migration introduced in AIRFLOW-3584 doesn't apply > - > > Key: AIRFLOW-4293 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4293 > Project: Apache Airflow > Issue Type: Bug >Reporter: Ash Berlin-Taylor >Assignee: Ash Berlin-Taylor >Priority: Minor > Fix For: 2.0.0, 1.10.4 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
