[GitHub] ron819 commented on issue #2823: fix to inconsistent task instance state log message
ron819 commented on issue #2823: fix to inconsistent task instance state log message URL: https://github.com/apache/incubator-airflow/pull/2823#issuecomment-444776617 Is this still needed? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work stopped] (AIRFLOW-3071) Unable to clear Val of Variable from the UI
[
https://issues.apache.org/jira/browse/AIRFLOW-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on AIRFLOW-3071 stopped by jack.
-
> Unable to clear Val of Variable from the UI
> ---
>
> Key: AIRFLOW-3071
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3071
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.10.0
>Reporter: jack
>Priority: Minor
> Labels: easyfix
>
> This is quite annoying bug.
>
> Reproduce steps:
> # Create a Variable.
> # Give the Variable a Val & save it.
> # Click edit Variable. You will see the Key with Red {color:#FF}*{color}
> and the value that you entered.
> # Remove the Val (leave the field blank) and click save.
> # No errors will appear. However if you will re-enter to the Variable you
> will see that the Blank value was not saved.
>
> Please allow to remove Val. This is also the intend behavior because the Val
> has no {color:#FF}*{color} near it.
> The current work around is to delete the Variable and re-create it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-3071) Unable to clear Val of Variable from the UI
[
https://issues.apache.org/jira/browse/AIRFLOW-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jack reassigned AIRFLOW-3071:
-
Assignee: (was: jack)
> Unable to clear Val of Variable from the UI
> ---
>
> Key: AIRFLOW-3071
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3071
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.10.0
>Reporter: jack
>Priority: Minor
> Labels: easyfix
>
> This is quite annoying bug.
>
> Reproduce steps:
> # Create a Variable.
> # Give the Variable a Val & save it.
> # Click edit Variable. You will see the Key with Red {color:#FF}*{color}
> and the value that you entered.
> # Remove the Val (leave the field blank) and click save.
> # No errors will appear. However if you will re-enter to the Variable you
> will see that the Blank value was not saved.
>
> Please allow to remove Val. This is also the intend behavior because the Val
> has no {color:#FF}*{color} near it.
> The current work around is to delete the Variable and re-create it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-3454) Task duration isn't updated while task is running
jack created AIRFLOW-3454: - Summary: Task duration isn't updated while task is running Key: AIRFLOW-3454 URL: https://issues.apache.org/jira/browse/AIRFLOW-3454 Project: Apache Airflow Issue Type: Bug Affects Versions: 1.9.0 Reporter: jack Attachments: task.png When task is running you can hover it on the UI and get some details. The Duration value isn't updated. It shows the same value whether it ran 2 min or 30 min. Refresh does not help. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3433) Create Google Cloud Spanner Hook
[
https://issues.apache.org/jira/browse/AIRFLOW-3433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711003#comment-16711003
]
ASF GitHub Bot commented on AIRFLOW-3433:
-
ryanyuan opened a new pull request #4284: [AIRFLOW-3433] Create Google Cloud
Spanner Hook
URL: https://github.com/apache/incubator-airflow/pull/4284
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following
[Airflow-3433](https://issues.apache.org/jira/browse/AIRFLOW-3433) issues and
references them in the PR title.
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
Add a hook to connect to GCP Cloud Spanner and perform operations.
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
tests.contrib.hooks.test_gcp_spanner_hook
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create Google Cloud Spanner Hook
>
>
> Key: AIRFLOW-3433
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3433
> Project: Apache Airflow
> Issue Type: New Feature
>Reporter: Ryan Yuan
>Assignee: Ryan Yuan
>Priority: Major
>
> Add Google Cloud Spanner hook to Airflow
> (https://cloud.google.com/spanner/docs/apis)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] ryanyuan opened a new pull request #4284: [AIRFLOW-3433] Create Google Cloud Spanner Hook
ryanyuan opened a new pull request #4284: [AIRFLOW-3433] Create Google Cloud
Spanner Hook
URL: https://github.com/apache/incubator-airflow/pull/4284
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following
[Airflow-3433](https://issues.apache.org/jira/browse/AIRFLOW-3433) issues and
references them in the PR title.
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
Add a hook to connect to GCP Cloud Spanner and perform operations.
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
tests.contrib.hooks.test_gcp_spanner_hook
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] jmcarp commented on issue #4279: [AIRFLOW-3444] Explicitly set transfer operator description.
jmcarp commented on issue #4279: [AIRFLOW-3444] Explicitly set transfer operator description. URL: https://github.com/apache/incubator-airflow/pull/4279#issuecomment-444750767 Updated; wdyt @kaxil ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4279: [AIRFLOW-3444] Explicitly set transfer operator description.
codecov-io edited a comment on issue #4279: [AIRFLOW-3444] Explicitly set transfer operator description. URL: https://github.com/apache/incubator-airflow/pull/4279#issuecomment-444370208 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4279?src=pr=h1) Report > Merging [#4279](https://codecov.io/gh/apache/incubator-airflow/pull/4279?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/9dce1f0740f69af0ee86709a1a34a002b245aa3e?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4279?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#4279 +/- ## === Coverage 78.08% 78.08% === Files 201 201 Lines 1645816458 === Hits1285112851 Misses 3607 3607 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4279?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4279?src=pr=footer). Last update [9dce1f0...d29994c](https://codecov.io/gh/apache/incubator-airflow/pull/4279?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4279: [AIRFLOW-3444] Explicitly set transfer operator description.
codecov-io edited a comment on issue #4279: [AIRFLOW-3444] Explicitly set transfer operator description. URL: https://github.com/apache/incubator-airflow/pull/4279#issuecomment-444370208 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4279?src=pr=h1) Report > Merging [#4279](https://codecov.io/gh/apache/incubator-airflow/pull/4279?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/9dce1f0740f69af0ee86709a1a34a002b245aa3e?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4279?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#4279 +/- ## === Coverage 78.08% 78.08% === Files 201 201 Lines 1645816458 === Hits1285112851 Misses 3607 3607 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4279?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4279?src=pr=footer). Last update [9dce1f0...d29994c](https://codecov.io/gh/apache/incubator-airflow/pull/4279?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4279: [AIRFLOW-3444] Explicitly set transfer operator description.
codecov-io edited a comment on issue #4279: [AIRFLOW-3444] Explicitly set transfer operator description. URL: https://github.com/apache/incubator-airflow/pull/4279#issuecomment-444370208 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4279?src=pr=h1) Report > Merging [#4279](https://codecov.io/gh/apache/incubator-airflow/pull/4279?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/9dce1f0740f69af0ee86709a1a34a002b245aa3e?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4279?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#4279 +/- ## === Coverage 78.08% 78.08% === Files 201 201 Lines 1645816458 === Hits1285112851 Misses 3607 3607 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4279?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4279?src=pr=footer). Last update [9dce1f0...d29994c](https://codecov.io/gh/apache/incubator-airflow/pull/4279?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (AIRFLOW-3453) Failed to run airflow resetdb
[
https://issues.apache.org/jira/browse/AIRFLOW-3453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ping Zhang closed AIRFLOW-3453.
---
Resolution: Invalid
> Failed to run airflow resetdb
> -
>
> Key: AIRFLOW-3453
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3453
> Project: Apache Airflow
> Issue Type: Bug
> Components: database
>Affects Versions: 1.10.1
> Environment: macOS Mojave, 10.14.1;
>Reporter: Ping Zhang
>Assignee: Ping Zhang
>Priority: Major
>
> macOS Mojave, _10.14.1_;
> Mysql:
> _Your MySQL connection id is 6034_
> _Server version: 8.0.12 Homebrew_
>
> Right after I ran `airflow initdb`,
> {code:java}
>
>
> _$ airflow resetdb -y
> [2018-12-05 16:13:49,427] {settings.py:174} INFO - setting.configure_orm():
> Using pool settings. pool_size=5, pool_recycle=1800
> [2018-12-05 16:13:49,744] {_init_.py:51} INFO - Using executor CeleryExecutor
> DB: mysql://root:***@localhost/airflow
> [2018-12-05 16:13:49,973] {db.py:358} INFO - Dropping tables that exist
> /Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/engine/default.py:470:
> Warning: (3719L, "'utf8' is currently an alias for the character set
> UTF8MB3, but will be an alias for UTF8MB4 in a future release. Please
> consider using UTF8MB4 in order to be unambiguous.")
> cursor.execute(statement, parameters)
> Traceback (most recent call last):
> File "/Users/ping_zhang/anaconda2/envs/apache-airflow/bin/airflow", line 4,
> in
> _import_('pkg_resources').run_script('apache-airflow==1.10.1', 'airflow')
> File
> "/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/pkg_resources/_init_.py",
> line 664, in run_script
> self.require(requires)[0].run_script(script_name, ns)
> File
> "/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/pkg_resources/_init_.py",
> line 1444, in run_script
> exec(code, namespace, namespace)
> File
> "/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/apache_airflow-1.10.1-py2.7.egg/EGG-INFO/scripts/airflow",
> line 32, in
> args.func(args)
> File
> "/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/apache_airflow-1.10.1-py2.7.egg/airflow/bin/cli.py",
> line 1020, in resetdb
> db_utils.resetdb(settings.RBAC)
> File
> "/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/apache_airflow-1.10.1-py2.7.egg/airflow/utils/db.py",
> line 360, in resetdb
> models.Base.metadata.drop_all(settings.engine)
> File
> "/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/sql/schema.py",
> line 3976, in drop_all
> tables=tables)
> File
> "/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
> line 1929, in _run_visitor
> conn._run_visitor(visitorcallable, element, **kwargs)
> File
> "/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
> line 1538, in _run_visitor
> **kwargs).traverse_single(element)
> File
> "/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py",
> line 121, in traverse_single
> return meth(obj, **kw)
> File
> "/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py",
> line 874, in visit_metadata
> table, drop_ok=True, _is_metadata_operation=True)
> File
> "/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py",
> line 121, in traverse_single
> return meth(obj, **kw)
> File
> "/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py",
> line 923, in visit_table
> self.connection.execute(DropTable(table))
> File
> "/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
> line 945, in execute
> return meth(self, multiparams, params)
> File
> "/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py",
> line 68, in _execute_on_connection
> return connection._execute_ddl(self, multiparams, params)
> File
> "/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
> line 1002, in _execute_ddl
> compiled
> File
> "/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
> line 1189, in _execute_context
> context)
> File
> "/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
> line 1402, in _handle_dbapi_exception
> exc_info
> File
> "/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/util/compat.py",
> line 203, in raise_from_cause
>
[jira] [Assigned] (AIRFLOW-3453) Failed to run airflow resetdb
[
https://issues.apache.org/jira/browse/AIRFLOW-3453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ping Zhang reassigned AIRFLOW-3453:
---
Assignee: Ping Zhang
> Failed to run airflow resetdb
> -
>
> Key: AIRFLOW-3453
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3453
> Project: Apache Airflow
> Issue Type: Bug
> Components: database
>Affects Versions: 1.10.1
> Environment: macOS Mojave, 10.14.1;
>Reporter: Ping Zhang
>Assignee: Ping Zhang
>Priority: Major
>
> macOS Mojave, _10.14.1_;
> Mysql:
> _Your MySQL connection id is 6034_
> _Server version: 8.0.12 Homebrew_
>
> Right after I ran `airflow initdb`,
> {code:java}
>
>
> _$ airflow resetdb -y
> [2018-12-05 16:13:49,427] {settings.py:174} INFO - setting.configure_orm():
> Using pool settings. pool_size=5, pool_recycle=1800
> [2018-12-05 16:13:49,744] {_init_.py:51} INFO - Using executor CeleryExecutor
> DB: mysql://root:***@localhost/airflow
> [2018-12-05 16:13:49,973] {db.py:358} INFO - Dropping tables that exist
> /Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/engine/default.py:470:
> Warning: (3719L, "'utf8' is currently an alias for the character set
> UTF8MB3, but will be an alias for UTF8MB4 in a future release. Please
> consider using UTF8MB4 in order to be unambiguous.")
> cursor.execute(statement, parameters)
> Traceback (most recent call last):
> File "/Users/ping_zhang/anaconda2/envs/apache-airflow/bin/airflow", line 4,
> in
> _import_('pkg_resources').run_script('apache-airflow==1.10.1', 'airflow')
> File
> "/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/pkg_resources/_init_.py",
> line 664, in run_script
> self.require(requires)[0].run_script(script_name, ns)
> File
> "/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/pkg_resources/_init_.py",
> line 1444, in run_script
> exec(code, namespace, namespace)
> File
> "/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/apache_airflow-1.10.1-py2.7.egg/EGG-INFO/scripts/airflow",
> line 32, in
> args.func(args)
> File
> "/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/apache_airflow-1.10.1-py2.7.egg/airflow/bin/cli.py",
> line 1020, in resetdb
> db_utils.resetdb(settings.RBAC)
> File
> "/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/apache_airflow-1.10.1-py2.7.egg/airflow/utils/db.py",
> line 360, in resetdb
> models.Base.metadata.drop_all(settings.engine)
> File
> "/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/sql/schema.py",
> line 3976, in drop_all
> tables=tables)
> File
> "/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
> line 1929, in _run_visitor
> conn._run_visitor(visitorcallable, element, **kwargs)
> File
> "/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
> line 1538, in _run_visitor
> **kwargs).traverse_single(element)
> File
> "/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py",
> line 121, in traverse_single
> return meth(obj, **kw)
> File
> "/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py",
> line 874, in visit_metadata
> table, drop_ok=True, _is_metadata_operation=True)
> File
> "/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py",
> line 121, in traverse_single
> return meth(obj, **kw)
> File
> "/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py",
> line 923, in visit_table
> self.connection.execute(DropTable(table))
> File
> "/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
> line 945, in execute
> return meth(self, multiparams, params)
> File
> "/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py",
> line 68, in _execute_on_connection
> return connection._execute_ddl(self, multiparams, params)
> File
> "/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
> line 1002, in _execute_ddl
> compiled
> File
> "/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
> line 1189, in _execute_context
> context)
> File
> "/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
> line 1402, in _handle_dbapi_exception
> exc_info
> File
> "/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/util/compat.py",
> line 203, in
[jira] [Updated] (AIRFLOW-3453) Failed to run airflow resetdb
[
https://issues.apache.org/jira/browse/AIRFLOW-3453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ping Zhang updated AIRFLOW-3453:
Description:
macOS Mojave, _10.14.1_;
Mysql:
_Your MySQL connection id is 6034_
_Server version: 8.0.12 Homebrew_
Right after I ran `airflow initdb`,
{code:java}
_$ airflow resetdb -y
[2018-12-05 16:13:49,427] {settings.py:174} INFO - setting.configure_orm():
Using pool settings. pool_size=5, pool_recycle=1800
[2018-12-05 16:13:49,744] {_init_.py:51} INFO - Using executor CeleryExecutor
DB: mysql://root:***@localhost/airflow
[2018-12-05 16:13:49,973] {db.py:358} INFO - Dropping tables that exist
/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/engine/default.py:470:
Warning: (3719L, "'utf8' is currently an alias for the character set UTF8MB3,
but will be an alias for UTF8MB4 in a future release. Please consider using
UTF8MB4 in order to be unambiguous.")
cursor.execute(statement, parameters)
Traceback (most recent call last):
File "/Users/ping_zhang/anaconda2/envs/apache-airflow/bin/airflow", line 4, in
_import_('pkg_resources').run_script('apache-airflow==1.10.1', 'airflow')
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/pkg_resources/_init_.py",
line 664, in run_script
self.require(requires)[0].run_script(script_name, ns)
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/pkg_resources/_init_.py",
line 1444, in run_script
exec(code, namespace, namespace)
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/apache_airflow-1.10.1-py2.7.egg/EGG-INFO/scripts/airflow",
line 32, in
args.func(args)
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/apache_airflow-1.10.1-py2.7.egg/airflow/bin/cli.py",
line 1020, in resetdb
db_utils.resetdb(settings.RBAC)
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/apache_airflow-1.10.1-py2.7.egg/airflow/utils/db.py",
line 360, in resetdb
models.Base.metadata.drop_all(settings.engine)
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/sql/schema.py",
line 3976, in drop_all
tables=tables)
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 1929, in _run_visitor
conn._run_visitor(visitorcallable, element, **kwargs)
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 1538, in _run_visitor
**kwargs).traverse_single(element)
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py",
line 121, in traverse_single
return meth(obj, **kw)
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py",
line 874, in visit_metadata
table, drop_ok=True, _is_metadata_operation=True)
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py",
line 121, in traverse_single
return meth(obj, **kw)
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py",
line 923, in visit_table
self.connection.execute(DropTable(table))
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 945, in execute
return meth(self, multiparams, params)
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py",
line 68, in _execute_on_connection
return connection._execute_ddl(self, multiparams, params)
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 1002, in _execute_ddl
compiled
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 1189, in _execute_context
context)
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 1402, in _handle_dbapi_exception
exc_info
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/util/compat.py",
line 203, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 1182, in _execute_context
context)
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/engine/default.py",
line 470, in do_execute
cursor.execute(statement, parameters)
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/MySQLdb/cursors.py",
line 250, in execute
self.errorhandler(self, exc, value)
File
[jira] [Created] (AIRFLOW-3453) Failed to run airflow resetdb
Ping Zhang created AIRFLOW-3453:
---
Summary: Failed to run airflow resetdb
Key: AIRFLOW-3453
URL: https://issues.apache.org/jira/browse/AIRFLOW-3453
Project: Apache Airflow
Issue Type: Bug
Components: database
Affects Versions: 1.10.1
Environment: macOS Mojave, 10.14.1;
Reporter: Ping Zhang
macOS Mojave, _10.14.1_;
Mysql:
_Your MySQL connection id is 6034_
_Server version: 8.0.12 Homebrew_
Right after I ran `airflow initdb`,
_$ airflow resetdb -y
[2018-12-05 16:13:49,427] \{settings.py:174} INFO - setting.configure_orm():
Using pool settings. pool_size=5, pool_recycle=1800
[2018-12-05 16:13:49,744] \{__init__.py:51} INFO - Using executor CeleryExecutor
DB: mysql://root:***@localhost/airflow
[2018-12-05 16:13:49,973] \{db.py:358} INFO - Dropping tables that exist
/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/engine/default.py:470:
Warning: (3719L, "'utf8' is currently an alias for the character set UTF8MB3,
but will be an alias for UTF8MB4 in a future release. Please consider using
UTF8MB4 in order to be unambiguous.")
cursor.execute(statement, parameters)
Traceback (most recent call last):
File "/Users/ping_zhang/anaconda2/envs/apache-airflow/bin/airflow", line 4, in
__import__('pkg_resources').run_script('apache-airflow==1.10.1', 'airflow')
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/pkg_resources/__init__.py",
line 664, in run_script
self.require(requires)[0].run_script(script_name, ns)
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/pkg_resources/__init__.py",
line 1444, in run_script
exec(code, namespace, namespace)
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/apache_airflow-1.10.1-py2.7.egg/EGG-INFO/scripts/airflow",
line 32, in
args.func(args)
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/apache_airflow-1.10.1-py2.7.egg/airflow/bin/cli.py",
line 1020, in resetdb
db_utils.resetdb(settings.RBAC)
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/apache_airflow-1.10.1-py2.7.egg/airflow/utils/db.py",
line 360, in resetdb
models.Base.metadata.drop_all(settings.engine)
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/sql/schema.py",
line 3976, in drop_all
tables=tables)
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 1929, in _run_visitor
conn._run_visitor(visitorcallable, element, **kwargs)
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 1538, in _run_visitor
**kwargs).traverse_single(element)
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py",
line 121, in traverse_single
return meth(obj, **kw)
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py",
line 874, in visit_metadata
table, drop_ok=True, _is_metadata_operation=True)
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py",
line 121, in traverse_single
return meth(obj, **kw)
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py",
line 923, in visit_table
self.connection.execute(DropTable(table))
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 945, in execute
return meth(self, multiparams, params)
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py",
line 68, in _execute_on_connection
return connection._execute_ddl(self, multiparams, params)
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 1002, in _execute_ddl
compiled
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 1189, in _execute_context
context)
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 1402, in _handle_dbapi_exception
exc_info
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/util/compat.py",
line 203, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 1182, in _execute_context
context)
File
"/Users/ping_zhang/anaconda2/envs/apache-airflow/lib/python2.7/site-packages/sqlalchemy/engine/default.py",
line 470, in do_execute
cursor.execute(statement, parameters)
File
[jira] [Created] (AIRFLOW-3452) Cannot view dags at /home page
Jinhui Zhang created AIRFLOW-3452:
-
Summary: Cannot view dags at /home page
Key: AIRFLOW-3452
URL: https://issues.apache.org/jira/browse/AIRFLOW-3452
Project: Apache Airflow
Issue Type: Bug
Affects Versions: 2.0.0
Reporter: Jinhui Zhang
I checked out the latest master branch(commit
{{[9dce1f0|https://github.com/apache/incubator-airflow/commit/9dce1f0740f69af0ee86709a1a34a002b245aa3e]}})
and restarted my Airflow webserver. But I cannot view any dag at the home
page. I inspected the frontend code and found there's a
{{style="display:none;"}} on the \{{main-content}}, and the source code says so
at
[https://github.com/apache/incubator-airflow/blob/master/airflow/www_rbac/templates/airflow/dags.html#L31]
. Is this a known issue? How should I fix it?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] yangaws commented on a change in pull request #4278: [AIRFLOW-2524] Add SageMaker doc to AWS integration section
yangaws commented on a change in pull request #4278: [AIRFLOW-2524] Add SageMaker doc to AWS integration section URL: https://github.com/apache/incubator-airflow/pull/4278#discussion_r239273501 ## File path: docs/integration.rst ## @@ -398,6 +398,70 @@ AwsFirehoseHook .. autoclass:: airflow.contrib.hooks.aws_firehose_hook.AwsFirehoseHook +Amazon SageMaker + + +For more instructions on using Amazon SageMaker in Airflow, please see `here`_. Review comment: Good point! Changed 'here' to 'SageMaker Python SDK README' This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko edited a comment on issue #4283: [AIRFLOW-3450] Remove unnecessary sigint handler
Fokko edited a comment on issue #4283: [AIRFLOW-3450] Remove unnecessary sigint
handler
URL:
https://github.com/apache/incubator-airflow/pull/4283#issuecomment-444676536
I'm running Docker (Cloud composer), but I don't get any explicit reason:
```
[2018-12-05 20:20:57,138] {jobs.py:401} ERROR - Got an exception!
Propagating...
Traceback (most recent call last):
File "/usr/local/lib/airflow/airflow/jobs.py", line 376, in helper
settings.configure_orm()
File "/usr/local/lib/airflow/airflow/settings.py", line 180, in
configure_orm
engine = create_engine(SQL_ALCHEMY_CONN, **engine_args)
File
"/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/__init__.py", line
391, in create_engine
return strategy.create(*args, **kwargs)
File
"/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line
162, in create
engine = engineclass(pool, dialect, u, **engine_args)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py",
line 1772, in __init__
self.echo = echo
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/log.py", line 217,
in __set__
instance_logger(instance, echoflag=value)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/log.py", line 173,
in instance_logger
def instance_logger(instance, echoflag=None):
File "/usr/local/lib/airflow/airflow/bin/cli.py", line 85, in
sigint_handler
sys.exit(0)
```
So my initial thought was to just let the exception propagate. Also the K8s
tests are still running fine :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-3451) Add Refresh All DAGs button to Web UI
Ian Davison created AIRFLOW-3451: Summary: Add Refresh All DAGs button to Web UI Key: AIRFLOW-3451 URL: https://issues.apache.org/jira/browse/AIRFLOW-3451 Project: Apache Airflow Issue Type: New Feature Components: webapp Affects Versions: 1.10.1 Reporter: Ian Davison Assignee: Ian Davison Fix For: 1.10.2 There is a refresh_all endpoint exposed on the webserver but no button in the UI that calls it, propose adding a button to the web UI's main page that allows a user to refresh all DAGs -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] Fokko commented on issue #4283: [AIRFLOW-3450] Remove unnecessary sigint handler
Fokko commented on issue #4283: [AIRFLOW-3450] Remove unnecessary sigint handler
URL:
https://github.com/apache/incubator-airflow/pull/4283#issuecomment-444676536
I'm running Docker (Cloud composer), but I don't get any explicit reason:
```
[2018-12-05 20:20:57,138] {jobs.py:401} ERROR - Got an exception!
Propagating...
Traceback (most recent call last):
File "/usr/local/lib/airflow/airflow/jobs.py", line 376, in helper
settings.configure_orm()
File "/usr/local/lib/airflow/airflow/settings.py", line 180, in
configure_orm
engine = create_engine(SQL_ALCHEMY_CONN, **engine_args)
File
"/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/__init__.py", line
391, in create_engine
return strategy.create(*args, **kwargs)
File
"/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line
162, in create
engine = engineclass(pool, dialect, u, **engine_args)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py",
line 1772, in __init__
self.echo = echo
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/log.py", line 217,
in __set__
instance_logger(instance, echoflag=value)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/log.py", line 173,
in instance_logger
def instance_logger(instance, echoflag=None):
File "/usr/local/lib/airflow/airflow/bin/cli.py", line 85, in
sigint_handler
sys.exit(0)
```
So my initial thought was to just let the exception propagate.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (AIRFLOW-3451) Add Refresh All DAGs button to Web UI
[ https://issues.apache.org/jira/browse/AIRFLOW-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Davison updated AIRFLOW-3451: - Description: There is a refresh_all endpoint exposed on the webserver, but no button in the UI that calls it, propose adding a button to the web UI's main page that allows a user to refresh all DAGs (was: There is a refresh_all endpoint exposed on the webserver but no button in the UI that calls it, propose adding a button to the web UI's main page that allows a user to refresh all DAGs ) > Add Refresh All DAGs button to Web UI > - > > Key: AIRFLOW-3451 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3451 > Project: Apache Airflow > Issue Type: New Feature > Components: webapp >Affects Versions: 1.10.1 >Reporter: Ian Davison >Assignee: Ian Davison >Priority: Minor > Fix For: 1.10.2 > > > There is a refresh_all endpoint exposed on the webserver, but no button in > the UI that calls it, propose adding a button to the web UI's main page that > allows a user to refresh all DAGs -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] codecov-io edited a comment on issue #4283: [AIRFLOW-3450] Remove unnecessary sigint handler
codecov-io edited a comment on issue #4283: [AIRFLOW-3450] Remove unnecessary sigint handler URL: https://github.com/apache/incubator-airflow/pull/4283#issuecomment-444666762 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4283?src=pr=h1) Report > Merging [#4283](https://codecov.io/gh/apache/incubator-airflow/pull/4283?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/9dce1f0740f69af0ee86709a1a34a002b245aa3e?src=pr=desc) will **increase** coverage by `0.02%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4283?src=pr=tree) ```diff @@Coverage Diff@@ ## master #4283 +/- ## = + Coverage 78.08% 78.1% +0.02% = Files 201 201 Lines 16458 16450 -8 = - Hits12851 12849 -2 + Misses 36073601 -6 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4283?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/bin/cli.py](https://codecov.io/gh/apache/incubator-airflow/pull/4283/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `65.09% <ø> (+0.5%)` | :arrow_up: | | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4283/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `92.29% <0%> (-0.05%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4283?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4283?src=pr=footer). Last update [9dce1f0...66f2e46](https://codecov.io/gh/apache/incubator-airflow/pull/4283?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #4283: [AIRFLOW-3450] Remove unnecessary sigint handler
codecov-io commented on issue #4283: [AIRFLOW-3450] Remove unnecessary sigint handler URL: https://github.com/apache/incubator-airflow/pull/4283#issuecomment-444666762 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4283?src=pr=h1) Report > Merging [#4283](https://codecov.io/gh/apache/incubator-airflow/pull/4283?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/9dce1f0740f69af0ee86709a1a34a002b245aa3e?src=pr=desc) will **increase** coverage by `0.02%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4283?src=pr=tree) ```diff @@Coverage Diff@@ ## master #4283 +/- ## = + Coverage 78.08% 78.1% +0.02% = Files 201 201 Lines 16458 16450 -8 = - Hits12851 12849 -2 + Misses 36073601 -6 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4283?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/bin/cli.py](https://codecov.io/gh/apache/incubator-airflow/pull/4283/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `65.09% <ø> (+0.5%)` | :arrow_up: | | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4283/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `92.29% <0%> (-0.05%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4283?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4283?src=pr=footer). Last update [9dce1f0...66f2e46](https://codecov.io/gh/apache/incubator-airflow/pull/4283?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #4258: [AIRFLOW-3423] Fix unauthenticated access for mongo hook
codecov-io commented on issue #4258: [AIRFLOW-3423] Fix unauthenticated access for mongo hook URL: https://github.com/apache/incubator-airflow/pull/4258#issuecomment-444660912 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4258?src=pr=h1) Report > Merging [#4258](https://codecov.io/gh/apache/incubator-airflow/pull/4258?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/d1d612e5c78c89487a05ab3cc86f7d3472ac2a5d?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4258?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4258 +/- ## == + Coverage 78.07% 78.08% +<.01% == Files 201 201 Lines 1645516458 +3 == + Hits1284812851 +3 Misses 3607 3607 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4258?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/logging\_config.py](https://codecov.io/gh/apache/incubator-airflow/pull/4258/diff?src=pr=tree#diff-YWlyZmxvdy9sb2dnaW5nX2NvbmZpZy5weQ==) | `97.56% <0%> (+0.06%)` | :arrow_up: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/incubator-airflow/pull/4258/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `59.67% <0%> (+0.14%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4258?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4258?src=pr=footer). Last update [d1d612e...da10ae8](https://codecov.io/gh/apache/incubator-airflow/pull/4258?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on issue #4283: [AIRFLOW-3450] Remove unnecessary sigint handler
Fokko commented on issue #4283: [AIRFLOW-3450] Remove unnecessary sigint handler URL: https://github.com/apache/incubator-airflow/pull/4283#issuecomment-444659184 @bolkedebruin Any idea why this empty sighandle is there? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] tmiller-msft commented on issue #4265: [AIRFLOW-3406] Implement an Azure CosmosDB operator
tmiller-msft commented on issue #4265: [AIRFLOW-3406] Implement an Azure CosmosDB operator URL: https://github.com/apache/incubator-airflow/pull/4265#issuecomment-444656081 Thanks @feng-tao, glad to have helped. Lmk if you need anything more. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4265: [AIRFLOW-3406] Implement an Azure CosmosDB operator
codecov-io edited a comment on issue #4265: [AIRFLOW-3406] Implement an Azure CosmosDB operator URL: https://github.com/apache/incubator-airflow/pull/4265#issuecomment-444328155 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4265?src=pr=h1) Report > Merging [#4265](https://codecov.io/gh/apache/incubator-airflow/pull/4265?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/d4f545fbe3cff93c61ed9757e470c8044ed976e1?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `75%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4265?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4265 +/- ## == - Coverage 78.08% 78.08% -0.01% == Files 201 201 Lines 1645816462 +4 == + Hits1285112854 +3 - Misses 3607 3608 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4265?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/db.py](https://codecov.io/gh/apache/incubator-airflow/pull/4265/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `33.33% <0%> (-0.27%)` | :arrow_down: | | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4265/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `92.34% <100%> (ø)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4265?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4265?src=pr=footer). Last update [d4f545f...a7d240d](https://codecov.io/gh/apache/incubator-airflow/pull/4265?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4265: [AIRFLOW-3406] Implement an Azure CosmosDB operator
codecov-io edited a comment on issue #4265: [AIRFLOW-3406] Implement an Azure CosmosDB operator URL: https://github.com/apache/incubator-airflow/pull/4265#issuecomment-444328155 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4265?src=pr=h1) Report > Merging [#4265](https://codecov.io/gh/apache/incubator-airflow/pull/4265?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/d4f545fbe3cff93c61ed9757e470c8044ed976e1?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `75%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4265?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4265 +/- ## == - Coverage 78.08% 78.08% -0.01% == Files 201 201 Lines 1645816462 +4 == + Hits1285112854 +3 - Misses 3607 3608 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4265?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/db.py](https://codecov.io/gh/apache/incubator-airflow/pull/4265/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `33.33% <0%> (-0.27%)` | :arrow_down: | | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4265/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `92.34% <100%> (ø)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4265?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4265?src=pr=footer). Last update [d4f545f...a7d240d](https://codecov.io/gh/apache/incubator-airflow/pull/4265?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-3450) Remove unnecessary signal handlers
Fokko Driesprong created AIRFLOW-3450: - Summary: Remove unnecessary signal handlers Key: AIRFLOW-3450 URL: https://issues.apache.org/jira/browse/AIRFLOW-3450 Project: Apache Airflow Issue Type: Task Components: cli Reporter: Fokko Driesprong Assignee: Fokko Driesprong -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3450) Remove unnecessary signal handlers
[
https://issues.apache.org/jira/browse/AIRFLOW-3450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710627#comment-16710627
]
ASF GitHub Bot commented on AIRFLOW-3450:
-
Fokko opened a new pull request #4283: [AIRFLOW-3450] Remove unnecessary sigint
handler
URL: https://github.com/apache/incubator-airflow/pull/4283
This obfuscate the actual exit signal.
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-3450\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3450
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-3450\], code changes always need a Jira issue.
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
The signal handler obfuscates the actual exit signal.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Remove unnecessary signal handlers
> ---
>
> Key: AIRFLOW-3450
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3450
> Project: Apache Airflow
> Issue Type: Task
> Components: cli
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] Fokko opened a new pull request #4283: [AIRFLOW-3450] Remove unnecessary sigint handler
Fokko opened a new pull request #4283: [AIRFLOW-3450] Remove unnecessary sigint
handler
URL: https://github.com/apache/incubator-airflow/pull/4283
This obfuscate the actual exit signal.
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-3450\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3450
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-3450\], code changes always need a Jira issue.
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
The signal handler obfuscates the actual exit signal.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] jmcarp commented on a change in pull request #4279: [AIRFLOW-3444] Explicitly set transfer operator description.
jmcarp commented on a change in pull request #4279: [AIRFLOW-3444] Explicitly set transfer operator description. URL: https://github.com/apache/incubator-airflow/pull/4279#discussion_r239233508 ## File path: airflow/contrib/operators/s3_to_gcs_transfer_operator.py ## @@ -45,15 +45,14 @@ class S3ToGoogleCloudStorageTransferOperator(BaseOperator): For this to work, the service account making the request must have domain-wide delegation enabled. :type delegate_to: str +:param description: Optional transfer service job description +:type description: str :param object_conditions: Transfer service object conditions; see https://cloud.google.com/storage-transfer/docs/reference/rest/v1/TransferSpec :type object_conditions: dict :param transfer_options: Transfer service transfer options; see https://cloud.google.com/storage-transfer/docs/reference/rest/v1/TransferSpec :type transfer_options: dict -:param job_kwargs: Additional transfer job options; see Review comment: The `GCPTransferServiceHook` already sets the schedule so that the job runs right away and only once. Then the hook waits for the job to complete. If we want to allow custom schedules, I think we'd want to make waiting optional--it wouldn't make sense to schedule a job a day in the future, then wait for it to finish. I'll revise and add those options later today. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-3447) Intended usage of ts_nodash macro broken with migration to new time system.
[
https://issues.apache.org/jira/browse/AIRFLOW-3447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik updated AIRFLOW-3447:
Fix Version/s: 1.10.2
> Intended usage of ts_nodash macro broken with migration to new time system.
> ---
>
> Key: AIRFLOW-3447
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3447
> Project: Apache Airflow
> Issue Type: Bug
> Components: core
>Reporter: Luka Draksler
>Priority: Minor
> Labels: easyfix
> Fix For: 1.10.2
>
>
> Migration to timezone aware times broke the intended usage of ts_nodash macro.
> ts_nodash is used in certain placeholders to create different names (table
> names, cluster names...). As such it is alphanumeric only, it contains no
> characters that could be deemed illegal by various naming restrictions.
> Migration to new time system changed that.
> As an example, this would be returned currently:
> {{20181205T125657.169324+}}
> {{before:}}
> {{20181204T03}}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] feng-tao commented on issue #4265: [AIRFLOW-3406] Implement an Azure CosmosDB operator
feng-tao commented on issue #4265: [AIRFLOW-3406] Implement an Azure CosmosDB operator URL: https://github.com/apache/incubator-airflow/pull/4265#issuecomment-444642729 lgtm, thanks for the contribution @tmiller-msft . Will commit once CI is pass. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3408) Systemd setup instructions mention deprecated variable
[
https://issues.apache.org/jira/browse/AIRFLOW-3408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710615#comment-16710615
]
ASF GitHub Bot commented on AIRFLOW-3408:
-
feng-tao closed pull request #4269: [AIRFLOW-3408] Remove outdated info from
Systemd Instructions
URL: https://github.com/apache/incubator-airflow/pull/4269
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/docs/howto/run-with-systemd.rst b/docs/howto/run-with-systemd.rst
index 131fc3ddc0..bbd2235ae4 100644
--- a/docs/howto/run-with-systemd.rst
+++ b/docs/howto/run-with-systemd.rst
@@ -27,6 +27,5 @@ have been tested on Redhat based systems. You can copy those
to
based system) you probably need to adjust the unit files.
Environment configuration is picked up from ``/etc/sysconfig/airflow``.
-An example file is supplied. Make sure to specify the ``SCHEDULER_RUNS``
-variable in this file when you run the scheduler. You
+An example file is supplied. You
can also define here, for example, ``AIRFLOW_HOME`` or ``AIRFLOW_CONFIG``.
diff --git a/scripts/systemd/README b/scripts/systemd/README
index c53144a04c..e32df5f52f 100644
--- a/scripts/systemd/README
+++ b/scripts/systemd/README
@@ -6,6 +6,6 @@ You can then start the different servers by using systemctl
start . Ena
systemctl enable .
By default the environment configuration points to /etc/sysconfig/airflow .
You can copy the "airflow" file in this
-directory and adjust it to your liking. Make sure to specify the
SCHEDULER_RUNS variable.
+directory and adjust it to your liking.
-With some minor changes they probably work on other systemd systems.
\ No newline at end of file
+With some minor changes they probably work on other systemd systems.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Systemd setup instructions mention deprecated variable
> --
>
> Key: AIRFLOW-3408
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3408
> Project: Apache Airflow
> Issue Type: Improvement
> Components: Documentation
>Reporter: Victor Villas Bôas Chaves
>Assignee: Kaxil Naik
>Priority: Minor
> Fix For: 1.10.2
>
>
> AIRFLOW-1698 was solved in code, but the documentation drifted.
> Places where {{SCHEDULER_RUNS}} is still mentioned as necessary adjustment:
> [https://github.com/apache/incubator-airflow/blob/53b89b98371c7bb993b242c341d3941e9ce09f9a/scripts/systemd/README]
> [https://github.com/apache/incubator-airflow/blob/b9fc03ea1ad5cea3c3aa668fcaca103f84167b9c/docs/howto/run-with-systemd.rst]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] feng-tao commented on issue #4269: [AIRFLOW-3408] Remove outdated info from Systemd Instructions
feng-tao commented on issue #4269: [AIRFLOW-3408] Remove outdated info from Systemd Instructions URL: https://github.com/apache/incubator-airflow/pull/4269#issuecomment-444642159 lgtm This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] feng-tao closed pull request #4269: [AIRFLOW-3408] Remove outdated info from Systemd Instructions
feng-tao closed pull request #4269: [AIRFLOW-3408] Remove outdated info from Systemd Instructions URL: https://github.com/apache/incubator-airflow/pull/4269 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/docs/howto/run-with-systemd.rst b/docs/howto/run-with-systemd.rst index 131fc3ddc0..bbd2235ae4 100644 --- a/docs/howto/run-with-systemd.rst +++ b/docs/howto/run-with-systemd.rst @@ -27,6 +27,5 @@ have been tested on Redhat based systems. You can copy those to based system) you probably need to adjust the unit files. Environment configuration is picked up from ``/etc/sysconfig/airflow``. -An example file is supplied. Make sure to specify the ``SCHEDULER_RUNS`` -variable in this file when you run the scheduler. You +An example file is supplied. You can also define here, for example, ``AIRFLOW_HOME`` or ``AIRFLOW_CONFIG``. diff --git a/scripts/systemd/README b/scripts/systemd/README index c53144a04c..e32df5f52f 100644 --- a/scripts/systemd/README +++ b/scripts/systemd/README @@ -6,6 +6,6 @@ You can then start the different servers by using systemctl start . Ena systemctl enable . By default the environment configuration points to /etc/sysconfig/airflow . You can copy the "airflow" file in this -directory and adjust it to your liking. Make sure to specify the SCHEDULER_RUNS variable. +directory and adjust it to your liking. -With some minor changes they probably work on other systemd systems. \ No newline at end of file +With some minor changes they probably work on other systemd systems. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3447) Intended usage of ts_nodash macro broken with migration to new time system.
[
https://issues.apache.org/jira/browse/AIRFLOW-3447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710606#comment-16710606
]
Kaxil Naik commented on AIRFLOW-3447:
-
[~ashb] Timezone is giving few headaches isn't it. For 1.10.2 can we just
resolve this with this solution given above? If there are many issues.
> Intended usage of ts_nodash macro broken with migration to new time system.
> ---
>
> Key: AIRFLOW-3447
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3447
> Project: Apache Airflow
> Issue Type: Bug
> Components: core
>Reporter: Luka Draksler
>Priority: Minor
> Labels: easyfix
> Fix For: 1.10.2
>
>
> Migration to timezone aware times broke the intended usage of ts_nodash macro.
> ts_nodash is used in certain placeholders to create different names (table
> names, cluster names...). As such it is alphanumeric only, it contains no
> characters that could be deemed illegal by various naming restrictions.
> Migration to new time system changed that.
> As an example, this would be returned currently:
> {{20181205T125657.169324+}}
> {{before:}}
> {{20181204T03}}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] kaxil closed pull request #4273: [AIRFLOW-XXX] GCP operators documentation clarifications
kaxil closed pull request #4273: [AIRFLOW-XXX] GCP operators documentation clarifications URL: https://github.com/apache/incubator-airflow/pull/4273 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/airflow/contrib/hooks/gcp_sql_hook.py b/airflow/contrib/hooks/gcp_sql_hook.py index 43e664d15c..3797680cbc 100644 --- a/airflow/contrib/hooks/gcp_sql_hook.py +++ b/airflow/contrib/hooks/gcp_sql_hook.py @@ -110,7 +110,7 @@ def create_instance(self, project_id, body): :param body: Body required by the Cloud SQL insert API, as described in https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/instances/insert#request-body. :type body: dict -:return: True if the operation succeeded, raises an error otherwise. +:return: True if the operation succeeded otherwise raises an error. :rtype: bool """ response = self.get_conn().instances().insert( @@ -134,7 +134,7 @@ def patch_instance(self, project_id, body, instance): :type body: dict :param instance: Cloud SQL instance ID. This does not include the project ID. :type instance: str -:return: True if the operation succeeded, raises an error otherwise. +:return: True if the operation succeeded otherwise raises an error. :rtype: bool """ response = self.get_conn().instances().patch( @@ -153,7 +153,7 @@ def delete_instance(self, project_id, instance): :type project_id: str :param instance: Cloud SQL instance ID. This does not include the project ID. :type instance: str -:return: True if the operation succeeded, raises an error otherwise +:return: True if the operation succeeded otherwise raises an error. :rtype: bool """ response = self.get_conn().instances().delete( @@ -194,7 +194,7 @@ def create_database(self, project, instance, body): :param body: The request body, as described in https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/databases/insert#request-body. :type body: dict -:return: True if the operation succeeded, raises an error otherwise. +:return: True if the operation succeeded otherwise raises an error. :rtype: bool """ response = self.get_conn().databases().insert( @@ -210,7 +210,7 @@ def patch_database(self, project, instance, database, body): Updates a database resource inside a Cloud SQL instance. This method supports patch semantics. -See: https://cloud.google.com/sql/docs/mysql/admin-api/how-tos/performance#patch +See https://cloud.google.com/sql/docs/mysql/admin-api/how-tos/performance#patch. :param project: Project ID of the project that contains the instance. :type project: str @@ -221,7 +221,7 @@ def patch_database(self, project, instance, database, body): :param body: The request body, as described in https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/databases/insert#request-body. :type body: dict -:return: True if the operation succeeded, raises an error otherwise. +:return: True if the operation succeeded otherwise raises an error. :rtype: bool """ response = self.get_conn().databases().patch( @@ -243,7 +243,7 @@ def delete_database(self, project, instance, database): :type instance: str :param database: Name of the database to be deleted in the instance. :type database: str -:return: True if the operation succeeded, raises an error otherwise. +:return: True if the operation succeeded otherwise raises an error. :rtype: bool """ response = self.get_conn().databases().delete( @@ -293,11 +293,11 @@ def _wait_for_operation_to_complete(self, project_id, operation_name): class CloudSqlProxyRunner(LoggingMixin): """ -Downloads and runs cloud-sql-proxy as subprocess of the python process. +Downloads and runs cloud-sql-proxy as subprocess of the Python process. The cloud-sql-proxy needs to be downloaded and started before we can connect to the Google Cloud SQL instance via database connection. It establishes -secure tunnel connection to the database - it authorizes using the +secure tunnel connection to the database. It authorizes using the GCP credentials that are passed by the configuration. More details about the proxy can be found here: @@ -438,7 +438,7 @@ def _get_credential_parameters(self, session): def start_proxy(self): """ -Starts Cloud Sql Proxy. +Starts Cloud SQL Proxy. You have to remember to stop the proxy if
[jira] [Updated] (AIRFLOW-2440) Google Cloud SQL import/export operator
[ https://issues.apache.org/jira/browse/AIRFLOW-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-2440: Priority: Trivial (was: Major) > Google Cloud SQL import/export operator > --- > > Key: AIRFLOW-2440 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2440 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib, gcp >Reporter: Guillermo Rodríguez Cano >Assignee: Szymon Przedwojski >Priority: Trivial > Fix For: 1.10.2 > > > Implement Google Cloud SQL Operator for import/export operations as a start. > Such operator implies the corresponding hook, which can be inspired by the > Google Dataflow hook for the management of the import/export jobs (checking > the status of an issued job). > API for [mysql|https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/] > and [postgres|https://cloud.google.com/sql/docs/postgres/admin-api/v1beta4/] > (they seem alike for these two operations at least) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2440) Google Cloud SQL import/export operator
[ https://issues.apache.org/jira/browse/AIRFLOW-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-2440: Affects Version/s: (was: 2.0.0) > Google Cloud SQL import/export operator > --- > > Key: AIRFLOW-2440 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2440 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib, gcp >Reporter: Guillermo Rodríguez Cano >Assignee: Szymon Przedwojski >Priority: Trivial > Fix For: 1.10.2 > > > Implement Google Cloud SQL Operator for import/export operations as a start. > Such operator implies the corresponding hook, which can be inspired by the > Google Dataflow hook for the management of the import/export jobs (checking > the status of an issued job). > API for [mysql|https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/] > and [postgres|https://cloud.google.com/sql/docs/postgres/admin-api/v1beta4/] > (they seem alike for these two operations at least) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2440) Google Cloud SQL import/export operator
[ https://issues.apache.org/jira/browse/AIRFLOW-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-2440. - Resolution: Fixed Fix Version/s: 1.10.2 > Google Cloud SQL import/export operator > --- > > Key: AIRFLOW-2440 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2440 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib, gcp >Reporter: Guillermo Rodríguez Cano >Assignee: Szymon Przedwojski >Priority: Major > Fix For: 1.10.2 > > > Implement Google Cloud SQL Operator for import/export operations as a start. > Such operator implies the corresponding hook, which can be inspired by the > Google Dataflow hook for the management of the import/export jobs (checking > the status of an issued job). > API for [mysql|https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/] > and [postgres|https://cloud.google.com/sql/docs/postgres/admin-api/v1beta4/] > (they seem alike for these two operations at least) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2440) Google Cloud SQL import/export operator
[
https://issues.apache.org/jira/browse/AIRFLOW-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710597#comment-16710597
]
ASF GitHub Bot commented on AIRFLOW-2440:
-
kaxil closed pull request #4251: [AIRFLOW-2440] Add Google Cloud SQL
import/export operator
URL: https://github.com/apache/incubator-airflow/pull/4251
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/contrib/example_dags/example_gcp_sql.py
b/airflow/contrib/example_dags/example_gcp_sql.py
index 45c0895b0f..c6838a2baf 100644
--- a/airflow/contrib/example_dags/example_gcp_sql.py
+++ b/airflow/contrib/example_dags/example_gcp_sql.py
@@ -29,22 +29,32 @@
"""
import os
-import datetime
+
+import re
import airflow
from airflow import models
-
from airflow.contrib.operators.gcp_sql_operator import
CloudSqlInstanceCreateOperator, \
CloudSqlInstancePatchOperator, CloudSqlInstanceDeleteOperator, \
CloudSqlInstanceDatabaseCreateOperator,
CloudSqlInstanceDatabasePatchOperator, \
-CloudSqlInstanceDatabaseDeleteOperator
+CloudSqlInstanceDatabaseDeleteOperator, CloudSqlInstanceExportOperator, \
+CloudSqlInstanceImportOperator
+from airflow.contrib.operators.gcs_acl_operator import \
+GoogleCloudStorageBucketCreateAclEntryOperator, \
+GoogleCloudStorageObjectCreateAclEntryOperator
# [START howto_operator_cloudsql_arguments]
PROJECT_ID = os.environ.get('PROJECT_ID', 'example-project')
-INSTANCE_NAME = os.environ.get('INSTANCE_NAME', 'testpostgres')
+INSTANCE_NAME = os.environ.get('INSTANCE_NAME', 'test-mysql')
+INSTANCE_NAME2 = os.environ.get('INSTANCE_NAME2', 'test-mysql2')
DB_NAME = os.environ.get('DB_NAME', 'testdb')
# [END howto_operator_cloudsql_arguments]
+# [START howto_operator_cloudsql_export_import_arguments]
+EXPORT_URI = os.environ.get('EXPORT_URI', 'gs://bucketName/fileName')
+IMPORT_URI = os.environ.get('IMPORT_URI', 'gs://bucketName/fileName')
+# [END howto_operator_cloudsql_export_import_arguments]
+
# Bodies below represent Cloud SQL instance resources:
# https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/instances
@@ -86,6 +96,16 @@
"region": "europe-west4",
}
# [END howto_operator_cloudsql_create_body]
+
+body2 = {
+"name": INSTANCE_NAME2,
+"settings": {
+"tier": "db-n1-standard-1",
+},
+"databaseVersion": "MYSQL_5_7",
+"region": "europe-west4",
+}
+
# [START howto_operator_cloudsql_patch_body]
patch_body = {
"name": INSTANCE_NAME,
@@ -102,6 +122,25 @@
}
}
# [END howto_operator_cloudsql_patch_body]
+# [START howto_operator_cloudsql_export_body]
+export_body = {
+"exportContext": {
+"fileType": "sql",
+"uri": EXPORT_URI,
+"sqlExportOptions": {
+"schemaOnly": False
+}
+}
+}
+# [END howto_operator_cloudsql_export_body]
+# [START howto_operator_cloudsql_import_body]
+import_body = {
+"importContext": {
+"fileType": "sql",
+"uri": IMPORT_URI
+}
+}
+# [END howto_operator_cloudsql_import_body]
# [START howto_operator_cloudsql_db_create_body]
db_create_body = {
"instance": INSTANCE_NAME,
@@ -123,16 +162,40 @@
with models.DAG(
'example_gcp_sql',
default_args=default_args,
-schedule_interval=datetime.timedelta(days=1)
+schedule_interval=None
) as dag:
+prev_task = None
+
+def next_dep(task, prev):
+prev >> task
+return task
+
+# ## #
+# ### INSTANCES SET UP # #
+# ## #
+
# [START howto_operator_cloudsql_create]
-sql_instance_create_task = CloudSqlInstanceCreateOperator(
+sql_instance_create = CloudSqlInstanceCreateOperator(
project_id=PROJECT_ID,
body=body,
instance=INSTANCE_NAME,
-task_id='sql_instance_create_task'
+task_id='sql_instance_create'
)
# [END howto_operator_cloudsql_create]
+prev_task = sql_instance_create
+
+sql_instance_create_2 = CloudSqlInstanceCreateOperator(
+project_id=PROJECT_ID,
+body=body2,
+instance=INSTANCE_NAME2,
+task_id='sql_instance_create_2'
+)
+prev_task = next_dep(sql_instance_create_2, prev_task)
+
+# ## #
+# ### MODIFYING INSTANCE AND ITS DATABASE ## #
+# ## #
+
# [START howto_operator_cloudsql_patch]
sql_instance_patch_task = CloudSqlInstancePatchOperator(
project_id=PROJECT_ID,
@@ -141,6 +204,8 @@
task_id='sql_instance_patch_task'
)
# [END howto_operator_cloudsql_patch]
+prev_task =
[GitHub] kaxil closed pull request #4251: [AIRFLOW-2440] Add Google Cloud SQL import/export operator
kaxil closed pull request #4251: [AIRFLOW-2440] Add Google Cloud SQL
import/export operator
URL: https://github.com/apache/incubator-airflow/pull/4251
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/contrib/example_dags/example_gcp_sql.py
b/airflow/contrib/example_dags/example_gcp_sql.py
index 45c0895b0f..c6838a2baf 100644
--- a/airflow/contrib/example_dags/example_gcp_sql.py
+++ b/airflow/contrib/example_dags/example_gcp_sql.py
@@ -29,22 +29,32 @@
"""
import os
-import datetime
+
+import re
import airflow
from airflow import models
-
from airflow.contrib.operators.gcp_sql_operator import
CloudSqlInstanceCreateOperator, \
CloudSqlInstancePatchOperator, CloudSqlInstanceDeleteOperator, \
CloudSqlInstanceDatabaseCreateOperator,
CloudSqlInstanceDatabasePatchOperator, \
-CloudSqlInstanceDatabaseDeleteOperator
+CloudSqlInstanceDatabaseDeleteOperator, CloudSqlInstanceExportOperator, \
+CloudSqlInstanceImportOperator
+from airflow.contrib.operators.gcs_acl_operator import \
+GoogleCloudStorageBucketCreateAclEntryOperator, \
+GoogleCloudStorageObjectCreateAclEntryOperator
# [START howto_operator_cloudsql_arguments]
PROJECT_ID = os.environ.get('PROJECT_ID', 'example-project')
-INSTANCE_NAME = os.environ.get('INSTANCE_NAME', 'testpostgres')
+INSTANCE_NAME = os.environ.get('INSTANCE_NAME', 'test-mysql')
+INSTANCE_NAME2 = os.environ.get('INSTANCE_NAME2', 'test-mysql2')
DB_NAME = os.environ.get('DB_NAME', 'testdb')
# [END howto_operator_cloudsql_arguments]
+# [START howto_operator_cloudsql_export_import_arguments]
+EXPORT_URI = os.environ.get('EXPORT_URI', 'gs://bucketName/fileName')
+IMPORT_URI = os.environ.get('IMPORT_URI', 'gs://bucketName/fileName')
+# [END howto_operator_cloudsql_export_import_arguments]
+
# Bodies below represent Cloud SQL instance resources:
# https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/instances
@@ -86,6 +96,16 @@
"region": "europe-west4",
}
# [END howto_operator_cloudsql_create_body]
+
+body2 = {
+"name": INSTANCE_NAME2,
+"settings": {
+"tier": "db-n1-standard-1",
+},
+"databaseVersion": "MYSQL_5_7",
+"region": "europe-west4",
+}
+
# [START howto_operator_cloudsql_patch_body]
patch_body = {
"name": INSTANCE_NAME,
@@ -102,6 +122,25 @@
}
}
# [END howto_operator_cloudsql_patch_body]
+# [START howto_operator_cloudsql_export_body]
+export_body = {
+"exportContext": {
+"fileType": "sql",
+"uri": EXPORT_URI,
+"sqlExportOptions": {
+"schemaOnly": False
+}
+}
+}
+# [END howto_operator_cloudsql_export_body]
+# [START howto_operator_cloudsql_import_body]
+import_body = {
+"importContext": {
+"fileType": "sql",
+"uri": IMPORT_URI
+}
+}
+# [END howto_operator_cloudsql_import_body]
# [START howto_operator_cloudsql_db_create_body]
db_create_body = {
"instance": INSTANCE_NAME,
@@ -123,16 +162,40 @@
with models.DAG(
'example_gcp_sql',
default_args=default_args,
-schedule_interval=datetime.timedelta(days=1)
+schedule_interval=None
) as dag:
+prev_task = None
+
+def next_dep(task, prev):
+prev >> task
+return task
+
+# ## #
+# ### INSTANCES SET UP # #
+# ## #
+
# [START howto_operator_cloudsql_create]
-sql_instance_create_task = CloudSqlInstanceCreateOperator(
+sql_instance_create = CloudSqlInstanceCreateOperator(
project_id=PROJECT_ID,
body=body,
instance=INSTANCE_NAME,
-task_id='sql_instance_create_task'
+task_id='sql_instance_create'
)
# [END howto_operator_cloudsql_create]
+prev_task = sql_instance_create
+
+sql_instance_create_2 = CloudSqlInstanceCreateOperator(
+project_id=PROJECT_ID,
+body=body2,
+instance=INSTANCE_NAME2,
+task_id='sql_instance_create_2'
+)
+prev_task = next_dep(sql_instance_create_2, prev_task)
+
+# ## #
+# ### MODIFYING INSTANCE AND ITS DATABASE ## #
+# ## #
+
# [START howto_operator_cloudsql_patch]
sql_instance_patch_task = CloudSqlInstancePatchOperator(
project_id=PROJECT_ID,
@@ -141,6 +204,8 @@
task_id='sql_instance_patch_task'
)
# [END howto_operator_cloudsql_patch]
+prev_task = next_dep(sql_instance_patch_task, prev_task)
+
# [START howto_operator_cloudsql_db_create]
sql_db_create_task = CloudSqlInstanceDatabaseCreateOperator(
project_id=PROJECT_ID,
@@ -149,6 +214,8 @@
[GitHub] codecov-io edited a comment on issue #4282: [AIRFLOW-XXX] Add Docstrings as Requirement in Contribution template
codecov-io edited a comment on issue #4282: [AIRFLOW-XXX] Add Docstrings as Requirement in Contribution template URL: https://github.com/apache/incubator-airflow/pull/4282#issuecomment-444634967 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4282?src=pr=h1) Report > Merging [#4282](https://codecov.io/gh/apache/incubator-airflow/pull/4282?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/c9f987173d637e28033b9331ea085fd98594b713?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4282?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4282 +/- ## == - Coverage 78.08% 78.07% -0.01% == Files 201 201 Lines 1645816458 == - Hits1285112850 -1 - Misses 3607 3608 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4282?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4282/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `92.29% <0%> (-0.05%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4282?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4282?src=pr=footer). Last update [c9f9871...a3807b4](https://codecov.io/gh/apache/incubator-airflow/pull/4282?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #4282: [AIRFLOW-XXX] Add Docstrings as Requirement in Contribution template
codecov-io commented on issue #4282: [AIRFLOW-XXX] Add Docstrings as Requirement in Contribution template URL: https://github.com/apache/incubator-airflow/pull/4282#issuecomment-444634967 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4282?src=pr=h1) Report > Merging [#4282](https://codecov.io/gh/apache/incubator-airflow/pull/4282?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/c9f987173d637e28033b9331ea085fd98594b713?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4282?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4282 +/- ## == - Coverage 78.08% 78.07% -0.01% == Files 201 201 Lines 1645816458 == - Hits1285112850 -1 - Misses 3607 3608 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4282?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4282/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `92.29% <0%> (-0.05%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4282?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4282?src=pr=footer). Last update [c9f9871...a3807b4](https://codecov.io/gh/apache/incubator-airflow/pull/4282?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] tmiller-msft commented on a change in pull request #4265: [AIRFLOW-3406] Implement an Azure CosmosDB operator
tmiller-msft commented on a change in pull request #4265: [AIRFLOW-3406]
Implement an Azure CosmosDB operator
URL: https://github.com/apache/incubator-airflow/pull/4265#discussion_r239224536
##
File path: airflow/contrib/hooks/azure_cosmos_hook.py
##
@@ -0,0 +1,287 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import azure.cosmos.cosmos_client as cosmos_client
+from azure.cosmos.errors import HTTPFailure
+import uuid
+
+from airflow.exceptions import AirflowBadRequest
+from airflow.hooks.base_hook import BaseHook
+
+
+class AzureCosmosDBHook(BaseHook):
+"""
+Interacts with Azure CosmosDB.
+
+login should be the endpoint uri, password should be the master key
+optionally, you can use the following extras to default these values
+{"database_name": "", "collection_name": "COLLECTION_NAME"}.
+
+:param azure_cosmos_conn_id: Reference to the Azure CosmosDB connection.
+:type azure_cosmos_conn_id: str
+"""
+
+def __init__(self, azure_cosmos_conn_id='azure_cosmos_default'):
+self.conn_id = azure_cosmos_conn_id
+self.connection = self.get_connection(self.conn_id)
+self.extras = self.connection.extra_dejson
+
+self.endpoint_uri = self.connection.login
+self.master_key = self.connection.password
+self.default_database_name = self.extras.get('database_name')
+self.default_collection_name = self.extras.get('collection_name')
+self.cosmos_client = None
+
+def get_conn(self):
+"""
+Return a cosmos db client.
+"""
+if self.cosmos_client is not None:
+return self.cosmos_client
+
+# Initialize the Python Azure Cosmos DB client
+self.cosmos_client = cosmos_client.CosmosClient(self.endpoint_uri,
{'masterKey': self.master_key})
+
+return self.cosmos_client
+
+def get_database_name(self, database_name=None):
+db_name = database_name
+if db_name is None:
+db_name = self.default_database_name
+
+if db_name is None:
+raise AirflowBadRequest("Database name must be specified")
+
+return db_name
+
+def get_collection_name(self, collection_name=None):
Review comment:
private, updated to reflect
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] tmiller-msft commented on a change in pull request #4265: [AIRFLOW-3406] Implement an Azure CosmosDB operator
tmiller-msft commented on a change in pull request #4265: [AIRFLOW-3406]
Implement an Azure CosmosDB operator
URL: https://github.com/apache/incubator-airflow/pull/4265#discussion_r239224748
##
File path: airflow/contrib/hooks/azure_cosmos_hook.py
##
@@ -0,0 +1,287 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import azure.cosmos.cosmos_client as cosmos_client
+from azure.cosmos.errors import HTTPFailure
+import uuid
+
+from airflow.exceptions import AirflowBadRequest
+from airflow.hooks.base_hook import BaseHook
+
+
+class AzureCosmosDBHook(BaseHook):
+"""
+Interacts with Azure CosmosDB.
+
+login should be the endpoint uri, password should be the master key
+optionally, you can use the following extras to default these values
+{"database_name": "", "collection_name": "COLLECTION_NAME"}.
+
+:param azure_cosmos_conn_id: Reference to the Azure CosmosDB connection.
+:type azure_cosmos_conn_id: str
+"""
+
+def __init__(self, azure_cosmos_conn_id='azure_cosmos_default'):
+self.conn_id = azure_cosmos_conn_id
+self.connection = self.get_connection(self.conn_id)
+self.extras = self.connection.extra_dejson
+
+self.endpoint_uri = self.connection.login
+self.master_key = self.connection.password
+self.default_database_name = self.extras.get('database_name')
+self.default_collection_name = self.extras.get('collection_name')
+self.cosmos_client = None
+
+def get_conn(self):
+"""
+Return a cosmos db client.
+"""
+if self.cosmos_client is not None:
+return self.cosmos_client
+
+# Initialize the Python Azure Cosmos DB client
+self.cosmos_client = cosmos_client.CosmosClient(self.endpoint_uri,
{'masterKey': self.master_key})
+
+return self.cosmos_client
+
+def get_database_name(self, database_name=None):
+db_name = database_name
+if db_name is None:
+db_name = self.default_database_name
+
+if db_name is None:
+raise AirflowBadRequest("Database name must be specified")
+
+return db_name
+
+def get_collection_name(self, collection_name=None):
+coll_name = collection_name
+if coll_name is None:
+coll_name = self.default_collection_name
+
+if coll_name is None:
+raise AirflowBadRequest("Collection name must be specified")
+
+return coll_name
+
+def does_collection_exist(self, collection_name, database_name=None):
+"""
+Checks if a collection exists in CosmosDB.
+"""
+if collection_name is None:
+raise AirflowBadRequest("Collection name cannot be None.")
+
+existing_container = list(self.get_conn().QueryContainers(
+GetDatabaseLink(self.get_database_name(database_name)), {
+"query": "SELECT * FROM r WHERE r.id=@id",
+"parameters": [
+{"name": "@id", "value": collection_name}
+]
+}))
+if len(existing_container) == 0:
+return False
+
+return True
+
+def create_collection(self, collection_name, database_name=None):
+"""
+Creates a new collection in the CosmosDB database.
+"""
+if collection_name is None:
+raise AirflowBadRequest("Collection name cannot be None.")
+
+# We need to check to see if this container already exists so we don't
try
+# to create it twice
+existing_container = list(self.get_conn().QueryContainers(
+GetDatabaseLink(self.get_database_name(database_name)), {
+"query": "SELECT * FROM r WHERE r.id=@id",
+"parameters": [
+{"name": "@id", "value": collection_name}
+]
+}))
+
+# Only create if we did not find it already existing
+if len(existing_container) == 0:
+self.get_conn().CreateContainer(
+GetDatabaseLink(self.get_database_name(database_name)),
+{"id": collection_name})
+
+def does_database_exist(self, database_name):
+
[GitHub] tmiller-msft commented on a change in pull request #4265: [AIRFLOW-3406] Implement an Azure CosmosDB operator
tmiller-msft commented on a change in pull request #4265: [AIRFLOW-3406]
Implement an Azure CosmosDB operator
URL: https://github.com/apache/incubator-airflow/pull/4265#discussion_r239224836
##
File path: airflow/contrib/hooks/azure_cosmos_hook.py
##
@@ -0,0 +1,287 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import azure.cosmos.cosmos_client as cosmos_client
+from azure.cosmos.errors import HTTPFailure
+import uuid
+
+from airflow.exceptions import AirflowBadRequest
+from airflow.hooks.base_hook import BaseHook
+
+
+class AzureCosmosDBHook(BaseHook):
+"""
+Interacts with Azure CosmosDB.
+
+login should be the endpoint uri, password should be the master key
+optionally, you can use the following extras to default these values
+{"database_name": "", "collection_name": "COLLECTION_NAME"}.
+
+:param azure_cosmos_conn_id: Reference to the Azure CosmosDB connection.
+:type azure_cosmos_conn_id: str
+"""
+
+def __init__(self, azure_cosmos_conn_id='azure_cosmos_default'):
+self.conn_id = azure_cosmos_conn_id
+self.connection = self.get_connection(self.conn_id)
+self.extras = self.connection.extra_dejson
+
+self.endpoint_uri = self.connection.login
+self.master_key = self.connection.password
+self.default_database_name = self.extras.get('database_name')
+self.default_collection_name = self.extras.get('collection_name')
+self.cosmos_client = None
+
+def get_conn(self):
+"""
+Return a cosmos db client.
+"""
+if self.cosmos_client is not None:
+return self.cosmos_client
+
+# Initialize the Python Azure Cosmos DB client
+self.cosmos_client = cosmos_client.CosmosClient(self.endpoint_uri,
{'masterKey': self.master_key})
+
+return self.cosmos_client
+
+def get_database_name(self, database_name=None):
+db_name = database_name
+if db_name is None:
+db_name = self.default_database_name
+
+if db_name is None:
+raise AirflowBadRequest("Database name must be specified")
+
+return db_name
+
+def get_collection_name(self, collection_name=None):
+coll_name = collection_name
+if coll_name is None:
+coll_name = self.default_collection_name
+
+if coll_name is None:
+raise AirflowBadRequest("Collection name must be specified")
+
+return coll_name
+
+def does_collection_exist(self, collection_name, database_name=None):
+"""
+Checks if a collection exists in CosmosDB.
+"""
+if collection_name is None:
+raise AirflowBadRequest("Collection name cannot be None.")
+
+existing_container = list(self.get_conn().QueryContainers(
+GetDatabaseLink(self.get_database_name(database_name)), {
+"query": "SELECT * FROM r WHERE r.id=@id",
+"parameters": [
+{"name": "@id", "value": collection_name}
+]
+}))
+if len(existing_container) == 0:
+return False
+
+return True
+
+def create_collection(self, collection_name, database_name=None):
+"""
+Creates a new collection in the CosmosDB database.
+"""
+if collection_name is None:
+raise AirflowBadRequest("Collection name cannot be None.")
+
+# We need to check to see if this container already exists so we don't
try
+# to create it twice
+existing_container = list(self.get_conn().QueryContainers(
+GetDatabaseLink(self.get_database_name(database_name)), {
+"query": "SELECT * FROM r WHERE r.id=@id",
+"parameters": [
+{"name": "@id", "value": collection_name}
+]
+}))
+
+# Only create if we did not find it already existing
+if len(existing_container) == 0:
+self.get_conn().CreateContainer(
+GetDatabaseLink(self.get_database_name(database_name)),
+{"id": collection_name})
+
+def does_database_exist(self, database_name):
+
[GitHub] tmiller-msft commented on a change in pull request #4265: [AIRFLOW-3406] Implement an Azure CosmosDB operator
tmiller-msft commented on a change in pull request #4265: [AIRFLOW-3406]
Implement an Azure CosmosDB operator
URL: https://github.com/apache/incubator-airflow/pull/4265#discussion_r239224796
##
File path: airflow/contrib/hooks/azure_cosmos_hook.py
##
@@ -0,0 +1,287 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import azure.cosmos.cosmos_client as cosmos_client
+from azure.cosmos.errors import HTTPFailure
+import uuid
+
+from airflow.exceptions import AirflowBadRequest
+from airflow.hooks.base_hook import BaseHook
+
+
+class AzureCosmosDBHook(BaseHook):
+"""
+Interacts with Azure CosmosDB.
+
+login should be the endpoint uri, password should be the master key
+optionally, you can use the following extras to default these values
+{"database_name": "", "collection_name": "COLLECTION_NAME"}.
+
+:param azure_cosmos_conn_id: Reference to the Azure CosmosDB connection.
+:type azure_cosmos_conn_id: str
+"""
+
+def __init__(self, azure_cosmos_conn_id='azure_cosmos_default'):
+self.conn_id = azure_cosmos_conn_id
+self.connection = self.get_connection(self.conn_id)
+self.extras = self.connection.extra_dejson
+
+self.endpoint_uri = self.connection.login
+self.master_key = self.connection.password
+self.default_database_name = self.extras.get('database_name')
+self.default_collection_name = self.extras.get('collection_name')
+self.cosmos_client = None
+
+def get_conn(self):
+"""
+Return a cosmos db client.
+"""
+if self.cosmos_client is not None:
+return self.cosmos_client
+
+# Initialize the Python Azure Cosmos DB client
+self.cosmos_client = cosmos_client.CosmosClient(self.endpoint_uri,
{'masterKey': self.master_key})
+
+return self.cosmos_client
+
+def get_database_name(self, database_name=None):
+db_name = database_name
+if db_name is None:
+db_name = self.default_database_name
+
+if db_name is None:
+raise AirflowBadRequest("Database name must be specified")
+
+return db_name
+
+def get_collection_name(self, collection_name=None):
+coll_name = collection_name
+if coll_name is None:
+coll_name = self.default_collection_name
+
+if coll_name is None:
+raise AirflowBadRequest("Collection name must be specified")
+
+return coll_name
+
+def does_collection_exist(self, collection_name, database_name=None):
+"""
+Checks if a collection exists in CosmosDB.
+"""
+if collection_name is None:
+raise AirflowBadRequest("Collection name cannot be None.")
+
+existing_container = list(self.get_conn().QueryContainers(
+GetDatabaseLink(self.get_database_name(database_name)), {
+"query": "SELECT * FROM r WHERE r.id=@id",
+"parameters": [
+{"name": "@id", "value": collection_name}
+]
+}))
+if len(existing_container) == 0:
+return False
+
+return True
+
+def create_collection(self, collection_name, database_name=None):
+"""
+Creates a new collection in the CosmosDB database.
+"""
+if collection_name is None:
+raise AirflowBadRequest("Collection name cannot be None.")
+
+# We need to check to see if this container already exists so we don't
try
+# to create it twice
+existing_container = list(self.get_conn().QueryContainers(
+GetDatabaseLink(self.get_database_name(database_name)), {
+"query": "SELECT * FROM r WHERE r.id=@id",
+"parameters": [
+{"name": "@id", "value": collection_name}
+]
+}))
+
+# Only create if we did not find it already existing
+if len(existing_container) == 0:
+self.get_conn().CreateContainer(
+GetDatabaseLink(self.get_database_name(database_name)),
+{"id": collection_name})
+
+def does_database_exist(self, database_name):
+
[GitHub] tmiller-msft commented on a change in pull request #4265: [AIRFLOW-3406] Implement an Azure CosmosDB operator
tmiller-msft commented on a change in pull request #4265: [AIRFLOW-3406]
Implement an Azure CosmosDB operator
URL: https://github.com/apache/incubator-airflow/pull/4265#discussion_r239224502
##
File path: airflow/contrib/hooks/azure_cosmos_hook.py
##
@@ -0,0 +1,287 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import azure.cosmos.cosmos_client as cosmos_client
+from azure.cosmos.errors import HTTPFailure
+import uuid
+
+from airflow.exceptions import AirflowBadRequest
+from airflow.hooks.base_hook import BaseHook
+
+
+class AzureCosmosDBHook(BaseHook):
+"""
+Interacts with Azure CosmosDB.
+
+login should be the endpoint uri, password should be the master key
+optionally, you can use the following extras to default these values
+{"database_name": "", "collection_name": "COLLECTION_NAME"}.
+
+:param azure_cosmos_conn_id: Reference to the Azure CosmosDB connection.
+:type azure_cosmos_conn_id: str
+"""
+
+def __init__(self, azure_cosmos_conn_id='azure_cosmos_default'):
+self.conn_id = azure_cosmos_conn_id
+self.connection = self.get_connection(self.conn_id)
+self.extras = self.connection.extra_dejson
+
+self.endpoint_uri = self.connection.login
+self.master_key = self.connection.password
+self.default_database_name = self.extras.get('database_name')
+self.default_collection_name = self.extras.get('collection_name')
+self.cosmos_client = None
+
+def get_conn(self):
+"""
+Return a cosmos db client.
+"""
+if self.cosmos_client is not None:
+return self.cosmos_client
+
+# Initialize the Python Azure Cosmos DB client
+self.cosmos_client = cosmos_client.CosmosClient(self.endpoint_uri,
{'masterKey': self.master_key})
+
+return self.cosmos_client
+
+def get_database_name(self, database_name=None):
Review comment:
private, updated to reflect
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (AIRFLOW-1552) Airflow Filter_by_owner not working with password_auth
[ https://issues.apache.org/jira/browse/AIRFLOW-1552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-1552: Fix Version/s: 1.10.2 > Airflow Filter_by_owner not working with password_auth > -- > > Key: AIRFLOW-1552 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1552 > Project: Apache Airflow > Issue Type: Bug > Components: configuration >Affects Versions: 1.8.0 > Environment: CentOS , python 2.7 >Reporter: raghu ram reddy >Priority: Major > Fix For: 1.10.2 > > > Airflow Filter_by_owner parameter is not working with password_auth. > I created sample user using the below code from airflow documentation and > enabled password_auth. > I'm able to login as the user created but by default this user is superuser > and there is noway to modify it, default all users created by PasswordUser > are superusers. > import airflow > from airflow import models, settings > from airflow.contrib.auth.backends.password_auth import PasswordUser > user = PasswordUser(models.User()) > user.username = 'test1' > user.password = 'test1' > user.is_superuser() > session = settings.Session() > session.add(user) > session.commit() > session.close() > exit() -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3449) Airflow DAG parsing logs aren't written when using S3 logging
James Meickle created AIRFLOW-3449:
--
Summary: Airflow DAG parsing logs aren't written when using S3
logging
Key: AIRFLOW-3449
URL: https://issues.apache.org/jira/browse/AIRFLOW-3449
Project: Apache Airflow
Issue Type: Bug
Components: logging, scheduler
Affects Versions: 1.10.1, 1.10.0
Reporter: James Meickle
The default Airflow logging class outputs provides some logs to stdout, some to
"task" folders, and some to "processor" folders (generated during DAG parsing).
The 1.10.0 logging update broke this, but only for users who are also using S3
logging. This is because of this feature in the default logging config file:
{code:python}
if REMOTE_LOGGING and REMOTE_BASE_LOG_FOLDER.startswith('s3://'):
DEFAULT_LOGGING_CONFIG['handlers'].update(REMOTE_HANDLERS['s3'])
{code}
That replaces this functioning handlers block:
{code:python}
'task': {
'class': 'airflow.utils.log.file_task_handler.FileTaskHandler',
'formatter': 'airflow',
'base_log_folder': os.path.expanduser(BASE_LOG_FOLDER),
'filename_template': FILENAME_TEMPLATE,
},
'processor': {
'class':
'airflow.utils.log.file_processor_handler.FileProcessorHandler',
'formatter': 'airflow',
'base_log_folder': os.path.expanduser(PROCESSOR_LOG_FOLDER),
'filename_template': PROCESSOR_FILENAME_TEMPLATE,
},
{code}
With this non-functioning block:
{code:python}
'task': {
'class': 'airflow.utils.log.s3_task_handler.S3TaskHandler',
'formatter': 'airflow',
'base_log_folder': os.path.expanduser(BASE_LOG_FOLDER),
's3_log_folder': REMOTE_BASE_LOG_FOLDER,
'filename_template': FILENAME_TEMPLATE,
},
'processor': {
'class': 'airflow.utils.log.s3_task_handler.S3TaskHandler',
'formatter': 'airflow',
'base_log_folder': os.path.expanduser(PROCESSOR_LOG_FOLDER),
's3_log_folder': REMOTE_BASE_LOG_FOLDER,
'filename_template': PROCESSOR_FILENAME_TEMPLATE,
},
{code}
The key issue here is that both "task" and "processor" are being given a
"S3TaskHandler" class to use for logging. But that is not a generic S3 class;
it's actually a subclass of FileTaskHandler!
https://github.com/apache/incubator-airflow/blob/1.10.1/airflow/utils/log/s3_task_handler.py#L26
Since the template vars don't match the template string, the path to log to
evaluates to garbage. The handler then silently fails to log anything at all.
It is likely that anyone using a default-like logging config, plus the remote
S3 logging feature, stopped getting DAG parsing logs (either locally *or* in
S3) as of 1.10.0
Commenting out the DAG parsing section of the S3 block fixed this on my
instance.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] tmiller-msft commented on a change in pull request #4265: [AIRFLOW-3406] Implement an Azure CosmosDB operator
tmiller-msft commented on a change in pull request #4265: [AIRFLOW-3406]
Implement an Azure CosmosDB operator
URL: https://github.com/apache/incubator-airflow/pull/4265#discussion_r239224262
##
File path: airflow/contrib/hooks/azure_cosmos_hook.py
##
@@ -0,0 +1,287 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import azure.cosmos.cosmos_client as cosmos_client
+from azure.cosmos.errors import HTTPFailure
+import uuid
+
+from airflow.exceptions import AirflowBadRequest
+from airflow.hooks.base_hook import BaseHook
+
+
+class AzureCosmosDBHook(BaseHook):
+"""
+Interacts with Azure CosmosDB.
+
+login should be the endpoint uri, password should be the master key
+optionally, you can use the following extras to default these values
+{"database_name": "", "collection_name": "COLLECTION_NAME"}.
+
+:param azure_cosmos_conn_id: Reference to the Azure CosmosDB connection.
+:type azure_cosmos_conn_id: str
+"""
+
+def __init__(self, azure_cosmos_conn_id='azure_cosmos_default'):
+self.conn_id = azure_cosmos_conn_id
+self.connection = self.get_connection(self.conn_id)
+self.extras = self.connection.extra_dejson
+
+self.endpoint_uri = self.connection.login
+self.master_key = self.connection.password
+self.default_database_name = self.extras.get('database_name')
+self.default_collection_name = self.extras.get('collection_name')
+self.cosmos_client = None
+
+def get_conn(self):
+"""
+Return a cosmos db client.
+"""
+if self.cosmos_client is not None:
+return self.cosmos_client
+
+# Initialize the Python Azure Cosmos DB client
+self.cosmos_client = cosmos_client.CosmosClient(self.endpoint_uri,
{'masterKey': self.master_key})
+
+return self.cosmos_client
+
+def get_database_name(self, database_name=None):
+db_name = database_name
+if db_name is None:
+db_name = self.default_database_name
+
+if db_name is None:
+raise AirflowBadRequest("Database name must be specified")
+
+return db_name
+
+def get_collection_name(self, collection_name=None):
+coll_name = collection_name
+if coll_name is None:
+coll_name = self.default_collection_name
+
+if coll_name is None:
+raise AirflowBadRequest("Collection name must be specified")
+
+return coll_name
+
+def does_collection_exist(self, collection_name, database_name=None):
+"""
+Checks if a collection exists in CosmosDB.
+"""
+if collection_name is None:
+raise AirflowBadRequest("Collection name cannot be None.")
+
+existing_container = list(self.get_conn().QueryContainers(
+GetDatabaseLink(self.get_database_name(database_name)), {
+"query": "SELECT * FROM r WHERE r.id=@id",
+"parameters": [
+{"name": "@id", "value": collection_name}
+]
+}))
+if len(existing_container) == 0:
+return False
+
+return True
+
+def create_collection(self, collection_name, database_name=None):
+"""
+Creates a new collection in the CosmosDB database.
+"""
+if collection_name is None:
+raise AirflowBadRequest("Collection name cannot be None.")
+
+# We need to check to see if this container already exists so we don't
try
+# to create it twice
+existing_container = list(self.get_conn().QueryContainers(
+GetDatabaseLink(self.get_database_name(database_name)), {
+"query": "SELECT * FROM r WHERE r.id=@id",
+"parameters": [
+{"name": "@id", "value": collection_name}
+]
+}))
+
+# Only create if we did not find it already existing
+if len(existing_container) == 0:
+self.get_conn().CreateContainer(
+GetDatabaseLink(self.get_database_name(database_name)),
+{"id": collection_name})
+
+def does_database_exist(self, database_name):
+
[GitHub] kaxil commented on a change in pull request #4278: [AIRFLOW-2524] Add SageMaker doc to AWS integration section
kaxil commented on a change in pull request #4278: [AIRFLOW-2524] Add SageMaker doc to AWS integration section URL: https://github.com/apache/incubator-airflow/pull/4278#discussion_r239222958 ## File path: docs/integration.rst ## @@ -398,6 +398,70 @@ AwsFirehoseHook .. autoclass:: airflow.contrib.hooks.aws_firehose_hook.AwsFirehoseHook +Amazon SageMaker + + +For more instructions on using Amazon SageMaker in Airflow, please see `here`_. Review comment: `here` might have conflicts, better to use a different name This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kaxil commented on a change in pull request #4279: [AIRFLOW-3444] Explicitly set transfer operator description.
kaxil commented on a change in pull request #4279: [AIRFLOW-3444] Explicitly set transfer operator description. URL: https://github.com/apache/incubator-airflow/pull/4279#discussion_r239222376 ## File path: airflow/contrib/operators/s3_to_gcs_transfer_operator.py ## @@ -45,15 +45,14 @@ class S3ToGoogleCloudStorageTransferOperator(BaseOperator): For this to work, the service account making the request must have domain-wide delegation enabled. :type delegate_to: str +:param description: Optional transfer service job description +:type description: str :param object_conditions: Transfer service object conditions; see https://cloud.google.com/storage-transfer/docs/reference/rest/v1/TransferSpec :type object_conditions: dict :param transfer_options: Transfer service transfer options; see https://cloud.google.com/storage-transfer/docs/reference/rest/v1/TransferSpec :type transfer_options: dict -:param job_kwargs: Additional transfer job options; see Review comment: We do have things like Schedule: https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs#Schedule that can go in `job_kwargs`, what do you think? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kaxil commented on a change in pull request #4282: [AIRFLOW-XXX] Add Docstrings as Requirement in Contribution template
kaxil commented on a change in pull request #4282: [AIRFLOW-XXX] Add Docstrings as Requirement in Contribution template URL: https://github.com/apache/incubator-airflow/pull/4282#discussion_r239216387 ## File path: .github/PULL_REQUEST_TEMPLATE.md ## @@ -28,6 +28,7 @@ Make sure you have checked _all_ steps below. - [ ] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. + - All the functions and the classes in the PR contain docstrings that explain what it does Review comment: Done, updated. https://pep8.org/#descriptive-naming-styles >The underscore prefix is meant as a hint to another programmer that a variable or method starting with a single underscore is intended for internal use. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kaxil commented on a change in pull request #4282: [AIRFLOW-XXX] Add Docstrings as Requirement in Contribution template
kaxil commented on a change in pull request #4282: [AIRFLOW-XXX] Add Docstrings as Requirement in Contribution template URL: https://github.com/apache/incubator-airflow/pull/4282#discussion_r239216387 ## File path: .github/PULL_REQUEST_TEMPLATE.md ## @@ -28,6 +28,7 @@ Make sure you have checked _all_ steps below. - [ ] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. + - All the functions and the classes in the PR contain docstrings that explain what it does Review comment: Done, updated. https://pep8.org/#descriptive-naming-styles >_single_leading_underscore: weak “internal use” indicator. E.g. from M import * does not import objects whose name starts with an underscore. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kaxil commented on a change in pull request #4282: [AIRFLOW-XXX] Add Docstrings as Requirement in Contribution template
kaxil commented on a change in pull request #4282: [AIRFLOW-XXX] Add Docstrings as Requirement in Contribution template URL: https://github.com/apache/incubator-airflow/pull/4282#discussion_r239216387 ## File path: .github/PULL_REQUEST_TEMPLATE.md ## @@ -28,6 +28,7 @@ Make sure you have checked _all_ steps below. - [ ] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. + - All the functions and the classes in the PR contain docstrings that explain what it does Review comment: Sure I agree, will update that. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kaxil closed pull request #4281: Documentation Update
kaxil closed pull request #4281: Documentation Update URL: https://github.com/apache/incubator-airflow/pull/4281 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/docs/installation.rst b/docs/installation.rst index baabea3eb0..0c8be1ff40 100644 --- a/docs/installation.rst +++ b/docs/installation.rst @@ -94,6 +94,8 @@ Here's the list of the subpackages and what they enable: +-+---+-+ | kerberos| ``pip install apache-airflow[kerberos]`` | Kerberos integration for Kerberized Hadoop | +-+---+-+ +| kubernetes | ``pip install apache-airflow[kubernetes]``| Kubernetes Executor and operator| ++-+---+-+ | ldap| ``pip install apache-airflow[ldap]`` | LDAP authentication for users | +-+---+-+ | mssql | ``pip install apache-airflow[mssql]`` | Microsoft SQL Server operators and hook,| This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] feng-tao commented on a change in pull request #4282: [AIRFLOW-XXX] Add Docstrings as Requirement in Contribution template
feng-tao commented on a change in pull request #4282: [AIRFLOW-XXX] Add Docstrings as Requirement in Contribution template URL: https://github.com/apache/incubator-airflow/pull/4282#discussion_r239212369 ## File path: .github/PULL_REQUEST_TEMPLATE.md ## @@ -28,6 +28,7 @@ Make sure you have checked _all_ steps below. - [ ] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. + - All the functions and the classes in the PR contain docstrings that explain what it does Review comment: I know python doesn't have public/private func. But sometimes we use underscore in the beginning of the func to indicate it is a public func. Not sure if it is a convention else where. But we could call `All the public functions ...` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kaxil edited a comment on issue #4282: [AIRFLOW-XXX] Add Docstrings as Requirement in Contribution template
kaxil edited a comment on issue #4282: [AIRFLOW-XXX] Add Docstrings as Requirement in Contribution template URL: https://github.com/apache/incubator-airflow/pull/4282#issuecomment-444618030 Would request all the committers / PR reviewers to take note of this, so that we are all on the same page. Please check if there are proper documentation and examples. Codebase becomes more readable and understandable if it has good comments, docs or examples with it where needed. @feng-tao @ashb @Fokko @msumit @r39132 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kaxil commented on issue #4282: [AIRFLOW-XXX] Add Docstrings as Requirement in Contribution template
kaxil commented on issue #4282: [AIRFLOW-XXX] Add Docstrings as Requirement in Contribution template URL: https://github.com/apache/incubator-airflow/pull/4282#issuecomment-444618030 Would request all the committers / PR reviewers to take note of this. @feng-tao @ashb @Fokko @msumit @r39132 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kaxil opened a new pull request #4282: [AIRFLOW-XXX] Add Docstrings as Requirement in Contribution template
kaxil opened a new pull request #4282: [AIRFLOW-XXX] Add Docstrings as
Requirement in Contribution template
URL: https://github.com/apache/incubator-airflow/pull/4282
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
Each of the function/method should have a docstring as a convention. We
don't need to check the code to see what the function does.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Not needed - Doc change
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] kaxil commented on a change in pull request #4267: [AIRFLOW-3411] create openfaas hook
kaxil commented on a change in pull request #4267: [AIRFLOW-3411] create
openfaas hook
URL: https://github.com/apache/incubator-airflow/pull/4267#discussion_r239206986
##
File path: airflow/contrib/hooks/openfaas_hook.py
##
@@ -0,0 +1,98 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.hooks.base_hook import BaseHook
+import requests
+from airflow import AirflowException
+
+OK_STATUS_CODE = 202
+
+
+class OpenFaasHook(BaseHook):
+"""
+Interact with Openfaas to query, deploy, invoke and update function
+
+:param function_name: Name of the function, Defaults to None
+:type query: str
+:param conn_id: openfass connection to use, Defaults to open_faas_default
+:type conn_id: str
+"""
+
+GET_FUNCTION = "/system/function/"
+INVOKE_ASYNC_FUNCTION = "/async-function/"
+DEPLOY_FUNCTION = "/system/functions"
+UPDATE_FUNCTION = "/system/functions"
+
+def __init__(self,
+ function_name=None,
+ conn_id='open_faas_default',
+ *args, **kwargs):
+self.function_name = function_name
+self.conn_id = conn_id
+super(BaseHook, self).__init__(*args, **kwargs)
+
+def get_conn(self):
+conn = self.get_connection(self.conn_id)
+return conn
+
+def deploy_function(self, overwrite_function_if_exist, body):
+if overwrite_function_if_exist:
+self.log.info("function already exist " + self.function_name + "
going to update")
+self.update_function(body)
+else:
+url = self.get_conn().host + self.DEPLOY_FUNCTION
Review comment:
Need to add better handling, what are the required fields in the connection.
We don't have any connection for `open_faas_default`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] kaxil commented on a change in pull request #4267: [AIRFLOW-3411] create openfaas hook
kaxil commented on a change in pull request #4267: [AIRFLOW-3411] create openfaas hook URL: https://github.com/apache/incubator-airflow/pull/4267#discussion_r239207823 ## File path: airflow/contrib/hooks/openfaas_hook.py ## @@ -0,0 +1,98 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +from airflow.hooks.base_hook import BaseHook +import requests +from airflow import AirflowException + +OK_STATUS_CODE = 202 + + +class OpenFaasHook(BaseHook): +""" +Interact with Openfaas to query, deploy, invoke and update function + +:param function_name: Name of the function, Defaults to None +:type query: str +:param conn_id: openfass connection to use, Defaults to open_faas_default +:type conn_id: str +""" Review comment: Add an example of what needs to be in open-faas connection This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kaxil commented on a change in pull request #4267: [AIRFLOW-3411] create openfaas hook
kaxil commented on a change in pull request #4267: [AIRFLOW-3411] create
openfaas hook
URL: https://github.com/apache/incubator-airflow/pull/4267#discussion_r239207248
##
File path: airflow/contrib/hooks/openfaas_hook.py
##
@@ -0,0 +1,98 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.hooks.base_hook import BaseHook
+import requests
+from airflow import AirflowException
+
+OK_STATUS_CODE = 202
+
+
+class OpenFaasHook(BaseHook):
+"""
+Interact with Openfaas to query, deploy, invoke and update function
+
+:param function_name: Name of the function, Defaults to None
+:type query: str
+:param conn_id: openfass connection to use, Defaults to open_faas_default
+:type conn_id: str
+"""
+
+GET_FUNCTION = "/system/function/"
+INVOKE_ASYNC_FUNCTION = "/async-function/"
+DEPLOY_FUNCTION = "/system/functions"
+UPDATE_FUNCTION = "/system/functions"
+
+def __init__(self,
+ function_name=None,
+ conn_id='open_faas_default',
+ *args, **kwargs):
+self.function_name = function_name
+self.conn_id = conn_id
+super(BaseHook, self).__init__(*args, **kwargs)
+
+def get_conn(self):
+conn = self.get_connection(self.conn_id)
+return conn
+
+def deploy_function(self, overwrite_function_if_exist, body):
+if overwrite_function_if_exist:
+self.log.info("function already exist " + self.function_name + "
going to update")
+self.update_function(body)
+else:
+url = self.get_conn().host + self.DEPLOY_FUNCTION
+self.log.info("deploying function " + url)
Review comment:
First alphabet of anything in Logging should be capital as a convention
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] jzucker2 commented on issue #3770: [AIRFLOW-3281] Fix Kubernetes operator with git-sync
jzucker2 commented on issue #3770: [AIRFLOW-3281] Fix Kubernetes operator with git-sync URL: https://github.com/apache/incubator-airflow/pull/3770#issuecomment-444611030 @odracci any chance I could help with this? It's really awesome work and I'd love to help get this merged since git-sync is completely broken in release This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] GabrielNicolasAvellaneda opened a new pull request #4281: Documentation Update
GabrielNicolasAvellaneda opened a new pull request #4281: Documentation Update
URL: https://github.com/apache/incubator-airflow/pull/4281
Clarifiy how to install the extra package to get *KubernetesPodOperator*
available.
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] feng-tao commented on a change in pull request #4265: [AIRFLOW-3406] Implement an Azure CosmosDB operator
feng-tao commented on a change in pull request #4265: [AIRFLOW-3406] Implement
an Azure CosmosDB operator
URL: https://github.com/apache/incubator-airflow/pull/4265#discussion_r239180352
##
File path: airflow/contrib/hooks/azure_cosmos_hook.py
##
@@ -0,0 +1,287 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import azure.cosmos.cosmos_client as cosmos_client
+from azure.cosmos.errors import HTTPFailure
+import uuid
+
+from airflow.exceptions import AirflowBadRequest
+from airflow.hooks.base_hook import BaseHook
+
+
+class AzureCosmosDBHook(BaseHook):
+"""
+Interacts with Azure CosmosDB.
+
+login should be the endpoint uri, password should be the master key
+optionally, you can use the following extras to default these values
+{"database_name": "", "collection_name": "COLLECTION_NAME"}.
+
+:param azure_cosmos_conn_id: Reference to the Azure CosmosDB connection.
+:type azure_cosmos_conn_id: str
+"""
+
+def __init__(self, azure_cosmos_conn_id='azure_cosmos_default'):
+self.conn_id = azure_cosmos_conn_id
+self.connection = self.get_connection(self.conn_id)
+self.extras = self.connection.extra_dejson
+
+self.endpoint_uri = self.connection.login
+self.master_key = self.connection.password
+self.default_database_name = self.extras.get('database_name')
+self.default_collection_name = self.extras.get('collection_name')
+self.cosmos_client = None
+
+def get_conn(self):
+"""
+Return a cosmos db client.
+"""
+if self.cosmos_client is not None:
+return self.cosmos_client
+
+# Initialize the Python Azure Cosmos DB client
+self.cosmos_client = cosmos_client.CosmosClient(self.endpoint_uri,
{'masterKey': self.master_key})
+
+return self.cosmos_client
+
+def get_database_name(self, database_name=None):
+db_name = database_name
+if db_name is None:
+db_name = self.default_database_name
+
+if db_name is None:
+raise AirflowBadRequest("Database name must be specified")
+
+return db_name
+
+def get_collection_name(self, collection_name=None):
Review comment:
same
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] feng-tao commented on a change in pull request #4265: [AIRFLOW-3406] Implement an Azure CosmosDB operator
feng-tao commented on a change in pull request #4265: [AIRFLOW-3406] Implement
an Azure CosmosDB operator
URL: https://github.com/apache/incubator-airflow/pull/4265#discussion_r239181190
##
File path: airflow/contrib/hooks/azure_cosmos_hook.py
##
@@ -0,0 +1,287 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import azure.cosmos.cosmos_client as cosmos_client
+from azure.cosmos.errors import HTTPFailure
+import uuid
+
+from airflow.exceptions import AirflowBadRequest
+from airflow.hooks.base_hook import BaseHook
+
+
+class AzureCosmosDBHook(BaseHook):
+"""
+Interacts with Azure CosmosDB.
+
+login should be the endpoint uri, password should be the master key
+optionally, you can use the following extras to default these values
+{"database_name": "", "collection_name": "COLLECTION_NAME"}.
+
+:param azure_cosmos_conn_id: Reference to the Azure CosmosDB connection.
+:type azure_cosmos_conn_id: str
+"""
+
+def __init__(self, azure_cosmos_conn_id='azure_cosmos_default'):
+self.conn_id = azure_cosmos_conn_id
+self.connection = self.get_connection(self.conn_id)
+self.extras = self.connection.extra_dejson
+
+self.endpoint_uri = self.connection.login
+self.master_key = self.connection.password
+self.default_database_name = self.extras.get('database_name')
+self.default_collection_name = self.extras.get('collection_name')
+self.cosmos_client = None
+
+def get_conn(self):
+"""
+Return a cosmos db client.
+"""
+if self.cosmos_client is not None:
+return self.cosmos_client
+
+# Initialize the Python Azure Cosmos DB client
+self.cosmos_client = cosmos_client.CosmosClient(self.endpoint_uri,
{'masterKey': self.master_key})
+
+return self.cosmos_client
+
+def get_database_name(self, database_name=None):
+db_name = database_name
+if db_name is None:
+db_name = self.default_database_name
+
+if db_name is None:
+raise AirflowBadRequest("Database name must be specified")
+
+return db_name
+
+def get_collection_name(self, collection_name=None):
+coll_name = collection_name
+if coll_name is None:
+coll_name = self.default_collection_name
+
+if coll_name is None:
+raise AirflowBadRequest("Collection name must be specified")
+
+return coll_name
+
+def does_collection_exist(self, collection_name, database_name=None):
+"""
+Checks if a collection exists in CosmosDB.
+"""
+if collection_name is None:
+raise AirflowBadRequest("Collection name cannot be None.")
+
+existing_container = list(self.get_conn().QueryContainers(
+GetDatabaseLink(self.get_database_name(database_name)), {
+"query": "SELECT * FROM r WHERE r.id=@id",
+"parameters": [
+{"name": "@id", "value": collection_name}
+]
+}))
+if len(existing_container) == 0:
+return False
+
+return True
+
+def create_collection(self, collection_name, database_name=None):
+"""
+Creates a new collection in the CosmosDB database.
+"""
+if collection_name is None:
+raise AirflowBadRequest("Collection name cannot be None.")
+
+# We need to check to see if this container already exists so we don't
try
+# to create it twice
+existing_container = list(self.get_conn().QueryContainers(
+GetDatabaseLink(self.get_database_name(database_name)), {
+"query": "SELECT * FROM r WHERE r.id=@id",
+"parameters": [
+{"name": "@id", "value": collection_name}
+]
+}))
+
+# Only create if we did not find it already existing
+if len(existing_container) == 0:
+self.get_conn().CreateContainer(
+GetDatabaseLink(self.get_database_name(database_name)),
+{"id": collection_name})
+
+def does_database_exist(self, database_name):
+"""
+
[GitHub] feng-tao commented on a change in pull request #4265: [AIRFLOW-3406] Implement an Azure CosmosDB operator
feng-tao commented on a change in pull request #4265: [AIRFLOW-3406] Implement
an Azure CosmosDB operator
URL: https://github.com/apache/incubator-airflow/pull/4265#discussion_r239180322
##
File path: airflow/contrib/hooks/azure_cosmos_hook.py
##
@@ -0,0 +1,287 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import azure.cosmos.cosmos_client as cosmos_client
+from azure.cosmos.errors import HTTPFailure
+import uuid
+
+from airflow.exceptions import AirflowBadRequest
+from airflow.hooks.base_hook import BaseHook
+
+
+class AzureCosmosDBHook(BaseHook):
+"""
+Interacts with Azure CosmosDB.
+
+login should be the endpoint uri, password should be the master key
+optionally, you can use the following extras to default these values
+{"database_name": "", "collection_name": "COLLECTION_NAME"}.
+
+:param azure_cosmos_conn_id: Reference to the Azure CosmosDB connection.
+:type azure_cosmos_conn_id: str
+"""
+
+def __init__(self, azure_cosmos_conn_id='azure_cosmos_default'):
+self.conn_id = azure_cosmos_conn_id
+self.connection = self.get_connection(self.conn_id)
+self.extras = self.connection.extra_dejson
+
+self.endpoint_uri = self.connection.login
+self.master_key = self.connection.password
+self.default_database_name = self.extras.get('database_name')
+self.default_collection_name = self.extras.get('collection_name')
+self.cosmos_client = None
+
+def get_conn(self):
+"""
+Return a cosmos db client.
+"""
+if self.cosmos_client is not None:
+return self.cosmos_client
+
+# Initialize the Python Azure Cosmos DB client
+self.cosmos_client = cosmos_client.CosmosClient(self.endpoint_uri,
{'masterKey': self.master_key})
+
+return self.cosmos_client
+
+def get_database_name(self, database_name=None):
Review comment:
are this private function or public function? (if latter, good to have
docstring)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] feng-tao commented on a change in pull request #4265: [AIRFLOW-3406] Implement an Azure CosmosDB operator
feng-tao commented on a change in pull request #4265: [AIRFLOW-3406] Implement
an Azure CosmosDB operator
URL: https://github.com/apache/incubator-airflow/pull/4265#discussion_r239180720
##
File path: airflow/contrib/hooks/azure_cosmos_hook.py
##
@@ -0,0 +1,287 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import azure.cosmos.cosmos_client as cosmos_client
+from azure.cosmos.errors import HTTPFailure
+import uuid
+
+from airflow.exceptions import AirflowBadRequest
+from airflow.hooks.base_hook import BaseHook
+
+
+class AzureCosmosDBHook(BaseHook):
+"""
+Interacts with Azure CosmosDB.
+
+login should be the endpoint uri, password should be the master key
+optionally, you can use the following extras to default these values
+{"database_name": "", "collection_name": "COLLECTION_NAME"}.
+
+:param azure_cosmos_conn_id: Reference to the Azure CosmosDB connection.
+:type azure_cosmos_conn_id: str
+"""
+
+def __init__(self, azure_cosmos_conn_id='azure_cosmos_default'):
+self.conn_id = azure_cosmos_conn_id
+self.connection = self.get_connection(self.conn_id)
+self.extras = self.connection.extra_dejson
+
+self.endpoint_uri = self.connection.login
+self.master_key = self.connection.password
+self.default_database_name = self.extras.get('database_name')
+self.default_collection_name = self.extras.get('collection_name')
+self.cosmos_client = None
+
+def get_conn(self):
+"""
+Return a cosmos db client.
+"""
+if self.cosmos_client is not None:
+return self.cosmos_client
+
+# Initialize the Python Azure Cosmos DB client
+self.cosmos_client = cosmos_client.CosmosClient(self.endpoint_uri,
{'masterKey': self.master_key})
+
+return self.cosmos_client
+
+def get_database_name(self, database_name=None):
+db_name = database_name
+if db_name is None:
+db_name = self.default_database_name
+
+if db_name is None:
+raise AirflowBadRequest("Database name must be specified")
+
+return db_name
+
+def get_collection_name(self, collection_name=None):
+coll_name = collection_name
+if coll_name is None:
+coll_name = self.default_collection_name
+
+if coll_name is None:
+raise AirflowBadRequest("Collection name must be specified")
+
+return coll_name
+
+def does_collection_exist(self, collection_name, database_name=None):
+"""
+Checks if a collection exists in CosmosDB.
+"""
+if collection_name is None:
+raise AirflowBadRequest("Collection name cannot be None.")
+
+existing_container = list(self.get_conn().QueryContainers(
+GetDatabaseLink(self.get_database_name(database_name)), {
+"query": "SELECT * FROM r WHERE r.id=@id",
+"parameters": [
+{"name": "@id", "value": collection_name}
+]
+}))
+if len(existing_container) == 0:
+return False
+
+return True
+
+def create_collection(self, collection_name, database_name=None):
+"""
+Creates a new collection in the CosmosDB database.
+"""
+if collection_name is None:
+raise AirflowBadRequest("Collection name cannot be None.")
+
+# We need to check to see if this container already exists so we don't
try
+# to create it twice
+existing_container = list(self.get_conn().QueryContainers(
+GetDatabaseLink(self.get_database_name(database_name)), {
+"query": "SELECT * FROM r WHERE r.id=@id",
+"parameters": [
+{"name": "@id", "value": collection_name}
+]
+}))
+
+# Only create if we did not find it already existing
+if len(existing_container) == 0:
+self.get_conn().CreateContainer(
+GetDatabaseLink(self.get_database_name(database_name)),
+{"id": collection_name})
+
+def does_database_exist(self, database_name):
+"""
+
[GitHub] feng-tao commented on a change in pull request #4265: [AIRFLOW-3406] Implement an Azure CosmosDB operator
feng-tao commented on a change in pull request #4265: [AIRFLOW-3406] Implement
an Azure CosmosDB operator
URL: https://github.com/apache/incubator-airflow/pull/4265#discussion_r239181246
##
File path: airflow/contrib/hooks/azure_cosmos_hook.py
##
@@ -0,0 +1,287 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import azure.cosmos.cosmos_client as cosmos_client
+from azure.cosmos.errors import HTTPFailure
+import uuid
+
+from airflow.exceptions import AirflowBadRequest
+from airflow.hooks.base_hook import BaseHook
+
+
+class AzureCosmosDBHook(BaseHook):
+"""
+Interacts with Azure CosmosDB.
+
+login should be the endpoint uri, password should be the master key
+optionally, you can use the following extras to default these values
+{"database_name": "", "collection_name": "COLLECTION_NAME"}.
+
+:param azure_cosmos_conn_id: Reference to the Azure CosmosDB connection.
+:type azure_cosmos_conn_id: str
+"""
+
+def __init__(self, azure_cosmos_conn_id='azure_cosmos_default'):
+self.conn_id = azure_cosmos_conn_id
+self.connection = self.get_connection(self.conn_id)
+self.extras = self.connection.extra_dejson
+
+self.endpoint_uri = self.connection.login
+self.master_key = self.connection.password
+self.default_database_name = self.extras.get('database_name')
+self.default_collection_name = self.extras.get('collection_name')
+self.cosmos_client = None
+
+def get_conn(self):
+"""
+Return a cosmos db client.
+"""
+if self.cosmos_client is not None:
+return self.cosmos_client
+
+# Initialize the Python Azure Cosmos DB client
+self.cosmos_client = cosmos_client.CosmosClient(self.endpoint_uri,
{'masterKey': self.master_key})
+
+return self.cosmos_client
+
+def get_database_name(self, database_name=None):
+db_name = database_name
+if db_name is None:
+db_name = self.default_database_name
+
+if db_name is None:
+raise AirflowBadRequest("Database name must be specified")
+
+return db_name
+
+def get_collection_name(self, collection_name=None):
+coll_name = collection_name
+if coll_name is None:
+coll_name = self.default_collection_name
+
+if coll_name is None:
+raise AirflowBadRequest("Collection name must be specified")
+
+return coll_name
+
+def does_collection_exist(self, collection_name, database_name=None):
+"""
+Checks if a collection exists in CosmosDB.
+"""
+if collection_name is None:
+raise AirflowBadRequest("Collection name cannot be None.")
+
+existing_container = list(self.get_conn().QueryContainers(
+GetDatabaseLink(self.get_database_name(database_name)), {
+"query": "SELECT * FROM r WHERE r.id=@id",
+"parameters": [
+{"name": "@id", "value": collection_name}
+]
+}))
+if len(existing_container) == 0:
+return False
+
+return True
+
+def create_collection(self, collection_name, database_name=None):
+"""
+Creates a new collection in the CosmosDB database.
+"""
+if collection_name is None:
+raise AirflowBadRequest("Collection name cannot be None.")
+
+# We need to check to see if this container already exists so we don't
try
+# to create it twice
+existing_container = list(self.get_conn().QueryContainers(
+GetDatabaseLink(self.get_database_name(database_name)), {
+"query": "SELECT * FROM r WHERE r.id=@id",
+"parameters": [
+{"name": "@id", "value": collection_name}
+]
+}))
+
+# Only create if we did not find it already existing
+if len(existing_container) == 0:
+self.get_conn().CreateContainer(
+GetDatabaseLink(self.get_database_name(database_name)),
+{"id": collection_name})
+
+def does_database_exist(self, database_name):
+"""
+
[GitHub] feng-tao commented on a change in pull request #4265: [AIRFLOW-3406] Implement an Azure CosmosDB operator
feng-tao commented on a change in pull request #4265: [AIRFLOW-3406] Implement
an Azure CosmosDB operator
URL: https://github.com/apache/incubator-airflow/pull/4265#discussion_r239181218
##
File path: airflow/contrib/hooks/azure_cosmos_hook.py
##
@@ -0,0 +1,287 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import azure.cosmos.cosmos_client as cosmos_client
+from azure.cosmos.errors import HTTPFailure
+import uuid
+
+from airflow.exceptions import AirflowBadRequest
+from airflow.hooks.base_hook import BaseHook
+
+
+class AzureCosmosDBHook(BaseHook):
+"""
+Interacts with Azure CosmosDB.
+
+login should be the endpoint uri, password should be the master key
+optionally, you can use the following extras to default these values
+{"database_name": "", "collection_name": "COLLECTION_NAME"}.
+
+:param azure_cosmos_conn_id: Reference to the Azure CosmosDB connection.
+:type azure_cosmos_conn_id: str
+"""
+
+def __init__(self, azure_cosmos_conn_id='azure_cosmos_default'):
+self.conn_id = azure_cosmos_conn_id
+self.connection = self.get_connection(self.conn_id)
+self.extras = self.connection.extra_dejson
+
+self.endpoint_uri = self.connection.login
+self.master_key = self.connection.password
+self.default_database_name = self.extras.get('database_name')
+self.default_collection_name = self.extras.get('collection_name')
+self.cosmos_client = None
+
+def get_conn(self):
+"""
+Return a cosmos db client.
+"""
+if self.cosmos_client is not None:
+return self.cosmos_client
+
+# Initialize the Python Azure Cosmos DB client
+self.cosmos_client = cosmos_client.CosmosClient(self.endpoint_uri,
{'masterKey': self.master_key})
+
+return self.cosmos_client
+
+def get_database_name(self, database_name=None):
+db_name = database_name
+if db_name is None:
+db_name = self.default_database_name
+
+if db_name is None:
+raise AirflowBadRequest("Database name must be specified")
+
+return db_name
+
+def get_collection_name(self, collection_name=None):
+coll_name = collection_name
+if coll_name is None:
+coll_name = self.default_collection_name
+
+if coll_name is None:
+raise AirflowBadRequest("Collection name must be specified")
+
+return coll_name
+
+def does_collection_exist(self, collection_name, database_name=None):
+"""
+Checks if a collection exists in CosmosDB.
+"""
+if collection_name is None:
+raise AirflowBadRequest("Collection name cannot be None.")
+
+existing_container = list(self.get_conn().QueryContainers(
+GetDatabaseLink(self.get_database_name(database_name)), {
+"query": "SELECT * FROM r WHERE r.id=@id",
+"parameters": [
+{"name": "@id", "value": collection_name}
+]
+}))
+if len(existing_container) == 0:
+return False
+
+return True
+
+def create_collection(self, collection_name, database_name=None):
+"""
+Creates a new collection in the CosmosDB database.
+"""
+if collection_name is None:
+raise AirflowBadRequest("Collection name cannot be None.")
+
+# We need to check to see if this container already exists so we don't
try
+# to create it twice
+existing_container = list(self.get_conn().QueryContainers(
+GetDatabaseLink(self.get_database_name(database_name)), {
+"query": "SELECT * FROM r WHERE r.id=@id",
+"parameters": [
+{"name": "@id", "value": collection_name}
+]
+}))
+
+# Only create if we did not find it already existing
+if len(existing_container) == 0:
+self.get_conn().CreateContainer(
+GetDatabaseLink(self.get_database_name(database_name)),
+{"id": collection_name})
+
+def does_database_exist(self, database_name):
+"""
+
[GitHub] codecov-io edited a comment on issue #4279: [AIRFLOW-3444] Explicitly set transfer operator description.
codecov-io edited a comment on issue #4279: [AIRFLOW-3444] Explicitly set transfer operator description. URL: https://github.com/apache/incubator-airflow/pull/4279#issuecomment-444370208 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4279?src=pr=h1) Report > Merging [#4279](https://codecov.io/gh/apache/incubator-airflow/pull/4279?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/9c04e8f339a6d84b2fff983e6584af2b81249652?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4279?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#4279 +/- ## === Coverage 78.08% 78.08% === Files 201 201 Lines 1645816458 === Hits1285112851 Misses 3607 3607 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4279?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4279?src=pr=footer). Last update [9c04e8f...7c0d72b](https://codecov.io/gh/apache/incubator-airflow/pull/4279?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] feng-tao closed pull request #4280: [AIRFLOW-XXX] Updated kubernetes.rst
feng-tao closed pull request #4280: [AIRFLOW-XXX] Updated kubernetes.rst
URL: https://github.com/apache/incubator-airflow/pull/4280
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/docs/kubernetes.rst b/docs/kubernetes.rst
index bb2d99e3ad..479cef87f6 100644
--- a/docs/kubernetes.rst
+++ b/docs/kubernetes.rst
@@ -34,6 +34,9 @@ Kubernetes Operator
from airflow.contrib.operators import KubernetesOperator
from airflow.contrib.operators.kubernetes_pod_operator import
KubernetesPodOperator
from airflow.contrib.kubernetes.secret import Secret
+from airflow.contrib.kubernetes.volume import Volume
+from airflow.contrib.kubernetes.volume_mount import VolumeMount
+
secret_file = Secret('volume', '/etc/sql_conn', 'airflow-secrets',
'sql_alchemy_conn')
secret_env = Secret('env', 'SQL_CONN', 'airflow-secrets',
'sql_alchemy_conn')
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-3445) MariaDB explicit_defaults_for_timestamp = 1 Does not work.
[
https://issues.apache.org/jira/browse/AIRFLOW-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710286#comment-16710286
]
Roger Kaufmann commented on AIRFLOW-3445:
-
No idea. I just went through each line and patched a fork (I'm quite new to
this ;))
[https://github.com/RogerKaufmann/incubator-airflow/commit/e63c35c450c4e95427cc70617f423416297cddd9]
> MariaDB explicit_defaults_for_timestamp = 1 Does not work.
> --
>
> Key: AIRFLOW-3445
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3445
> Project: Apache Airflow
> Issue Type: Bug
> Components: database
>Affects Versions: 1.10.1
> Environment: Hosted VM on the Google Cloud Platform, Compute Engine:
> Machine type: n1-standard-2 (2 vCPUs, 7.5 GB memory)
> Operating System CentOS
>Reporter: Conor Molloy
>Priority: Blocker
> Fix For: 1.10.2
>
>
> {{Running into an issue when running }}
> {{`airflow upgradedb`}}
> {{ going from `1.9` -> `1.10.1`}}
> {{}}
> {code:java}
> `sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1193,
> "Unknown system variable 'explicit_defaults_for_timestamp'") [SQL: 'SELECT
> @@explicit_defaults_for_timestamp']`{code}
> {{I saw this link on the airflow website.}}
> {{[https://airflow.readthedocs.io/en/stable/faq.html#how-to-fix-exception-global-variable-explicit-defaults-for-timestamp-needs-to-be-on-1|http://example.com]}}
> {{Here it says you can set}}
> {code:java}
> `explicit_defaults_for_timestamp = 1`{code}
> {{in the _my.cnf_ file. However I am using Mariadb and when I add this to the
> _my.cnf_ file the}}
> {noformat}
> mariadb.service{noformat}
> {{fails to start up. Has anyone else come across this issue?}}
>
> The output from
> {code:java}
> `SHOW VARIABLES like '%version%'`{code}
> was
> {code:java}
> `+-+--+`
> `| Variable_name | Value |`
> `+-+--+`
> `| innodb_version | 5.5.59-MariaDB-38.11 |`
> `| protocol_version | 10 |`
> `| slave_type_conversions | |`
> `| version | 5.5.60-MariaDB |`
> `| version_comment | MariaDB Server |`
> `| version_compile_machine | x86_64 |`
> `| version_compile_os | Linux |`
> `+-+--+`{code}
> The MariaDB does not have the argument as it is a MySQL only feature.
> [https://mariadb.com/kb/en/library/system-variable-differences-between-mariadb-100-and-mysql-56/|http://example.com]
> There may need to be a check for MariaDB before upgrading, as mentioned by
> Ash in this Slack thread.
> [https://apache-airflow.slack.com/archives/CCQB40SQJ/p1543918149008100|http://example.com]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3447) Intended usage of ts_nodash macro broken with migration to new time system.
[
https://issues.apache.org/jira/browse/AIRFLOW-3447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710235#comment-16710235
]
Luka Draksler commented on AIRFLOW-3447:
We simply replaced ts_nodash in models.py with
{code:java}
ts_nodash = self.execution_date.strftime('%Y%m%dT%H%M%S')
{code}
resulting in the same value as before
> Intended usage of ts_nodash macro broken with migration to new time system.
> ---
>
> Key: AIRFLOW-3447
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3447
> Project: Apache Airflow
> Issue Type: Bug
> Components: core
>Reporter: Luka Draksler
>Priority: Minor
> Labels: easyfix
>
> Migration to timezone aware times broke the intended usage of ts_nodash macro.
> ts_nodash is used in certain placeholders to create different names (table
> names, cluster names...). As such it is alphanumeric only, it contains no
> characters that could be deemed illegal by various naming restrictions.
> Migration to new time system changed that.
> As an example, this would be returned currently:
> {{20181205T125657.169324+}}
> {{before:}}
> {{20181204T03}}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] codecov-io edited a comment on issue #4251: [AIRFLOW-2440] Add Google Cloud SQL import/export operator
codecov-io edited a comment on issue #4251: [AIRFLOW-2440] Add Google Cloud SQL import/export operator URL: https://github.com/apache/incubator-airflow/pull/4251#issuecomment-442388853 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4251?src=pr=h1) Report > Merging [#4251](https://codecov.io/gh/apache/incubator-airflow/pull/4251?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/9c04e8f339a6d84b2fff983e6584af2b81249652?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4251?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4251 +/- ## == - Coverage 78.08% 78.07% -0.01% == Files 201 201 Lines 1645816458 == - Hits1285112850 -1 - Misses 3607 3608 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4251?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4251/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `92.29% <0%> (-0.05%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4251?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4251?src=pr=footer). Last update [9c04e8f...3e7b8c2](https://codecov.io/gh/apache/incubator-airflow/pull/4251?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3445) MariaDB explicit_defaults_for_timestamp = 1 Does not work.
[
https://issues.apache.org/jira/browse/AIRFLOW-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710134#comment-16710134
]
Ash Berlin-Taylor commented on AIRFLOW-3445:
Ah I see. I can't quite parse that diff though - what format is it in?
> MariaDB explicit_defaults_for_timestamp = 1 Does not work.
> --
>
> Key: AIRFLOW-3445
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3445
> Project: Apache Airflow
> Issue Type: Bug
> Components: database
>Affects Versions: 1.10.1
> Environment: Hosted VM on the Google Cloud Platform, Compute Engine:
> Machine type: n1-standard-2 (2 vCPUs, 7.5 GB memory)
> Operating System CentOS
>Reporter: Conor Molloy
>Priority: Blocker
> Fix For: 1.10.2
>
>
> {{Running into an issue when running }}
> {{`airflow upgradedb`}}
> {{ going from `1.9` -> `1.10.1`}}
> {{}}
> {code:java}
> `sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1193,
> "Unknown system variable 'explicit_defaults_for_timestamp'") [SQL: 'SELECT
> @@explicit_defaults_for_timestamp']`{code}
> {{I saw this link on the airflow website.}}
> {{[https://airflow.readthedocs.io/en/stable/faq.html#how-to-fix-exception-global-variable-explicit-defaults-for-timestamp-needs-to-be-on-1|http://example.com]}}
> {{Here it says you can set}}
> {code:java}
> `explicit_defaults_for_timestamp = 1`{code}
> {{in the _my.cnf_ file. However I am using Mariadb and when I add this to the
> _my.cnf_ file the}}
> {noformat}
> mariadb.service{noformat}
> {{fails to start up. Has anyone else come across this issue?}}
>
> The output from
> {code:java}
> `SHOW VARIABLES like '%version%'`{code}
> was
> {code:java}
> `+-+--+`
> `| Variable_name | Value |`
> `+-+--+`
> `| innodb_version | 5.5.59-MariaDB-38.11 |`
> `| protocol_version | 10 |`
> `| slave_type_conversions | |`
> `| version | 5.5.60-MariaDB |`
> `| version_comment | MariaDB Server |`
> `| version_compile_machine | x86_64 |`
> `| version_compile_os | Linux |`
> `+-+--+`{code}
> The MariaDB does not have the argument as it is a MySQL only feature.
> [https://mariadb.com/kb/en/library/system-variable-differences-between-mariadb-100-and-mysql-56/|http://example.com]
> There may need to be a check for MariaDB before upgrading, as mentioned by
> Ash in this Slack thread.
> [https://apache-airflow.slack.com/archives/CCQB40SQJ/p1543918149008100|http://example.com]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3445) MariaDB explicit_defaults_for_timestamp = 1 Does not work.
[
https://issues.apache.org/jira/browse/AIRFLOW-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710126#comment-16710126
]
Roger Kaufmann commented on AIRFLOW-3445:
-
You are correct, the per-session workaround wouldn't work, but the patch
proposed by Feng Lu should (see link below). At least on our cloudfoundry
instance of mariadb it created the tables successfully (so far it works, take
it with a grain of salt...). I guess it depends on how its gonna be solved...
[https://www.mail-archive.com/dev@airflow.incubator.apache.org/msg06640.html]
> MariaDB explicit_defaults_for_timestamp = 1 Does not work.
> --
>
> Key: AIRFLOW-3445
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3445
> Project: Apache Airflow
> Issue Type: Bug
> Components: database
>Affects Versions: 1.10.1
> Environment: Hosted VM on the Google Cloud Platform, Compute Engine:
> Machine type: n1-standard-2 (2 vCPUs, 7.5 GB memory)
> Operating System CentOS
>Reporter: Conor Molloy
>Priority: Blocker
> Fix For: 1.10.2
>
>
> {{Running into an issue when running }}
> {{`airflow upgradedb`}}
> {{ going from `1.9` -> `1.10.1`}}
> {{}}
> {code:java}
> `sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1193,
> "Unknown system variable 'explicit_defaults_for_timestamp'") [SQL: 'SELECT
> @@explicit_defaults_for_timestamp']`{code}
> {{I saw this link on the airflow website.}}
> {{[https://airflow.readthedocs.io/en/stable/faq.html#how-to-fix-exception-global-variable-explicit-defaults-for-timestamp-needs-to-be-on-1|http://example.com]}}
> {{Here it says you can set}}
> {code:java}
> `explicit_defaults_for_timestamp = 1`{code}
> {{in the _my.cnf_ file. However I am using Mariadb and when I add this to the
> _my.cnf_ file the}}
> {noformat}
> mariadb.service{noformat}
> {{fails to start up. Has anyone else come across this issue?}}
>
> The output from
> {code:java}
> `SHOW VARIABLES like '%version%'`{code}
> was
> {code:java}
> `+-+--+`
> `| Variable_name | Value |`
> `+-+--+`
> `| innodb_version | 5.5.59-MariaDB-38.11 |`
> `| protocol_version | 10 |`
> `| slave_type_conversions | |`
> `| version | 5.5.60-MariaDB |`
> `| version_comment | MariaDB Server |`
> `| version_compile_machine | x86_64 |`
> `| version_compile_os | Linux |`
> `+-+--+`{code}
> The MariaDB does not have the argument as it is a MySQL only feature.
> [https://mariadb.com/kb/en/library/system-variable-differences-between-mariadb-100-and-mysql-56/|http://example.com]
> There may need to be a check for MariaDB before upgrading, as mentioned by
> Ash in this Slack thread.
> [https://apache-airflow.slack.com/archives/CCQB40SQJ/p1543918149008100|http://example.com]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2941) example_http_operator.py Python 3.7 invalid syntax
[
https://issues.apache.org/jira/browse/AIRFLOW-2941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor resolved AIRFLOW-2941.
Resolution: Duplicate
> example_http_operator.py Python 3.7 invalid syntax
> --
>
> Key: AIRFLOW-2941
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2941
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Jon Davies
>Priority: Major
>
> example_http_operator.py fails on Python 3.7 with:
> {code:java}
> [2018-08-23 08:45:26,827] {models.py:365} ERROR - Failed to import:
> /usr/local/lib/python3.7/site-packages/airflow/example_dags/example_http_operator.py
> Traceback (most recent call last):
> File "/usr/local/lib/python3.7/site-packages/airflow/models.py", line 362,
> in process_file
> m = imp.load_source(mod_name, filepath)
> File "/usr/local/lib/python3.7/imp.py", line 172, in load_source
> module = _load(spec)
> File "", line 696, in _load
> File "", line 677, in _load_unlocked
> File "", line 728, in exec_module
> File "", line 219, in _call_with_frames_removed
> File
> "/usr/local/lib/python3.7/site-packages/airflow/example_dags/example_http_operator.py",
> line 27, in
> from airflow.operators.http_operator import SimpleHttpOperator
> File
> "/usr/local/lib/python3.7/site-packages/airflow/operators/http_operator.py",
> line 21, in
> from airflow.hooks.http_hook import HttpHook
> File "/usr/local/lib/python3.7/site-packages/airflow/hooks/http_hook.py",
> line 23, in
> import tenacity
> File "/usr/local/lib/python3.7/site-packages/tenacity/__init__.py", line 352
> from tenacity.async import AsyncRetrying
> ^
> SyntaxError: invalid syntax
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3445) MariaDB explicit_defaults_for_timestamp = 1 Does not work.
[
https://issues.apache.org/jira/browse/AIRFLOW-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710106#comment-16710106
]
Ash Berlin-Taylor commented on AIRFLOW-3445:
Tangentially related I think, as MariaDB doesn't have a
{{explicit_defaults_for_timestamp}} variable, so the check to enable it (and
the per-session workaround proposed in AIRFLOW-3036) won't work - we simply
need to not check on MariaDB and trust it has sensible defaults. I think?
> MariaDB explicit_defaults_for_timestamp = 1 Does not work.
> --
>
> Key: AIRFLOW-3445
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3445
> Project: Apache Airflow
> Issue Type: Bug
> Components: database
>Affects Versions: 1.10.1
> Environment: Hosted VM on the Google Cloud Platform, Compute Engine:
> Machine type: n1-standard-2 (2 vCPUs, 7.5 GB memory)
> Operating System CentOS
>Reporter: Conor Molloy
>Priority: Blocker
> Fix For: 1.10.2
>
>
> {{Running into an issue when running }}
> {{`airflow upgradedb`}}
> {{ going from `1.9` -> `1.10.1`}}
> {{}}
> {code:java}
> `sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1193,
> "Unknown system variable 'explicit_defaults_for_timestamp'") [SQL: 'SELECT
> @@explicit_defaults_for_timestamp']`{code}
> {{I saw this link on the airflow website.}}
> {{[https://airflow.readthedocs.io/en/stable/faq.html#how-to-fix-exception-global-variable-explicit-defaults-for-timestamp-needs-to-be-on-1|http://example.com]}}
> {{Here it says you can set}}
> {code:java}
> `explicit_defaults_for_timestamp = 1`{code}
> {{in the _my.cnf_ file. However I am using Mariadb and when I add this to the
> _my.cnf_ file the}}
> {noformat}
> mariadb.service{noformat}
> {{fails to start up. Has anyone else come across this issue?}}
>
> The output from
> {code:java}
> `SHOW VARIABLES like '%version%'`{code}
> was
> {code:java}
> `+-+--+`
> `| Variable_name | Value |`
> `+-+--+`
> `| innodb_version | 5.5.59-MariaDB-38.11 |`
> `| protocol_version | 10 |`
> `| slave_type_conversions | |`
> `| version | 5.5.60-MariaDB |`
> `| version_comment | MariaDB Server |`
> `| version_compile_machine | x86_64 |`
> `| version_compile_os | Linux |`
> `+-+--+`{code}
> The MariaDB does not have the argument as it is a MySQL only feature.
> [https://mariadb.com/kb/en/library/system-variable-differences-between-mariadb-100-and-mysql-56/|http://example.com]
> There may need to be a check for MariaDB before upgrading, as mentioned by
> Ash in this Slack thread.
> [https://apache-airflow.slack.com/archives/CCQB40SQJ/p1543918149008100|http://example.com]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3448) Syntax error when importing tenacity on python 3.7
[
https://issues.apache.org/jira/browse/AIRFLOW-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor resolved AIRFLOW-3448.
Resolution: Duplicate
> Syntax error when importing tenacity on python 3.7
> --
>
> Key: AIRFLOW-3448
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3448
> Project: Apache Airflow
> Issue Type: Bug
> Components: hooks
>Affects Versions: 1.10.1
> Environment: Python 3.7
>Reporter: Tiago Reis
>Assignee: Tiago Reis
>Priority: Major
> Labels: easyfix
> Fix For: 1.10.2
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Tenacity is used for the retry mechanism in HTTP hooks. With the introduction
> of {{async}} as a reserved keyword on Python 3.7, Tenacity 4.8.0 is broken
> with a syntax error from {{tenacity.async import AsyncRetrying}}. Updating to
> 4.10.0 will solve this
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2837) tenacity 4.8.0 breaks with python3.7
[ https://issues.apache.org/jira/browse/AIRFLOW-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor resolved AIRFLOW-2837. Resolution: Duplicate > tenacity 4.8.0 breaks with python3.7 > > > Key: AIRFLOW-2837 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2837 > Project: Apache Airflow > Issue Type: Bug >Reporter: Adrian Bridgett >Priority: Minor > > Tenacity 4.8.0 (as in setup.py) uses the reserved async keyword. > Tenacity seems to lack a changelog, 4.12.0 seems to fix the problem but I > don't know what breaking changes may have occurred. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3445) MariaDB explicit_defaults_for_timestamp = 1 Does not work.
[
https://issues.apache.org/jira/browse/AIRFLOW-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710101#comment-16710101
]
Roger Kaufmann commented on AIRFLOW-3445:
-
I think this is related to
[https://issues.apache.org/jira/projects/AIRFLOW/issues/AIRFLOW-3036]
> MariaDB explicit_defaults_for_timestamp = 1 Does not work.
> --
>
> Key: AIRFLOW-3445
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3445
> Project: Apache Airflow
> Issue Type: Bug
> Components: database
>Affects Versions: 1.10.1
> Environment: Hosted VM on the Google Cloud Platform, Compute Engine:
> Machine type: n1-standard-2 (2 vCPUs, 7.5 GB memory)
> Operating System CentOS
>Reporter: Conor Molloy
>Priority: Blocker
> Fix For: 1.10.2
>
>
> {{Running into an issue when running }}
> {{`airflow upgradedb`}}
> {{ going from `1.9` -> `1.10.1`}}
> {{}}
> {code:java}
> `sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1193,
> "Unknown system variable 'explicit_defaults_for_timestamp'") [SQL: 'SELECT
> @@explicit_defaults_for_timestamp']`{code}
> {{I saw this link on the airflow website.}}
> {{[https://airflow.readthedocs.io/en/stable/faq.html#how-to-fix-exception-global-variable-explicit-defaults-for-timestamp-needs-to-be-on-1|http://example.com]}}
> {{Here it says you can set}}
> {code:java}
> `explicit_defaults_for_timestamp = 1`{code}
> {{in the _my.cnf_ file. However I am using Mariadb and when I add this to the
> _my.cnf_ file the}}
> {noformat}
> mariadb.service{noformat}
> {{fails to start up. Has anyone else come across this issue?}}
>
> The output from
> {code:java}
> `SHOW VARIABLES like '%version%'`{code}
> was
> {code:java}
> `+-+--+`
> `| Variable_name | Value |`
> `+-+--+`
> `| innodb_version | 5.5.59-MariaDB-38.11 |`
> `| protocol_version | 10 |`
> `| slave_type_conversions | |`
> `| version | 5.5.60-MariaDB |`
> `| version_comment | MariaDB Server |`
> `| version_compile_machine | x86_64 |`
> `| version_compile_os | Linux |`
> `+-+--+`{code}
> The MariaDB does not have the argument as it is a MySQL only feature.
> [https://mariadb.com/kb/en/library/system-variable-differences-between-mariadb-100-and-mysql-56/|http://example.com]
> There may need to be a check for MariaDB before upgrading, as mentioned by
> Ash in this Slack thread.
> [https://apache-airflow.slack.com/archives/CCQB40SQJ/p1543918149008100|http://example.com]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-3448) Syntax error when importing tenacity on python 3.7
Tiago Reis created AIRFLOW-3448:
---
Summary: Syntax error when importing tenacity on python 3.7
Key: AIRFLOW-3448
URL: https://issues.apache.org/jira/browse/AIRFLOW-3448
Project: Apache Airflow
Issue Type: Bug
Components: hooks
Affects Versions: 1.10.1
Environment: Python 3.7
Reporter: Tiago Reis
Assignee: Tiago Reis
Fix For: 1.10.2
Tenacity is used for the retry mechanism in HTTP hooks. With the introduction
of {{async}} as a reserved keyword on Python 3.7, Tenacity 4.8.0 is broken with
a syntax error from {{tenacity.async import AsyncRetrying}}. Updating to 4.10.0
will solve this
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3058) Airflow log & multi-threading
[
https://issues.apache.org/jira/browse/AIRFLOW-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710075#comment-16710075
]
jack commented on AIRFLOW-3058:
---
Yep. Added -u flag to the run of the script solve this issue.
I think it's worth mentioning somewhere in the docs though... the problem shown
here isn't the delay of the print but the fact that the time stamp is incorrect.
The timestamp is of the actual print to the log and not of the time it was
entered to the buffer. This is highly confusing.
> Airflow log & multi-threading
> -
>
> Key: AIRFLOW-3058
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3058
> Project: Apache Airflow
> Issue Type: Task
>Reporter: jack
>Priority: Major
> Attachments: 456.PNG, Sni.PNG
>
>
> The airflow log does not show messages in real time when executing scripts
> with Multi-threading.
>
> for example:
>
> The left is the Airflow log time. the right is the actual time of the print
> in my code. If I would execute the script without airflow the console will
> show the times on the right.
> !Sni.PNG!
> {code:java}
> 2018-09-13 14:19:17,325] {base_task_runner.py:98} INFO - Subtask: [2018-09-13
> 14:19:17,289] {bash_operator.py:101} INFO - 2018-09-13 14:14:55.230044
> Thread: Thread-1 Generate page: #0 run #0 with URL:
> http://...=2=0=1000
> [2018-09-13 14:19:17,325] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,289] {bash_operator.py:101} INFO - 2018-09-13
> 14:14:55.231635 Thread: Thread-2 Generate page: #1 run #0 with URL:
> http://...=2=1000=1000
> [2018-09-13 14:19:17,326] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,289] {bash_operator.py:101} INFO - 2018-09-13
> 14:14:55.233226 Thread: Thread-3 Generate page: #2 run #0 with URL:
> http://...=2=2000=1000
> [2018-09-13 14:19:17,326] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,289] {bash_operator.py:101} INFO - 2018-09-13
> 14:14:55.234020 Thread: Thread-4 Generate page: #3 run #0 with URL:
> http://...=2=3000=1000
> [2018-09-13 14:19:17,326] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,290] {bash_operator.py:101} INFO - 2018-09-13
> 14:15:43.100122 Thread: Thread-1 page 0 finished. Length is 1000
> [2018-09-13 14:19:17,326] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,290] {bash_operator.py:101} INFO - 2018-09-13
> 14:15:43.100877 Thread: Thread-1 Generate page: #4 run #0 with URL:
> http://...=2=4000=1000
> [2018-09-13 14:19:17,326] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,290] {bash_operator.py:101} INFO - 2018-09-13
> 14:15:46.254536 Thread: Thread-3 page 2 finished. Length is 1000
> [2018-09-13 14:19:17,327] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,290] {bash_operator.py:101} INFO - 2018-09-13
> 14:15:46.255508 Thread: Thread-3 Generate page: #5 run #0 with URL:
> http://...=2=5000=1000
> [2018-09-13 14:19:17,327] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,290] {bash_operator.py:101} INFO - 2018-09-13
> 14:15:51.096360 Thread: Thread-2 page 1 finished. Length is 1000
> [2018-09-13 14:19:17,327] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,290] {bash_operator.py:101} INFO - 2018-09-13
> 14:15:51.097269 Thread: Thread-2 Generate page: #6 run #0 with URL:
> http://...=2=6000=1000
> [2018-09-13 14:19:17,327] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,290] {bash_operator.py:101} INFO - 2018-09-13
> 14:15:53.112621 Thread: Thread-4 page 3 finished. Length is 1000
> [2018-09-13 14:19:17,327] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,290] {bash_operator.py:101} INFO - 2018-09-13
> 14:15:53.113455 Thread: Thread-4 Generate page: #7 run #0 with URL:
> http://...=2=7000=1000
> [2018-09-13 14:19:17,327] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,290] {bash_operator.py:101} INFO - 2018-09-13
> 14:16:37.345343 Thread: Thread-3 Generate page: #8 run #0 with URL:
> http://...=2=8000=1000
> [2018-09-13 14:19:17,328] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,290] {bash_operator.py:101} INFO - 2018-09-13
> 14:16:37.701201 Thread: Thread-2 Generate page: #9 run #0 with URL:
> http://...=2=9000=1000
> [2018-09-13 14:19:17,328] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,291] {bash_operator.py:101} INFO - 2018-09-13
> 14:16:47.283796 Thread: Thread-1 page 4 finished. Length is 1000
> [2018-09-13 14:19:17,328] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,291] {bash_operator.py:101} INFO - 2018-09-13
> 14:17:27.169359 Thread: Thread-2 page 9 finished. Length is 1000
>
> {code}
> This never happens when executing regular code.. Happens only with
> multi-threading. I have some other scripts that the
[GitHub] kaxil commented on issue #4274: [AIRFLOW-3438] Fix default value of udf_config in BQOperator
kaxil commented on issue #4274: [AIRFLOW-3438] Fix default value of udf_config in BQOperator URL: https://github.com/apache/incubator-airflow/pull/4274#issuecomment-86776 @xnuinside No worries, can you review this one and test it with your GCP project, please? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-3447) Intended usage of ts_nodash macro broken with migration to new time system.
[
https://issues.apache.org/jira/browse/AIRFLOW-3447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luka Draksler updated AIRFLOW-3447:
---
Description:
Migration to timezone aware times broke the intended usage of ts_nodash macro.
ts_nodash is used in certain placeholders to create different names (table
names, cluster names...). As such it is alphanumeric only, it contains no
characters that could be deemed illegal by various naming restrictions.
Migration to new time system changed that.
As an example, this would be returned currently:
{{20181205T125657.169324+}}
{{before:}}
{{20181204T03}}
was:
Migration to timezone aware times broke the intended usage of ts_nodash macro.
ts_nodash is used in certain placeholders to create different names (table
names, cluster names...). As such it is alphanumeric only, it contains no
characters that could be deemed illegal by various naming restrictions.
Migration to new time system changed that.
As an example, this would be returned currently:
{{20181205T125657.169324+}}
{{before:}}
{{20181204T03}}{{}}
> Intended usage of ts_nodash macro broken with migration to new time system.
> ---
>
> Key: AIRFLOW-3447
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3447
> Project: Apache Airflow
> Issue Type: Bug
> Components: core
>Reporter: Luka Draksler
>Priority: Minor
> Labels: easyfix
>
> Migration to timezone aware times broke the intended usage of ts_nodash macro.
> ts_nodash is used in certain placeholders to create different names (table
> names, cluster names...). As such it is alphanumeric only, it contains no
> characters that could be deemed illegal by various naming restrictions.
> Migration to new time system changed that.
> As an example, this would be returned currently:
> {{20181205T125657.169324+}}
> {{before:}}
> {{20181204T03}}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3058) Airflow log & multi-threading
[
https://issues.apache.org/jira/browse/AIRFLOW-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710064#comment-16710064
]
Ash Berlin-Taylor commented on AIRFLOW-3058:
Was this fixed by disabling buffered IO from your python script? (for anyone
else finding this ticket in the future)
> Airflow log & multi-threading
> -
>
> Key: AIRFLOW-3058
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3058
> Project: Apache Airflow
> Issue Type: Task
>Reporter: jack
>Priority: Major
> Attachments: 456.PNG, Sni.PNG
>
>
> The airflow log does not show messages in real time when executing scripts
> with Multi-threading.
>
> for example:
>
> The left is the Airflow log time. the right is the actual time of the print
> in my code. If I would execute the script without airflow the console will
> show the times on the right.
> !Sni.PNG!
> {code:java}
> 2018-09-13 14:19:17,325] {base_task_runner.py:98} INFO - Subtask: [2018-09-13
> 14:19:17,289] {bash_operator.py:101} INFO - 2018-09-13 14:14:55.230044
> Thread: Thread-1 Generate page: #0 run #0 with URL:
> http://...=2=0=1000
> [2018-09-13 14:19:17,325] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,289] {bash_operator.py:101} INFO - 2018-09-13
> 14:14:55.231635 Thread: Thread-2 Generate page: #1 run #0 with URL:
> http://...=2=1000=1000
> [2018-09-13 14:19:17,326] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,289] {bash_operator.py:101} INFO - 2018-09-13
> 14:14:55.233226 Thread: Thread-3 Generate page: #2 run #0 with URL:
> http://...=2=2000=1000
> [2018-09-13 14:19:17,326] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,289] {bash_operator.py:101} INFO - 2018-09-13
> 14:14:55.234020 Thread: Thread-4 Generate page: #3 run #0 with URL:
> http://...=2=3000=1000
> [2018-09-13 14:19:17,326] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,290] {bash_operator.py:101} INFO - 2018-09-13
> 14:15:43.100122 Thread: Thread-1 page 0 finished. Length is 1000
> [2018-09-13 14:19:17,326] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,290] {bash_operator.py:101} INFO - 2018-09-13
> 14:15:43.100877 Thread: Thread-1 Generate page: #4 run #0 with URL:
> http://...=2=4000=1000
> [2018-09-13 14:19:17,326] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,290] {bash_operator.py:101} INFO - 2018-09-13
> 14:15:46.254536 Thread: Thread-3 page 2 finished. Length is 1000
> [2018-09-13 14:19:17,327] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,290] {bash_operator.py:101} INFO - 2018-09-13
> 14:15:46.255508 Thread: Thread-3 Generate page: #5 run #0 with URL:
> http://...=2=5000=1000
> [2018-09-13 14:19:17,327] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,290] {bash_operator.py:101} INFO - 2018-09-13
> 14:15:51.096360 Thread: Thread-2 page 1 finished. Length is 1000
> [2018-09-13 14:19:17,327] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,290] {bash_operator.py:101} INFO - 2018-09-13
> 14:15:51.097269 Thread: Thread-2 Generate page: #6 run #0 with URL:
> http://...=2=6000=1000
> [2018-09-13 14:19:17,327] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,290] {bash_operator.py:101} INFO - 2018-09-13
> 14:15:53.112621 Thread: Thread-4 page 3 finished. Length is 1000
> [2018-09-13 14:19:17,327] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,290] {bash_operator.py:101} INFO - 2018-09-13
> 14:15:53.113455 Thread: Thread-4 Generate page: #7 run #0 with URL:
> http://...=2=7000=1000
> [2018-09-13 14:19:17,327] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,290] {bash_operator.py:101} INFO - 2018-09-13
> 14:16:37.345343 Thread: Thread-3 Generate page: #8 run #0 with URL:
> http://...=2=8000=1000
> [2018-09-13 14:19:17,328] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,290] {bash_operator.py:101} INFO - 2018-09-13
> 14:16:37.701201 Thread: Thread-2 Generate page: #9 run #0 with URL:
> http://...=2=9000=1000
> [2018-09-13 14:19:17,328] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,291] {bash_operator.py:101} INFO - 2018-09-13
> 14:16:47.283796 Thread: Thread-1 page 4 finished. Length is 1000
> [2018-09-13 14:19:17,328] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,291] {bash_operator.py:101} INFO - 2018-09-13
> 14:17:27.169359 Thread: Thread-2 page 9 finished. Length is 1000
>
> {code}
> This never happens when executing regular code.. Happens only with
> multi-threading. I have some other scripts that the airflow print appears
> after more than 30 minutes.
>
> Check this one:
> hours of delay and then printing everything together. These are not real
> time. the prints in the log has no correlation to the
[jira] [Commented] (AIRFLOW-3447) Intended usage of ts_nodash macro broken with migration to new time system.
[
https://issues.apache.org/jira/browse/AIRFLOW-3447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710063#comment-16710063
]
Ash Berlin-Taylor commented on AIRFLOW-3447:
[~kaxilnaik] one for 1.10.2? I think reverting the behaviour back (by stripping
off the TZ info) is the right thing to do here?
> Intended usage of ts_nodash macro broken with migration to new time system.
> ---
>
> Key: AIRFLOW-3447
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3447
> Project: Apache Airflow
> Issue Type: Bug
> Components: core
>Reporter: Luka Draksler
>Priority: Minor
> Labels: easyfix
>
> Migration to timezone aware times broke the intended usage of ts_nodash macro.
> ts_nodash is used in certain placeholders to create different names (table
> names, cluster names...). As such it is alphanumeric only, it contains no
> characters that could be deemed illegal by various naming restrictions.
> Migration to new time system changed that.
> As an example, this would be returned currently:
> {{20181205T125657.169324+}}
> {{before:}}
> {{20181204T03}}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (AIRFLOW-3058) Airflow log & multi-threading
[
https://issues.apache.org/jira/browse/AIRFLOW-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jack closed AIRFLOW-3058.
-
Resolution: Invalid
> Airflow log & multi-threading
> -
>
> Key: AIRFLOW-3058
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3058
> Project: Apache Airflow
> Issue Type: Task
>Reporter: jack
>Priority: Major
> Attachments: 456.PNG, Sni.PNG
>
>
> The airflow log does not show messages in real time when executing scripts
> with Multi-threading.
>
> for example:
>
> The left is the Airflow log time. the right is the actual time of the print
> in my code. If I would execute the script without airflow the console will
> show the times on the right.
> !Sni.PNG!
> {code:java}
> 2018-09-13 14:19:17,325] {base_task_runner.py:98} INFO - Subtask: [2018-09-13
> 14:19:17,289] {bash_operator.py:101} INFO - 2018-09-13 14:14:55.230044
> Thread: Thread-1 Generate page: #0 run #0 with URL:
> http://...=2=0=1000
> [2018-09-13 14:19:17,325] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,289] {bash_operator.py:101} INFO - 2018-09-13
> 14:14:55.231635 Thread: Thread-2 Generate page: #1 run #0 with URL:
> http://...=2=1000=1000
> [2018-09-13 14:19:17,326] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,289] {bash_operator.py:101} INFO - 2018-09-13
> 14:14:55.233226 Thread: Thread-3 Generate page: #2 run #0 with URL:
> http://...=2=2000=1000
> [2018-09-13 14:19:17,326] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,289] {bash_operator.py:101} INFO - 2018-09-13
> 14:14:55.234020 Thread: Thread-4 Generate page: #3 run #0 with URL:
> http://...=2=3000=1000
> [2018-09-13 14:19:17,326] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,290] {bash_operator.py:101} INFO - 2018-09-13
> 14:15:43.100122 Thread: Thread-1 page 0 finished. Length is 1000
> [2018-09-13 14:19:17,326] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,290] {bash_operator.py:101} INFO - 2018-09-13
> 14:15:43.100877 Thread: Thread-1 Generate page: #4 run #0 with URL:
> http://...=2=4000=1000
> [2018-09-13 14:19:17,326] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,290] {bash_operator.py:101} INFO - 2018-09-13
> 14:15:46.254536 Thread: Thread-3 page 2 finished. Length is 1000
> [2018-09-13 14:19:17,327] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,290] {bash_operator.py:101} INFO - 2018-09-13
> 14:15:46.255508 Thread: Thread-3 Generate page: #5 run #0 with URL:
> http://...=2=5000=1000
> [2018-09-13 14:19:17,327] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,290] {bash_operator.py:101} INFO - 2018-09-13
> 14:15:51.096360 Thread: Thread-2 page 1 finished. Length is 1000
> [2018-09-13 14:19:17,327] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,290] {bash_operator.py:101} INFO - 2018-09-13
> 14:15:51.097269 Thread: Thread-2 Generate page: #6 run #0 with URL:
> http://...=2=6000=1000
> [2018-09-13 14:19:17,327] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,290] {bash_operator.py:101} INFO - 2018-09-13
> 14:15:53.112621 Thread: Thread-4 page 3 finished. Length is 1000
> [2018-09-13 14:19:17,327] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,290] {bash_operator.py:101} INFO - 2018-09-13
> 14:15:53.113455 Thread: Thread-4 Generate page: #7 run #0 with URL:
> http://...=2=7000=1000
> [2018-09-13 14:19:17,327] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,290] {bash_operator.py:101} INFO - 2018-09-13
> 14:16:37.345343 Thread: Thread-3 Generate page: #8 run #0 with URL:
> http://...=2=8000=1000
> [2018-09-13 14:19:17,328] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,290] {bash_operator.py:101} INFO - 2018-09-13
> 14:16:37.701201 Thread: Thread-2 Generate page: #9 run #0 with URL:
> http://...=2=9000=1000
> [2018-09-13 14:19:17,328] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,291] {bash_operator.py:101} INFO - 2018-09-13
> 14:16:47.283796 Thread: Thread-1 page 4 finished. Length is 1000
> [2018-09-13 14:19:17,328] {base_task_runner.py:98} INFO - Subtask:
> [2018-09-13 14:19:17,291] {bash_operator.py:101} INFO - 2018-09-13
> 14:17:27.169359 Thread: Thread-2 page 9 finished. Length is 1000
>
> {code}
> This never happens when executing regular code.. Happens only with
> multi-threading. I have some other scripts that the airflow print appears
> after more than 30 minutes.
>
> Check this one:
> hours of delay and then printing everything together. These are not real
> time. the prints in the log has no correlation to the actual time the command
> was executed.
>
> !456.PNG!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3447) Intended usage of ts_nodash macro broken with migration to new time system.
[
https://issues.apache.org/jira/browse/AIRFLOW-3447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luka Draksler updated AIRFLOW-3447:
---
Summary: Intended usage of ts_nodash macro broken with migration to new
time system. (was: Migration to timezone aware times broke the intended usage
of ts_nodash macro.)
> Intended usage of ts_nodash macro broken with migration to new time system.
> ---
>
> Key: AIRFLOW-3447
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3447
> Project: Apache Airflow
> Issue Type: Bug
> Components: core
>Reporter: Luka Draksler
>Priority: Minor
> Labels: easyfix
>
> Migration to timezone aware times broke the intended usage of ts_nodash macro.
> ts_nodash is used in certain placeholders to create different names (table
> names, cluster names...). As such it is alphanumeric only, it contains no
> characters that could be deemed illegal by various naming restrictions.
> Migration to new time system changed that.
> As an example, this would be returned currently:
> {{20181205T125657.169324+}}
> {{before:}}
> {{20181204T03}}{{}}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-3447) Migration to timezone aware times broke the intended usage of ts_nodash macro.
Luka Draksler created AIRFLOW-3447:
--
Summary: Migration to timezone aware times broke the intended
usage of ts_nodash macro.
Key: AIRFLOW-3447
URL: https://issues.apache.org/jira/browse/AIRFLOW-3447
Project: Apache Airflow
Issue Type: Bug
Components: core
Reporter: Luka Draksler
Migration to timezone aware times broke the intended usage of ts_nodash macro.
ts_nodash is used in certain placeholders to create different names (table
names, cluster names...). As such it is alphanumeric only, it contains no
characters that could be deemed illegal by various naming restrictions.
Migration to new time system changed that.
As an example, this would be returned currently:
{{20181205T125657.169324+}}
{{before:}}
{{20181204T03}}{{}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] codecov-io edited a comment on issue #4280: Updated kubernetes.rst
codecov-io edited a comment on issue #4280: Updated kubernetes.rst URL: https://github.com/apache/incubator-airflow/pull/4280#issuecomment-77609 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4280?src=pr=h1) Report > Merging [#4280](https://codecov.io/gh/apache/incubator-airflow/pull/4280?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/9c04e8f339a6d84b2fff983e6584af2b81249652?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4280?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#4280 +/- ## === Coverage 78.08% 78.08% === Files 201 201 Lines 1645816458 === Hits1285112851 Misses 3607 3607 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4280?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4280?src=pr=footer). Last update [9c04e8f...98b6a9a](https://codecov.io/gh/apache/incubator-airflow/pull/4280?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #4280: Updated kubernetes.rst
codecov-io commented on issue #4280: Updated kubernetes.rst URL: https://github.com/apache/incubator-airflow/pull/4280#issuecomment-77609 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4280?src=pr=h1) Report > Merging [#4280](https://codecov.io/gh/apache/incubator-airflow/pull/4280?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/9c04e8f339a6d84b2fff983e6584af2b81249652?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4280?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#4280 +/- ## === Coverage 78.08% 78.08% === Files 201 201 Lines 1645816458 === Hits1285112851 Misses 3607 3607 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4280?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4280?src=pr=footer). Last update [9c04e8f...98b6a9a](https://codecov.io/gh/apache/incubator-airflow/pull/4280?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sunnykrGupta opened a new pull request #4280: Updated kubernetes.rst
sunnykrGupta opened a new pull request #4280: Updated kubernetes.rst
URL: https://github.com/apache/incubator-airflow/pull/4280
import modules to complete the example set.
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-3446) Add operators for Google Cloud BigTable
Dariusz Aniszewski created AIRFLOW-3446: --- Summary: Add operators for Google Cloud BigTable Key: AIRFLOW-3446 URL: https://issues.apache.org/jira/browse/AIRFLOW-3446 Project: Apache Airflow Issue Type: Improvement Reporter: Dariusz Aniszewski Assignee: Dariusz Aniszewski Proposed operators: * BigTableInstanceCreateOperator * BigTableInstanceDeleteOperator * BigTableTableCreateOperator * BigTableTableDeleteOperator * BigTableClusterUpdateOperator * BigTableTableWaitForReplicationSensor -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] mik-laj edited a comment on issue #730: [AIRFLOW-594] Load plugins and workflows from packages
mik-laj edited a comment on issue #730: [AIRFLOW-594] Load plugins and workflows from packages URL: https://github.com/apache/incubator-airflow/pull/730#issuecomment-60774 @mahrz24 This is very interesting to me. It would simplify my work and also allow for better modularization of applications. Currently, most plugins are kept in this repo, or kept in other repositories and have to be manually copied. In the second case it is annoying, because the repository has worse versioning. Packages were created as the target format for distributing Python code and I hope that Airflow will follow this idea. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
