[GitHub] [airflow] codecov-io commented on issue #5390: [AIRFLOW-XXX] fix typos in README.md
codecov-io commented on issue #5390: [AIRFLOW-XXX] fix typos in README.md URL: https://github.com/apache/airflow/pull/5390#issuecomment-500106936 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5390?src=pr=h1) Report > Merging [#5390](https://codecov.io/gh/apache/airflow/pull/5390?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/abb506ba5b92b0a1eb447dcf880c84f3710a1c88?src=pr=desc) will **decrease** coverage by `0.19%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/5390?src=pr=tree) ```diff @@Coverage Diff@@ ## master#5390 +/- ## = - Coverage 78.75% 78.56% -0.2% = Files 481 481 Lines 3020130201 = - Hits2378623727 -59 - Misses 6415 6474 +59 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5390?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/5390/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `33.33% <0%> (-66.67%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/5390/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `50% <0%> (-50%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/5390/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.03% <0%> (-47.33%)` | :arrow_down: | | [airflow/kubernetes/pod\_generator.py](https://codecov.io/gh/apache/airflow/pull/5390/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9nZW5lcmF0b3IucHk=) | `40% <0%> (-45.72%)` | :arrow_down: | | [airflow/kubernetes/kube\_client.py](https://codecov.io/gh/apache/airflow/pull/5390/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL2t1YmVfY2xpZW50LnB5) | `30.43% <0%> (-43.48%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/5390/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `56.96% <0%> (-41.78%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/5390/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `87.17% <0%> (-4.77%)` | :arrow_down: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/5390/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `92.61% <0%> (+0.17%)` | :arrow_up: | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/5390/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `88.59% <0%> (+0.87%)` | :arrow_up: | | [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/5390/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `65% <0%> (+1.11%)` | :arrow_up: | | ... and [4 more](https://codecov.io/gh/apache/airflow/pull/5390/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5390?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5390?src=pr=footer). Last update [abb506b...9569d95](https://codecov.io/gh/apache/airflow/pull/5390?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kurtqq opened a new pull request #5390: [AIRFLOW-XXX] fix typos in README.md
kurtqq opened a new pull request #5390: [AIRFLOW-XXX] fix typos in README.md URL: https://github.com/apache/airflow/pull/5390 fix typos This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mrshu commented on issue #5276: [AIRFLOW-4414] AWSAthenaOperator: Push QueryExecutionID to XCom
mrshu commented on issue #5276: [AIRFLOW-4414] AWSAthenaOperator: Push QueryExecutionID to XCom URL: https://github.com/apache/airflow/pull/5276#issuecomment-500108040 @feluelle Sorry it took me this long to get back to this. I've added a test as you suggested and also refactored the `AWSAthenaOperator`'s test suite a bit: - the Athena query ID is now stored in a separate constant, so that it can be easily compared in tests - the Athena query ID is now a random UUID rather than `'1234'` - the test suite now contains a simple DAG, which is necessary for testing XComs If you would like to try the changes out yourself, it should be as easy as running ```bash ./run_unit_tests.sh tests.contrib.operators.test_aws_athena_operator:TestAWSAthenaOperator -s --logging-level=DEBUG ``` Please do let me know if there is anything else I can do to get this to "mergeable" state. Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] BasPH merged pull request #5390: [AIRFLOW-XXX] fix typos in README.md
BasPH merged pull request #5390: [AIRFLOW-XXX] fix typos in README.md URL: https://github.com/apache/airflow/pull/5390 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #5388: [AIRFLOW-4490] DagRun.conf returns empty dictionary by default
codecov-io commented on issue #5388: [AIRFLOW-4490] DagRun.conf returns empty dictionary by default URL: https://github.com/apache/airflow/pull/5388#issuecomment-500101598 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5388?src=pr=h1) Report > Merging [#5388](https://codecov.io/gh/apache/airflow/pull/5388?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/abb506ba5b92b0a1eb447dcf880c84f3710a1c88?src=pr=desc) will **increase** coverage by `0.26%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/5388?src=pr=tree) ```diff @@Coverage Diff @@ ## master#5388 +/- ## == + Coverage 78.75% 79.02% +0.26% == Files 481 481 Lines 3020130208 +7 == + Hits2378623873 +87 + Misses 6415 6335 -80 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5388?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/skipmixin.py](https://codecov.io/gh/apache/airflow/pull/5388/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvc2tpcG1peGluLnB5) | `100% <100%> (ø)` | :arrow_up: | | [airflow/models/dagrun.py](https://codecov.io/gh/apache/airflow/pull/5388/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFncnVuLnB5) | `95.97% <100%> (+0.12%)` | :arrow_up: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/5388/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `92.45% <100%> (+0.01%)` | :arrow_up: | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/5388/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `88.59% <0%> (+0.87%)` | :arrow_up: | | [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/5388/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `65% <0%> (+1.11%)` | :arrow_up: | | [airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/airflow/pull/5388/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5) | `77.72% <0%> (+1.77%)` | :arrow_up: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/5388/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `80.95% <0%> (+4.76%)` | :arrow_up: | | [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/5388/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `100% <0%> (+100%)` | :arrow_up: | | [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/5388/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `100% <0%> (+100%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5388?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5388?src=pr=footer). Last update [abb506b...a91847e](https://codecov.io/gh/apache/airflow/pull/5388?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] BasPH commented on a change in pull request #5362: [AIRFLOW-4669] Make airflow/dag Pylint compatible
BasPH commented on a change in pull request #5362: [AIRFLOW-4669] Make airflow/dag Pylint compatible URL: https://github.com/apache/airflow/pull/5362#discussion_r291817115 ## File path: airflow/dag/base_dag.py ## @@ -17,6 +17,8 @@ # specific language governing permissions and limitations # under the License. +"""Base classes for DAG and DagBag.""" Review comment: Would you be okay with keeping the one-liner if it fits on a single line? So ```python """Base classes for DAG and DagBag.""" ``` If line length <= 110, otherwise: ```python """ Docs here... line 1 line 2 and more """ ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] BasPH commented on a change in pull request #5362: [AIRFLOW-4669] Make airflow/dag Pylint compatible
BasPH commented on a change in pull request #5362: [AIRFLOW-4669] Make airflow/dag Pylint compatible URL: https://github.com/apache/airflow/pull/5362#discussion_r291817115 ## File path: airflow/dag/base_dag.py ## @@ -17,6 +17,8 @@ # specific language governing permissions and limitations # under the License. +"""Base classes for DAG and DagBag.""" Review comment: Would you be okay with keeping the one-liner if it fits on a single line? This is also described in PEP: https://www.python.org/dev/peps/pep-0257/#one-line-docstrings. So ```python """Base classes for DAG and DagBag.""" ``` If line length <= 110, otherwise: ```python """ Docs here... line 1 line 2 and more """ ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] BasPH commented on a change in pull request #5361: [AIRFLOW-4670] Make airflow/example_dags Pylint compatible
BasPH commented on a change in pull request #5361: [AIRFLOW-4670] Make
airflow/example_dags Pylint compatible
URL: https://github.com/apache/airflow/pull/5361#discussion_r291817403
##
File path: airflow/example_dags/example_skip_dag.py
##
@@ -30,21 +32,31 @@
# Create some placeholder operators
class DummySkipOperator(DummyOperator):
+"""Dummy operator which always skips the task."""
+
ui_color = '#e8b7e4'
def execute(self, context):
raise AirflowSkipException
-def create_test_pipeline(suffix, trigger_rule, dag):
-skip_operator =
DummySkipOperator(task_id='skip_operator_{}'.format(suffix), dag=dag)
-always_true = DummyOperator(task_id='always_true_{}'.format(suffix),
dag=dag)
-join = DummyOperator(task_id=trigger_rule, dag=dag,
trigger_rule=trigger_rule)
-final = DummyOperator(task_id='final_{}'.format(suffix), dag=dag)
+def create_test_pipeline(suffix, trigger_rule, dag_):
+"""
+Instantiate a number of operators for the given DAG.
Review comment:
I remember you telling my about this yes, forgot once again.
I tried searching for the exact problem but couldn't find it, I thought you
said the docs don't get rendered correctly without the whitespace? E.g. the
BashOperator.execute() pydoc is written without whitespace but the docs look
okay:
https://airflow.apache.org/_api/airflow/operators/bash_operator/index.html#airflow.operators.bash_operator.BashOperator.execute.
Could you let me know the exact issue?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #5363: [AIRFLOW-4725] Fix setup.py PEP440 & Sphinx-PyPI-upload dependency
codecov-io commented on issue #5363: [AIRFLOW-4725] Fix setup.py PEP440 & Sphinx-PyPI-upload dependency URL: https://github.com/apache/airflow/pull/5363#issuecomment-500156020 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5363?src=pr=h1) Report > Merging [#5363](https://codecov.io/gh/apache/airflow/pull/5363?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/827d6d40c59642cf3eb79238e9a733479de4ea60?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/5363?src=pr=tree) ```diff @@Coverage Diff @@ ## master#5363 +/- ## == - Coverage 79.03% 79.02% -0.01% == Files 481 481 Lines 3020130201 == - Hits2386823867 -1 - Misses 6333 6334 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5363?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/version.py](https://codecov.io/gh/apache/airflow/pull/5363/diff?src=pr=tree#diff-YWlyZmxvdy92ZXJzaW9uLnB5) | `100% <100%> (ø)` | :arrow_up: | | [airflow/contrib/operators/ssh\_operator.py](https://codecov.io/gh/apache/airflow/pull/5363/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zc2hfb3BlcmF0b3IucHk=) | `82.27% <0%> (-1.27%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5363?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5363?src=pr=footer). Last update [827d6d4...a87d5ab](https://codecov.io/gh/apache/airflow/pull/5363?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #5363: [AIRFLOW-4725] Fix setup.py PEP440 & Sphinx-PyPI-upload dependency
codecov-io commented on issue #5363: [AIRFLOW-4725] Fix setup.py PEP440 & Sphinx-PyPI-upload dependency URL: https://github.com/apache/airflow/pull/5363#issuecomment-500156021 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5363?src=pr=h1) Report > Merging [#5363](https://codecov.io/gh/apache/airflow/pull/5363?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/827d6d40c59642cf3eb79238e9a733479de4ea60?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/5363?src=pr=tree) ```diff @@Coverage Diff @@ ## master#5363 +/- ## == - Coverage 79.03% 79.02% -0.01% == Files 481 481 Lines 3020130201 == - Hits2386823867 -1 - Misses 6333 6334 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5363?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/version.py](https://codecov.io/gh/apache/airflow/pull/5363/diff?src=pr=tree#diff-YWlyZmxvdy92ZXJzaW9uLnB5) | `100% <100%> (ø)` | :arrow_up: | | [airflow/contrib/operators/ssh\_operator.py](https://codecov.io/gh/apache/airflow/pull/5363/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zc2hfb3BlcmF0b3IucHk=) | `82.27% <0%> (-1.27%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5363?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5363?src=pr=footer). Last update [827d6d4...a87d5ab](https://codecov.io/gh/apache/airflow/pull/5363?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] BasPH commented on issue #4763: [AIRFLOW-3946] Deprecate utils.file.TemporaryDirectory
BasPH commented on issue #4763: [AIRFLOW-3946] Deprecate utils.file.TemporaryDirectory URL: https://github.com/apache/airflow/pull/4763#issuecomment-500158934 Good point :-) Removed all usage of `airflow.utils.file.TemporaryDirectory`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #5361: [AIRFLOW-4670] Make airflow/example_dags Pylint compatible
mik-laj commented on a change in pull request #5361: [AIRFLOW-4670] Make
airflow/example_dags Pylint compatible
URL: https://github.com/apache/airflow/pull/5361#discussion_r291818634
##
File path: airflow/example_dags/example_skip_dag.py
##
@@ -30,21 +32,31 @@
# Create some placeholder operators
class DummySkipOperator(DummyOperator):
+"""Dummy operator which always skips the task."""
+
ui_color = '#e8b7e4'
def execute(self, context):
raise AirflowSkipException
-def create_test_pipeline(suffix, trigger_rule, dag):
-skip_operator =
DummySkipOperator(task_id='skip_operator_{}'.format(suffix), dag=dag)
-always_true = DummyOperator(task_id='always_true_{}'.format(suffix),
dag=dag)
-join = DummyOperator(task_id=trigger_rule, dag=dag,
trigger_rule=trigger_rule)
-final = DummyOperator(task_id='final_{}'.format(suffix), dag=dag)
+def create_test_pipeline(suffix, trigger_rule, dag_):
+"""
+Instantiate a number of operators for the given DAG.
Review comment:
It's contains the empty line after description.
https://user-images.githubusercontent.com/12058428/59151832-2b421c80-8a3a-11e9-90a7-b8a1581ab013.png;>
I do not know why it is required, but the documentation without this line is
not correctly rendered.
I made a small change in http_operator.py to demonstrate problem.
https://user-images.githubusercontent.com/12058428/59151959-c4bdfe00-8a3b-11e9-9f6c-38bb984b71bd.png;>
I receive a error message:
```
/app/airflow/docs/_api/airflow/operators/http_operator/index.rst:36:
WARNING: Unexpected indentation.
/app/airflow/docs/_api/airflow/operators/http_operator/index.rst:37:
WARNING: Block quote ends without a blank line; unexpected unindent.
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #5361: [AIRFLOW-4670] Make airflow/example_dags Pylint compatible
mik-laj commented on a change in pull request #5361: [AIRFLOW-4670] Make
airflow/example_dags Pylint compatible
URL: https://github.com/apache/airflow/pull/5361#discussion_r291818667
##
File path: airflow/example_dags/example_skip_dag.py
##
@@ -30,21 +32,31 @@
# Create some placeholder operators
class DummySkipOperator(DummyOperator):
+"""Dummy operator which always skips the task."""
+
ui_color = '#e8b7e4'
def execute(self, context):
raise AirflowSkipException
-def create_test_pipeline(suffix, trigger_rule, dag):
-skip_operator =
DummySkipOperator(task_id='skip_operator_{}'.format(suffix), dag=dag)
-always_true = DummyOperator(task_id='always_true_{}'.format(suffix),
dag=dag)
-join = DummyOperator(task_id=trigger_rule, dag=dag,
trigger_rule=trigger_rule)
-final = DummyOperator(task_id='final_{}'.format(suffix), dag=dag)
+def create_test_pipeline(suffix, trigger_rule, dag_):
+"""
+Instantiate a number of operators for the given DAG.
Review comment:
Original content is rendered correctly:
https://user-images.githubusercontent.com/12058428/59151971-ed45f800-8a3b-11e9-9a0f-b46933c7d346.png;>
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] BasPH commented on a change in pull request #5361: [AIRFLOW-4670] Make airflow/example_dags Pylint compatible
BasPH commented on a change in pull request #5361: [AIRFLOW-4670] Make
airflow/example_dags Pylint compatible
URL: https://github.com/apache/airflow/pull/5361#discussion_r291818749
##
File path: airflow/example_dags/example_skip_dag.py
##
@@ -30,21 +32,31 @@
# Create some placeholder operators
class DummySkipOperator(DummyOperator):
+"""Dummy operator which always skips the task."""
+
ui_color = '#e8b7e4'
def execute(self, context):
raise AirflowSkipException
-def create_test_pipeline(suffix, trigger_rule, dag):
-skip_operator =
DummySkipOperator(task_id='skip_operator_{}'.format(suffix), dag=dag)
-always_true = DummyOperator(task_id='always_true_{}'.format(suffix),
dag=dag)
-join = DummyOperator(task_id=trigger_rule, dag=dag,
trigger_rule=trigger_rule)
-final = DummyOperator(task_id='final_{}'.format(suffix), dag=dag)
+def create_test_pipeline(suffix, trigger_rule, dag_):
+"""
+Instantiate a number of operators for the given DAG.
Review comment:
Ah my bad, I interpreted it wrong... (was thinking of whitespace after the
pydoc instead of description...)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #5302: [AIRFLOW-4374] Make enum-like-classes inherit from enum
codecov-io edited a comment on issue #5302: [AIRFLOW-4374] Make enum-like-classes inherit from enum URL: https://github.com/apache/airflow/pull/5302#issuecomment-493740734 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5302?src=pr=h1) Report > Merging [#5302](https://codecov.io/gh/apache/airflow/pull/5302?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/827d6d40c59642cf3eb79238e9a733479de4ea60?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `89.65%`. [](https://codecov.io/gh/apache/airflow/pull/5302?src=pr=tree) ```diff @@Coverage Diff @@ ## master#5302 +/- ## == - Coverage 79.03% 79.02% -0.01% == Files 481 481 Lines 3020130206 +5 == + Hits2386823869 +1 - Misses 6333 6337 +4 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5302?src=pr=tree) | Coverage Δ | | |---|---|---| | [...ow/contrib/example\_dags/example\_qubole\_operator.py](https://codecov.io/gh/apache/airflow/pull/5302/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3F1Ym9sZV9vcGVyYXRvci5weQ==) | `0% <0%> (ø)` | :arrow_up: | | [airflow/example\_dags/example\_skip\_dag.py](https://codecov.io/gh/apache/airflow/pull/5302/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9za2lwX2RhZy5weQ==) | `95.23% <100%> (+0.23%)` | :arrow_up: | | [airflow/utils/weight\_rule.py](https://codecov.io/gh/apache/airflow/pull/5302/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy93ZWlnaHRfcnVsZS5weQ==) | `100% <100%> (ø)` | :arrow_up: | | [airflow/example\_dags/example\_branch\_operator.py](https://codecov.io/gh/apache/airflow/pull/5302/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9icmFuY2hfb3BlcmF0b3IucHk=) | `100% <100%> (ø)` | :arrow_up: | | [airflow/utils/trigger\_rule.py](https://codecov.io/gh/apache/airflow/pull/5302/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy90cmlnZ2VyX3J1bGUucHk=) | `100% <100%> (ø)` | :arrow_up: | | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/5302/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `93.56% <84.61%> (-0.41%)` | :arrow_down: | | [airflow/contrib/operators/ssh\_operator.py](https://codecov.io/gh/apache/airflow/pull/5302/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zc2hfb3BlcmF0b3IucHk=) | `82.27% <0%> (-1.27%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5302?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5302?src=pr=footer). Last update [827d6d4...359e967](https://codecov.io/gh/apache/airflow/pull/5302?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #5361: [AIRFLOW-4670] Make airflow/example_dags Pylint compatible
codecov-io edited a comment on issue #5361: [AIRFLOW-4670] Make airflow/example_dags Pylint compatible URL: https://github.com/apache/airflow/pull/5361#issuecomment-500166678 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5361?src=pr=h1) Report > Merging [#5361](https://codecov.io/gh/apache/airflow/pull/5361?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/827d6d40c59642cf3eb79238e9a733479de4ea60?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `83.33%`. [](https://codecov.io/gh/apache/airflow/pull/5361?src=pr=tree) ```diff @@Coverage Diff @@ ## master#5361 +/- ## == - Coverage 79.03% 79.02% -0.01% == Files 481 481 Lines 3020130213 +12 == + Hits2386823877 +9 - Misses 6333 6336 +3 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5361?src=pr=tree) | Coverage Δ | | |---|---|---| | [...w/example\_dags/example\_latest\_only\_with\_trigger.py](https://codecov.io/gh/apache/airflow/pull/5361/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9sYXRlc3Rfb25seV93aXRoX3RyaWdnZXIucHk=) | `100% <ø> (ø)` | :arrow_up: | | [...low/example\_dags/example\_trigger\_controller\_dag.py](https://codecov.io/gh/apache/airflow/pull/5361/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV90cmlnZ2VyX2NvbnRyb2xsZXJfZGFnLnB5) | `56.25% <0%> (-3.75%)` | :arrow_down: | | [...low/example\_dags/example\_short\_circuit\_operator.py](https://codecov.io/gh/apache/airflow/pull/5361/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9zaG9ydF9jaXJjdWl0X29wZXJhdG9yLnB5) | `100% <100%> (ø)` | :arrow_up: | | [airflow/example\_dags/example\_bash\_operator.py](https://codecov.io/gh/apache/airflow/pull/5361/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9iYXNoX29wZXJhdG9yLnB5) | `94.44% <100%> (+0.32%)` | :arrow_up: | | [airflow/example\_dags/example\_pig\_operator.py](https://codecov.io/gh/apache/airflow/pull/5361/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9waWdfb3BlcmF0b3IucHk=) | `100% <100%> (ø)` | :arrow_up: | | [airflow/example\_dags/example\_python\_operator.py](https://codecov.io/gh/apache/airflow/pull/5361/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9weXRob25fb3BlcmF0b3IucHk=) | `94.44% <100%> (+0.32%)` | :arrow_up: | | [airflow/example\_dags/example\_skip\_dag.py](https://codecov.io/gh/apache/airflow/pull/5361/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9za2lwX2RhZy5weQ==) | `95.23% <100%> (+0.23%)` | :arrow_up: | | [...ample\_dags/example\_branch\_python\_dop\_operator\_3.py](https://codecov.io/gh/apache/airflow/pull/5361/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9icmFuY2hfcHl0aG9uX2RvcF9vcGVyYXRvcl8zLnB5) | `75% <100%> (+1.66%)` | :arrow_up: | | [airflow/example\_dags/example\_subdag\_operator.py](https://codecov.io/gh/apache/airflow/pull/5361/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9zdWJkYWdfb3BlcmF0b3IucHk=) | `100% <100%> (ø)` | :arrow_up: | | [airflow/example\_dags/subdags/subdag.py](https://codecov.io/gh/apache/airflow/pull/5361/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3Mvc3ViZGFncy9zdWJkYWcucHk=) | `100% <100%> (ø)` | :arrow_up: | | ... and [9 more](https://codecov.io/gh/apache/airflow/pull/5361/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5361?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5361?src=pr=footer). Last update [827d6d4...d305cfb](https://codecov.io/gh/apache/airflow/pull/5361?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #5361: [AIRFLOW-4670] Make airflow/example_dags Pylint compatible
codecov-io commented on issue #5361: [AIRFLOW-4670] Make airflow/example_dags Pylint compatible URL: https://github.com/apache/airflow/pull/5361#issuecomment-500166678 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5361?src=pr=h1) Report > Merging [#5361](https://codecov.io/gh/apache/airflow/pull/5361?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/827d6d40c59642cf3eb79238e9a733479de4ea60?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `83.33%`. [](https://codecov.io/gh/apache/airflow/pull/5361?src=pr=tree) ```diff @@Coverage Diff @@ ## master#5361 +/- ## == - Coverage 79.03% 79.02% -0.01% == Files 481 481 Lines 3020130213 +12 == + Hits2386823877 +9 - Misses 6333 6336 +3 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5361?src=pr=tree) | Coverage Δ | | |---|---|---| | [...w/example\_dags/example\_latest\_only\_with\_trigger.py](https://codecov.io/gh/apache/airflow/pull/5361/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9sYXRlc3Rfb25seV93aXRoX3RyaWdnZXIucHk=) | `100% <ø> (ø)` | :arrow_up: | | [...low/example\_dags/example\_trigger\_controller\_dag.py](https://codecov.io/gh/apache/airflow/pull/5361/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV90cmlnZ2VyX2NvbnRyb2xsZXJfZGFnLnB5) | `56.25% <0%> (-3.75%)` | :arrow_down: | | [...low/example\_dags/example\_short\_circuit\_operator.py](https://codecov.io/gh/apache/airflow/pull/5361/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9zaG9ydF9jaXJjdWl0X29wZXJhdG9yLnB5) | `100% <100%> (ø)` | :arrow_up: | | [airflow/example\_dags/example\_bash\_operator.py](https://codecov.io/gh/apache/airflow/pull/5361/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9iYXNoX29wZXJhdG9yLnB5) | `94.44% <100%> (+0.32%)` | :arrow_up: | | [airflow/example\_dags/example\_pig\_operator.py](https://codecov.io/gh/apache/airflow/pull/5361/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9waWdfb3BlcmF0b3IucHk=) | `100% <100%> (ø)` | :arrow_up: | | [airflow/example\_dags/example\_python\_operator.py](https://codecov.io/gh/apache/airflow/pull/5361/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9weXRob25fb3BlcmF0b3IucHk=) | `94.44% <100%> (+0.32%)` | :arrow_up: | | [airflow/example\_dags/example\_skip\_dag.py](https://codecov.io/gh/apache/airflow/pull/5361/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9za2lwX2RhZy5weQ==) | `95.23% <100%> (+0.23%)` | :arrow_up: | | [...ample\_dags/example\_branch\_python\_dop\_operator\_3.py](https://codecov.io/gh/apache/airflow/pull/5361/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9icmFuY2hfcHl0aG9uX2RvcF9vcGVyYXRvcl8zLnB5) | `75% <100%> (+1.66%)` | :arrow_up: | | [airflow/example\_dags/example\_subdag\_operator.py](https://codecov.io/gh/apache/airflow/pull/5361/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9zdWJkYWdfb3BlcmF0b3IucHk=) | `100% <100%> (ø)` | :arrow_up: | | [airflow/example\_dags/subdags/subdag.py](https://codecov.io/gh/apache/airflow/pull/5361/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3Mvc3ViZGFncy9zdWJkYWcucHk=) | `100% <100%> (ø)` | :arrow_up: | | ... and [9 more](https://codecov.io/gh/apache/airflow/pull/5361/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5361?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5361?src=pr=footer). Last update [827d6d4...d305cfb](https://codecov.io/gh/apache/airflow/pull/5361?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #5361: [AIRFLOW-4670] Make airflow/example_dags Pylint compatible
codecov-io edited a comment on issue #5361: [AIRFLOW-4670] Make airflow/example_dags Pylint compatible URL: https://github.com/apache/airflow/pull/5361#issuecomment-500166678 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5361?src=pr=h1) Report > Merging [#5361](https://codecov.io/gh/apache/airflow/pull/5361?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/827d6d40c59642cf3eb79238e9a733479de4ea60?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `83.33%`. [](https://codecov.io/gh/apache/airflow/pull/5361?src=pr=tree) ```diff @@Coverage Diff @@ ## master#5361 +/- ## == - Coverage 79.03% 79.02% -0.01% == Files 481 481 Lines 3020130213 +12 == + Hits2386823877 +9 - Misses 6333 6336 +3 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5361?src=pr=tree) | Coverage Δ | | |---|---|---| | [...w/example\_dags/example\_latest\_only\_with\_trigger.py](https://codecov.io/gh/apache/airflow/pull/5361/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9sYXRlc3Rfb25seV93aXRoX3RyaWdnZXIucHk=) | `100% <ø> (ø)` | :arrow_up: | | [...low/example\_dags/example\_trigger\_controller\_dag.py](https://codecov.io/gh/apache/airflow/pull/5361/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV90cmlnZ2VyX2NvbnRyb2xsZXJfZGFnLnB5) | `56.25% <0%> (-3.75%)` | :arrow_down: | | [...low/example\_dags/example\_short\_circuit\_operator.py](https://codecov.io/gh/apache/airflow/pull/5361/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9zaG9ydF9jaXJjdWl0X29wZXJhdG9yLnB5) | `100% <100%> (ø)` | :arrow_up: | | [airflow/example\_dags/example\_bash\_operator.py](https://codecov.io/gh/apache/airflow/pull/5361/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9iYXNoX29wZXJhdG9yLnB5) | `94.44% <100%> (+0.32%)` | :arrow_up: | | [airflow/example\_dags/example\_pig\_operator.py](https://codecov.io/gh/apache/airflow/pull/5361/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9waWdfb3BlcmF0b3IucHk=) | `100% <100%> (ø)` | :arrow_up: | | [airflow/example\_dags/example\_python\_operator.py](https://codecov.io/gh/apache/airflow/pull/5361/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9weXRob25fb3BlcmF0b3IucHk=) | `94.44% <100%> (+0.32%)` | :arrow_up: | | [airflow/example\_dags/example\_skip\_dag.py](https://codecov.io/gh/apache/airflow/pull/5361/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9za2lwX2RhZy5weQ==) | `95.23% <100%> (+0.23%)` | :arrow_up: | | [...ample\_dags/example\_branch\_python\_dop\_operator\_3.py](https://codecov.io/gh/apache/airflow/pull/5361/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9icmFuY2hfcHl0aG9uX2RvcF9vcGVyYXRvcl8zLnB5) | `75% <100%> (+1.66%)` | :arrow_up: | | [airflow/example\_dags/example\_subdag\_operator.py](https://codecov.io/gh/apache/airflow/pull/5361/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9zdWJkYWdfb3BlcmF0b3IucHk=) | `100% <100%> (ø)` | :arrow_up: | | [airflow/example\_dags/subdags/subdag.py](https://codecov.io/gh/apache/airflow/pull/5361/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3Mvc3ViZGFncy9zdWJkYWcucHk=) | `100% <100%> (ø)` | :arrow_up: | | ... and [9 more](https://codecov.io/gh/apache/airflow/pull/5361/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5361?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5361?src=pr=footer). Last update [827d6d4...d305cfb](https://codecov.io/gh/apache/airflow/pull/5361?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kurtqq commented on issue #5356: [AIRFLOW-4363] Fix Json encoding error when retrieving `status` from cli in docker operator
kurtqq commented on issue #5356: [AIRFLOW-4363] Fix Json encoding error when retrieving `status` from cli in docker operator URL: https://github.com/apache/airflow/pull/5356#issuecomment-500130501 I remembered seeing similar report on Mac & docker: https://github.com/docker/docker-py/issues/1059 So shouldn't the fix be setting `decode=True` when calling `APIClient` ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kurtqq commented on issue #5385: [AIRFLOW-4743] Add environment variables support to SSHOperator
kurtqq commented on issue #5385: [AIRFLOW-4743] Add environment variables support to SSHOperator URL: https://github.com/apache/airflow/pull/5385#issuecomment-500131106 You are probably gonna need to add/update the tests with the new parameter: https://github.com/apache/airflow/blob/master/tests/contrib/operators/test_ssh_operator.py This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ms32035 commented on issue #5385: [AIRFLOW-4743] Add environment variables support to SSHOperator
ms32035 commented on issue #5385: [AIRFLOW-4743] Add environment variables support to SSHOperator URL: https://github.com/apache/airflow/pull/5385#issuecomment-500138916 It defaults to `None`, so existing test should cover it, but added one test with a non default value. Whether these values are next handled correctly would be in scope of paramiko not Airflow testing, and also depends whether an ssh server accepts environment variables. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #5385: [AIRFLOW-4743] Add environment variables support to SSHOperator
codecov-io commented on issue #5385: [AIRFLOW-4743] Add environment variables support to SSHOperator URL: https://github.com/apache/airflow/pull/5385#issuecomment-500141643 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5385?src=pr=h1) Report > Merging [#5385](https://codecov.io/gh/apache/airflow/pull/5385?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/abb506ba5b92b0a1eb447dcf880c84f3710a1c88?src=pr=desc) will **increase** coverage by `0.26%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/5385?src=pr=tree) ```diff @@Coverage Diff @@ ## master#5385 +/- ## == + Coverage 78.75% 79.02% +0.26% == Files 481 481 Lines 3020130202 +1 == + Hits2378623868 +82 + Misses 6415 6334 -81 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5385?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/contrib/operators/ssh\_operator.py](https://codecov.io/gh/apache/airflow/pull/5385/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zc2hfb3BlcmF0b3IucHk=) | `82.5% <100%> (+0.22%)` | :arrow_up: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/5385/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `92.61% <0%> (+0.17%)` | :arrow_up: | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/5385/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `88.59% <0%> (+0.87%)` | :arrow_up: | | [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/5385/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `65% <0%> (+1.11%)` | :arrow_up: | | [airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/airflow/pull/5385/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5) | `77.72% <0%> (+1.77%)` | :arrow_up: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/5385/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `80.95% <0%> (+4.76%)` | :arrow_up: | | [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/5385/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `100% <0%> (+100%)` | :arrow_up: | | [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/5385/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `100% <0%> (+100%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5385?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5385?src=pr=footer). Last update [abb506b...0f41660](https://codecov.io/gh/apache/airflow/pull/5385?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kurtqq commented on issue #4520: [AIRFLOW-3703] Support DnsPolicy for KubernetesPodOperator
kurtqq commented on issue #4520: [AIRFLOW-3703] Support DnsPolicy for KubernetesPodOperator URL: https://github.com/apache/airflow/pull/4520#issuecomment-500145868 > > > > As a general comment, I think your new parameter should be called `dns_policy` instead of `dnspolicy` for consistency with the other parameters available. > > Probably, but that ship might have sailed already with `hostnetwork` and `configmaps`? (Though I do agree) It can be changed later for 2.0 What is blocking this PR from being merged? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] BasPH commented on a change in pull request #5363: [AIRFLOW-4725] Fix setup.py PEP440 & Sphinx-PyPI-upload dependency
BasPH commented on a change in pull request #5363: [AIRFLOW-4725] Fix setup.py PEP440 & Sphinx-PyPI-upload dependency URL: https://github.com/apache/airflow/pull/5363#discussion_r291816957 ## File path: setup.py ## @@ -171,7 +179,7 @@ def write_version(filename=os.path.join(*['airflow', doc = [ 'sphinx-argparse>=0.1.13', 'sphinx-autoapi>=0.7.1', -'Sphinx-PyPI-upload>=0.2.1', +'Sphinx-PyPI-upload3>=0.2.2', Review comment: Removed the dependency + related config in setup.cfg. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-4184) Add an AWS Athena Helper to insert into table
[
https://issues.apache.org/jira/browse/AIRFLOW-4184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16859350#comment-16859350
]
ASF GitHub Bot commented on AIRFLOW-4184:
-
bryanyang0528 commented on pull request #4996: [AIRFLOW-4184] Add an athena
helper to insert into table
URL: https://github.com/apache/airflow/pull/4996
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add an AWS Athena Helper to insert into table
> -
>
> Key: AIRFLOW-4184
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4184
> Project: Apache Airflow
> Issue Type: New Feature
>Reporter: Bryan Yang
>Assignee: Bryan Yang
>Priority: Major
>
> AWS Athena does not support {{inert into table}} clause now, but this
> function is really critical for ETL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] potiuk commented on issue #5361: [AIRFLOW-4670] Make airflow/example_dags Pylint compatible
potiuk commented on issue #5361: [AIRFLOW-4670] Make airflow/example_dags Pylint compatible URL: https://github.com/apache/airflow/pull/5361#issuecomment-500186555 General comment: @BasPH - maybe we should disable the "pointless-statement" pylint check. It seems like heavily polluting the example docs now and I don't find this particular problem is dangerous or often. I think the benefits out-weight the cost of polluted example docs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #5275: [AIRFLOW-4505] New tag for only PY3 packages
potiuk commented on issue #5275: [AIRFLOW-4505] New tag for only PY3 packages URL: https://github.com/apache/airflow/pull/5275#issuecomment-500185049 @raphaelauv What regression ? I cannot see any scenario where it could break things ? If you are using py2 it will continue working as it did before. It's just py3 will not install snakebite . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #5335: [AIRFLOW-4588] Add GoogleApiClientHook
potiuk commented on issue #5335: [AIRFLOW-4588] Add GoogleApiClientHook URL: https://github.com/apache/airflow/pull/5335#issuecomment-500186292 @feluelle - do you want to follow up on that ? My only concern with that one is that it introduces a new base GCP hook where we already have one. Maybe you could simply start with rewriting your [code](https://gist.github.com/feluelle/ff7d14617c983a3230d91b180d206ca2#file-google_api_integration_in_airflow-py-L51) to use the GCP API base hook (including the credentials retrieval that is already there) and then you could move some of the methods to the base hook (like the pagination). Then you could also contribute the whole Google API to S3 operator. It seems useful to have such generic solution. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-4725) Make setup.py run without errors/warnings
[ https://issues.apache.org/jira/browse/AIRFLOW-4725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16859366#comment-16859366 ] ASF subversion and git services commented on AIRFLOW-4725: -- Commit 4fa83c422318defa4e65bf209a189e542a989135 in airflow's branch refs/heads/v1-10-test from Bas Harenslak [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=4fa83c4 ] [AIRFLOW-4725] Fix setup.py PEP440 & Sphinx-PyPI-upload dependency (#5363) > Make setup.py run without errors/warnings > - > > Key: AIRFLOW-4725 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4725 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > Fix For: 2.0.0 > > > Currently running "python setup.py --help-commands" (on master) throws a > warning + error. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4725) Make setup.py run without errors/warnings
[ https://issues.apache.org/jira/browse/AIRFLOW-4725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16859364#comment-16859364 ] ASF subversion and git services commented on AIRFLOW-4725: -- Commit 18b27387f57d2fb215036bb79c91ae4fa4fe09f6 in airflow's branch refs/heads/master from Bas Harenslak [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=18b2738 ] [AIRFLOW-4725] Fix setup.py PEP440 & Sphinx-PyPI-upload dependency (#5363) > Make setup.py run without errors/warnings > - > > Key: AIRFLOW-4725 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4725 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > > Currently running "python setup.py --help-commands" (on master) throws a > warning + error. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4725) Make setup.py run without errors/warnings
[ https://issues.apache.org/jira/browse/AIRFLOW-4725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-4725. --- Resolution: Fixed Fix Version/s: 2.0.0 > Make setup.py run without errors/warnings > - > > Key: AIRFLOW-4725 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4725 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > Fix For: 2.0.0 > > > Currently running "python setup.py --help-commands" (on master) throws a > warning + error. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] milton0825 commented on a change in pull request #5386: AIRFLOW-4748 Print stacktrace for dagbag process exceptions
milton0825 commented on a change in pull request #5386: AIRFLOW-4748 Print
stacktrace for dagbag process exceptions
URL: https://github.com/apache/airflow/pull/5386#discussion_r291824346
##
File path: airflow/models/dagbag.py
##
@@ -233,7 +235,8 @@ def process_file(self, filepath, only_if_updated=True,
safe_mode=True):
mods.append(m)
except Exception as e:
self.log.exception("Failed to import: %s", filepath)
-self.import_errors[filepath] = str(e)
+self.import_errors[filepath] = 'ERROR MESSAGE [' +
str(e) + '] ' \
+ 'STACK TRACE [' +
traceback.format_exc() + ']'
Review comment:
```suggestion
+ 'STACK TRACE [' +
traceback.format_exc() + ']'
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] milton0825 commented on a change in pull request #5386: AIRFLOW-4748 Print stacktrace for dagbag process exceptions
milton0825 commented on a change in pull request #5386: AIRFLOW-4748 Print
stacktrace for dagbag process exceptions
URL: https://github.com/apache/airflow/pull/5386#discussion_r291824342
##
File path: airflow/models/dagbag.py
##
@@ -198,7 +199,8 @@ def process_file(self, filepath, only_if_updated=True,
safe_mode=True):
mods.append(m)
except Exception as e:
self.log.exception("Failed to import: %s", filepath)
-self.import_errors[filepath] = str(e)
+self.import_errors[filepath] = 'ERROR MESSAGE [' + str(e)
+ '] ' \
+ 'STACK TRACE [' +
traceback.format_exc() + ']'
Review comment:
```suggestion
+ 'STACK TRACE [' +
traceback.format_exc() + ']'
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (AIRFLOW-4725) Make setup.py run without errors/warnings
[ https://issues.apache.org/jira/browse/AIRFLOW-4725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-4725: -- Fix Version/s: 1.10.4 > Make setup.py run without errors/warnings > - > > Key: AIRFLOW-4725 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4725 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > Fix For: 1.10.4, 2.0.0 > > > Currently running "python setup.py --help-commands" (on master) throws a > warning + error. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] potiuk commented on a change in pull request #5362: [AIRFLOW-4669] Make airflow/dag Pylint compatible
potiuk commented on a change in pull request #5362: [AIRFLOW-4669] Make airflow/dag Pylint compatible URL: https://github.com/apache/airflow/pull/5362#discussion_r291827624 ## File path: airflow/dag/base_dag.py ## @@ -17,6 +17,8 @@ # specific language governing permissions and limitations # under the License. +"""Base classes for DAG and DagBag.""" Review comment: I am good with that - one liner is ok. PEP is always right. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #5362: [AIRFLOW-4669] Make airflow/dag Pylint compatible
potiuk commented on a change in pull request #5362: [AIRFLOW-4669] Make airflow/dag Pylint compatible URL: https://github.com/apache/airflow/pull/5362#discussion_r291827624 ## File path: airflow/dag/base_dag.py ## @@ -17,6 +17,8 @@ # specific language governing permissions and limitations # under the License. +"""Base classes for DAG and DagBag.""" Review comment: I am good with that. PEP is always right. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-4725) Make setup.py run without errors/warnings
[ https://issues.apache.org/jira/browse/AIRFLOW-4725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16859363#comment-16859363 ] ASF GitHub Bot commented on AIRFLOW-4725: - potiuk commented on pull request #5363: [AIRFLOW-4725] Fix setup.py PEP440 & Sphinx-PyPI-upload dependency URL: https://github.com/apache/airflow/pull/5363 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Make setup.py run without errors/warnings > - > > Key: AIRFLOW-4725 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4725 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > > Currently running "python setup.py --help-commands" (on master) throws a > warning + error. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] potiuk merged pull request #5363: [AIRFLOW-4725] Fix setup.py PEP440 & Sphinx-PyPI-upload dependency
potiuk merged pull request #5363: [AIRFLOW-4725] Fix setup.py PEP440 & Sphinx-PyPI-upload dependency URL: https://github.com/apache/airflow/pull/5363 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] bryanyang0528 commented on a change in pull request #4996: [AIRFLOW-4184] Add an athena helper to insert into table
bryanyang0528 commented on a change in pull request #4996: [AIRFLOW-4184] Add an athena helper to insert into table URL: https://github.com/apache/airflow/pull/4996#discussion_r291825956 ## File path: airflow/contrib/hooks/aws_athena_hook.py ## @@ -148,3 +156,105 @@ def stop_query(self, query_execution_id): :return: dict """ return self.conn.stop_query_execution(QueryExecutionId=query_execution_id) + + +class AWSAthenaHelpers(AWSAthenaHook): +""" +The Athena Helpers contains helper methods to execute queries against +Athena. The methods can be used directly by operators. +""" + +def __init__(self, aws_conn_id='aws_default', region_name=None, *args, **kwargs): +super(AWSAthenaHelpers, self).__init__( +aws_conn_id=aws_conn_id, region_name=region_name, **kwargs) +self.region_name = region_name +self.s3_hook = None +self.glue_hook = None + +def get_s3_hook(self): +""" +check if s3 hook exists already or create one and return it +:return: s3 hook +""" +if not self.s3_hook: +self.s3_hook = S3Hook( +aws_conn_id=self.aws_conn_id, verify=self.verify) +return self.s3_hook + +def get_glue_hook(self): +""" +check if glue hook exists already or create one and return it +:return: glue hook +""" +if not self.glue_hook: +self.glue_hook = AwsGlueCatalogHook( +aws_conn_id=self.aws_conn_id, region_name=self.region_name) +return self.glue_hook + +def run_insert_into_table(self, src_db, src_table, dst_db, dst_table, mode='error'): +""" +insert data in s3 from the source table to the destination table Review comment: Thank you for the suggestion. I will close this issue and create a new one. I know EMR could do that thing, but it takes a long time to create an EMR cluster and compute on the EMR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] bryanyang0528 closed pull request #4996: [AIRFLOW-4184] Add an athena helper to insert into table
bryanyang0528 closed pull request #4996: [AIRFLOW-4184] Add an athena helper to insert into table URL: https://github.com/apache/airflow/pull/4996 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dossett commented on issue #5386: AIRFLOW-4748 Print stacktrace for dagbag process exceptions
dossett commented on issue #5386: AIRFLOW-4748 Print stacktrace for dagbag process exceptions URL: https://github.com/apache/airflow/pull/5386#issuecomment-500177728 One of the test suites passed, some others failed, I can't tell if the failures are connected to this change or not. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
