[GitHub] [airflow] feluelle edited a comment on issue #4810: [AIRFLOW-3985] Added unary + operator for easier linear DAG building
feluelle edited a comment on issue #4810: [AIRFLOW-3985] Added unary + operator for easier linear DAG building URL: https://github.com/apache/airflow/pull/4810#issuecomment-469642996 The `+` operator for this kind of dependency chaining looks very strange to me. Otherwise I think your idea is great. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3795) provide_context is not a passable parameter for PythonVirtualenvOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-3795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16785386#comment-16785386 ] Sergio Soto commented on AIRFLOW-3795: -- This bug is related with other > provide_context is not a passable parameter for PythonVirtualenvOperator > > > Key: AIRFLOW-3795 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3795 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Reporter: Susannah Doss >Assignee: Sergio Soto Núñez >Priority: Trivial > Fix For: 1.10.3 > > > `PythonVirtualenvOperator` does not allow me to specify > `provide_context=True`: > https://github.com/apache/airflow/blob/83cb9c3acdd3b4eeadf1cab3cb45d644c3e9ede0/airflow/operators/python_operator.py#L242 > However, I am able to do so when I use the plain `PythonOperator`. I can't > see a reason why I wouldn't be allowed to have it be set to `True` when using > a `PythonVirtualenvOperator`. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4024) Improve local client tests
Peter van 't Hof created AIRFLOW-4024: - Summary: Improve local client tests Key: AIRFLOW-4024 URL: https://issues.apache.org/jira/browse/AIRFLOW-4024 Project: Apache Airflow Issue Type: Improvement Reporter: Peter van 't Hof Assignee: Peter van 't Hof -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] ffinfo opened a new pull request #4850: [AIRFLOW-???] Improve local client tests
ffinfo opened a new pull request #4850: [AIRFLOW-???] Improve local client tests
URL: https://github.com/apache/airflow/pull/4850
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title.
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
### Description
Make tests more stateless
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feng-tao commented on issue #4849: [AIRFLOW-XXX] Avoid spamming the log from Airflow security manager
feng-tao commented on issue #4849: [AIRFLOW-XXX] Avoid spamming the log from Airflow security manager URL: https://github.com/apache/airflow/pull/4849#issuecomment-469987974 cc @ashb this is part of today's pr discussion. Let me know if I need to create a jira. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feng-tao opened a new pull request #4849: [AIRFLOW-XXX] Avoid spamming the log from Airflow security manager
feng-tao opened a new pull request #4849: [AIRFLOW-XXX] Avoid spamming the log
from Airflow security manager
URL: https://github.com/apache/airflow/pull/4849
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement
Proposal([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
Avoid spamming the log as these logs are used when developing the features.
Now it will log by every gunicorn worker per every starts up.
Change it to debug level.
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-3948) Operator list as templated field issue
[ https://issues.apache.org/jira/browse/AIRFLOW-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16785271#comment-16785271 ] ASF GitHub Bot commented on AIRFLOW-3948: - ryanyuan commented on pull request #4770: [AIRFLOW-3948] GCSToBQOperator list as templating field issue URL: https://github.com/apache/airflow/pull/4770 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Operator list as templated field issue > -- > > Key: AIRFLOW-3948 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3948 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Ryan Yuan >Assignee: Ryan Yuan >Priority: Major > > Fixing issue with operators' list as templated field issue > > For example, with GoogleCloudStorageToBigQueryOperator's templated fields, > `source_objects` and `schema_fields`, if parsed in values are lists, they > will be recognized as strings, causing error returned by GCS. > e.g. > Variable before templating, source_objects = ['file_1.csv', 'file_2.csv'] > Variable after templating, source_objects = "['file_1.csv', 'file_2.csv']" > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] ryanyuan closed pull request #4770: [AIRFLOW-3948] GCSToBQOperator list as templating field issue
ryanyuan closed pull request #4770: [AIRFLOW-3948] GCSToBQOperator list as templating field issue URL: https://github.com/apache/airflow/pull/4770 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3948) Operator list as templated field issue
[
https://issues.apache.org/jira/browse/AIRFLOW-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16785267#comment-16785267
]
ASF GitHub Bot commented on AIRFLOW-3948:
-
ryanyuan commented on pull request #4848: [AIRFLOW-3948] Operator list as
templated field issue
URL: https://github.com/apache/airflow/pull/4848
Operator list as templated field
This is a global fix for #4770
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement
Proposal([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Operator list as templated field issue
> --
>
> Key: AIRFLOW-3948
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3948
> Project: Apache Airflow
> Issue Type: Improvement
>Reporter: Ryan Yuan
>Assignee: Ryan Yuan
>Priority: Major
>
> Fixing issue with operators' list as templated field issue
>
> For example, with GoogleCloudStorageToBigQueryOperator's templated fields,
> `source_objects` and `schema_fields`, if parsed in values are lists, they
> will be recognized as strings, causing error returned by GCS.
> e.g.
> Variable before templating, source_objects = ['file_1.csv', 'file_2.csv']
> Variable after templating, source_objects = "['file_1.csv', 'file_2.csv']"
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] ryanyuan opened a new pull request #4848: [AIRFLOW-3948] Operator list as templated field issue
ryanyuan opened a new pull request #4848: [AIRFLOW-3948] Operator list as
templated field issue
URL: https://github.com/apache/airflow/pull/4848
Operator list as templated field
This is a global fix for #4770
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement
Proposal([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] XD-DENG commented on a change in pull request #4828: [AIRFLOW-4002] Optionally open debugger on errors in airflow test.
XD-DENG commented on a change in pull request #4828: [AIRFLOW-4002] Optionally
open debugger on errors in airflow test.
URL: https://github.com/apache/airflow/pull/4828#discussion_r262805339
##
File path: airflow/bin/cli.py
##
@@ -2032,6 +2047,11 @@ class CLIFactory(object):
'task_params': Arg(
("-tp", "--task_params"),
help="Sends a JSON params dict to the task"),
+'post_mortem': Arg(
+("-pm", "--post_mortem"),
+action="store_true",
Review comment:
Yes you're right. Thanks for clarification.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] XD-DENG commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details
XD-DENG commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details URL: https://github.com/apache/airflow/pull/4569#issuecomment-469983947 @feng-tao sure. Maybe let me create a ticket for this later maybe. Will tag you then for our reference. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-3948) Operator list as templated field issue
[ https://issues.apache.org/jira/browse/AIRFLOW-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Yuan updated AIRFLOW-3948: --- Description: Fixing issue with operators' list as templated field issue For example, with GoogleCloudStorageToBigQueryOperator's templated fields, `source_objects` and `schema_fields`, if parsed in values are lists, they will be recognized as strings, causing error returned by GCS. e.g. Variable before templating, source_objects = ['file_1.csv', 'file_2.csv'] Variable after templating, source_objects = "['file_1.csv', 'file_2.csv']" was: With GoogleCloudStorageToBigQueryOperator's templating fields, `source_objects` and `schema_fields`, if parsed in values are lists, they will be recognized as strings, causing error returned by GCS. e.g. Variable before templating, source_objects = ['file_1.csv', 'file_2.csv'] Variable after templating, source_objects = "['file_1.csv', 'file_2.csv']" Summary: Operator list as templated field issue (was: GCSToBQOperator list as templating field issue) > Operator list as templated field issue > -- > > Key: AIRFLOW-3948 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3948 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Ryan Yuan >Assignee: Ryan Yuan >Priority: Major > > Fixing issue with operators' list as templated field issue > > For example, with GoogleCloudStorageToBigQueryOperator's templated fields, > `source_objects` and `schema_fields`, if parsed in values are lists, they > will be recognized as strings, causing error returned by GCS. > e.g. > Variable before templating, source_objects = ['file_1.csv', 'file_2.csv'] > Variable after templating, source_objects = "['file_1.csv', 'file_2.csv']" > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] sasso commented on issue #4574: [AIRFLOW-3751] Ignore malformed ldap schema option
sasso commented on issue #4574: [AIRFLOW-3751] Ignore malformed ldap schema option URL: https://github.com/apache/airflow/pull/4574#issuecomment-469983575 :pray: This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feng-tao edited a comment on issue #4642: [AIRFLOW-2694] Declare permissions in DAG definition
feng-tao edited a comment on issue #4642: [AIRFLOW-2694] Declare permissions in DAG definition URL: https://github.com/apache/airflow/pull/4642#issuecomment-469981324 There are so many different discussions today. Here is another long list(https://github.com/apache/airflow/pull/4569). @astahlman , ideally we should, and that is part of original plan / discussion for DAG level access(https://github.com/apache/airflow/pull/3197). We(Me, Max, Joy) discussed multiple approaches for scheduler auto updated within that pr: 1. moved all the FAB models inside Airflow model, which scheduler is aware of the FAB models. And scheduler could directly update the tables in this case. Not a very ideal approach as it will fail once FAB models get updated which requires Airflow updated the model file as well. But ideally it should work with this approach. 2. RBAC is strictly a web construct(Max B's term) which means scheduler is unaware of roles & users, maybe as the supervisor it periodically calls a REST endpoint to trigger creating new perms from new DAGs. This approach has latency / performance concern. We(that pr) end up with create a cli for periodically sync_perm. Not sure how @ashb's approach would work by adding two columns in DAG table given there are no Airflow models defined what are "roles"(only exist in FAB models). The role column I assume should be some FK to the ab_roles table. Not sure what I described makes sense to you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feng-tao edited a comment on issue #4642: [AIRFLOW-2694] Declare permissions in DAG definition
feng-tao edited a comment on issue #4642: [AIRFLOW-2694] Declare permissions in DAG definition URL: https://github.com/apache/airflow/pull/4642#issuecomment-469981324 There are so many different discussions today. Here is another long list(https://github.com/apache/airflow/pull/4569). @astahlman , ideally we should, and that is part of original plan / discussion for DAG level access(https://github.com/apache/airflow/pull/3197). We(Me, Max, Joy) discussed multiple approaches for scheduler auto updated within that pr: 1. moved all the FAB models inside Airflow model, which scheduler is aware of the FAB models. And scheduler could directly update the tables in this case. 2. RBAC is strictly a web construct(Max B's term) which means scheduler is unaware of roles & users, maybe as the supervisor it periodically calls a REST endpoint to trigger creating new perms from new DAGs. This approach has latency / performance concern. We(that pr) end up with create a cli for periodically sync_perm. Not sure how @ashb's approach would work by adding two columns in DAG table given there are no Airflow models defined what are "roles"(only exist in FAB models). The role column I assume should be some FK to the ab_roles table. Not sure what I described makes sense to you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feng-tao commented on issue #4642: [AIRFLOW-2694] Declare permissions in DAG definition
feng-tao commented on issue #4642: [AIRFLOW-2694] Declare permissions in DAG definition URL: https://github.com/apache/airflow/pull/4642#issuecomment-469981324 There are so many different discussions today. Here is another long list(https://github.com/apache/airflow/pull/4569). @astahlman , ideally we should, and that is part of original plan / discussion for DAG level access(https://github.com/apache/airflow/pull/3197). We(Me, Max, Joy) discussed multiple approaches for scheduler auto updated inside the pr: 1. moved all the FAB models inside Airflow model, which scheduler is aware of the FAB models. And scheduler could directly update the tables in this case. 2. RBAC is strictly a web construct(Max B's term) which means scheduler is unaware of roles & users, maybe as the supervisor it periodically calls a REST endpoint to trigger creating new perms from new DAGs. This approach has latency / performance concern. We(that pr) end up with create a cli for periodically sync_perm. Not sure how @ashb's approach would work by adding two columns in DAG table given there are no Airflow models defined what are "roles"(only exist in FAB models). The role column I assume should be some FK to the ab_roles table. Not sure what I described makes sense to you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feng-tao commented on issue #4685: [AIRFLOW-3862] Check types with mypy.
feng-tao commented on issue #4685: [AIRFLOW-3862] Check types with mypy. URL: https://github.com/apache/airflow/pull/4685#issuecomment-469979194 side question, I thought mypy doesn't work with python 2.7? Or we won't do mypy when it is in py2.7? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] jmcarp commented on a change in pull request #4685: [AIRFLOW-3862] Check types with mypy.
jmcarp commented on a change in pull request #4685: [AIRFLOW-3862] Check types
with mypy.
URL: https://github.com/apache/airflow/pull/4685#discussion_r262799908
##
File path: airflow/contrib/operators/adls_list_operator.py
##
@@ -46,7 +48,7 @@ class AzureDataLakeStorageListOperator(BaseOperator):
azure_data_lake_conn_id='azure_data_lake_default'
)
"""
-template_fields = ('path',)
+template_fields = ('path',) # type: Sequence[str]
Review comment:
We don't have to add type annotations for every operator--we have to add
annotations on operators that are later subclassed. Changing the annotation to
`Iterable[str]` doesn't change this. The issue is that a subclass defining
`template_fields` as e.g. `("path", )` overrides its type from `Sequence[str]`
to `Tuple[str]`; when the subclass of the subclass overrides the value to e.g.
`("src_adls", "dest_gcs")`, its new type of `Tuple[str, str]` is incompatible
with `Tuple[str]`. We could also set the types in `BaseOperator` to be
`List[str]` like we talked about earlier, but that would mean changing all
operators to use lists, which I think we agree we shouldn't do.
Here's a simplified example, using python3-style annotations for convenience:
```python
from typing import Iterable
class A:
a: Iterable[str] = ("a", )
class B(A):
a = ("a", "b")
class C(B):
a = ("a", "b", "c")
```
This isn't valid, because the class variable gets typed as `Tuple[str, str]`
in `B` but `Tuple[str, str, str]` in `C`:
```
error: Incompatible types in assignment (expression has type "Tuple[str,
str, str]", base class "B" defined the type as "Tuple[str, str]")
```
If we annotate `B.a` with `Iterable[str]`, then mypy passes.
Altogether, the annotations we have so far are the simplest solution that I
can think of without a larger refactor: we only have to annotate variables in a
handful of classes (`BaseOperator` and other operators that are later
subclassed), and we don't have to change tuples to lists for all operators.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feng-tao commented on issue #4712: [AIRFLOW-3892] Create Redis pub sub sensor
feng-tao commented on issue #4712: [AIRFLOW-3892] Create Redis pub sub sensor URL: https://github.com/apache/airflow/pull/4712#issuecomment-469969811 @mans2singh , I think we are fine. But could we remove that example dag as I don't think we need that given the sensor is not complicated? Thanks. Once you remove, I think we are good to go. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mans2singh commented on issue #4712: [AIRFLOW-3892] Create Redis pub sub sensor
mans2singh commented on issue #4712: [AIRFLOW-3892] Create Redis pub sub sensor URL: https://github.com/apache/airflow/pull/4712#issuecomment-469967564 @zhongjiajie - I kicked off another build and it is [passing](https://travis-ci.org/mans2singh/airflow/builds/502373240) but this page still shows failed build. Please let me know what I could be missing. Thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (AIRFLOW-4023) Fix DataProcOperator templating field names
[ https://issues.apache.org/jira/browse/AIRFLOW-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Yuan closed AIRFLOW-4023. -- Resolution: Not A Problem > Fix DataProcOperator templating field names > --- > > Key: AIRFLOW-4023 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4023 > Project: Apache Airflow > Issue Type: Bug >Affects Versions: 1.10.2 >Reporter: Ryan Yuan >Assignee: Ryan Yuan >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] jmcarp commented on a change in pull request #4828: [AIRFLOW-4002] Optionally open debugger on errors in airflow test.
jmcarp commented on a change in pull request #4828: [AIRFLOW-4002] Optionally
open debugger on errors in airflow test.
URL: https://github.com/apache/airflow/pull/4828#discussion_r262790267
##
File path: airflow/bin/cli.py
##
@@ -2032,6 +2047,11 @@ class CLIFactory(object):
'task_params': Arg(
("-tp", "--task_params"),
help="Sends a JSON params dict to the task"),
+'post_mortem': Arg(
+("-pm", "--post_mortem"),
+action="store_true",
Review comment:
In argparse, store_true sets the default to false, so I think this is
correct.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] jmcarp commented on a change in pull request #4828: [AIRFLOW-4002] Optionally open debugger on errors in airflow test.
jmcarp commented on a change in pull request #4828: [AIRFLOW-4002] Optionally
open debugger on errors in airflow test.
URL: https://github.com/apache/airflow/pull/4828#discussion_r262790183
##
File path: airflow/bin/cli.py
##
@@ -686,10 +687,24 @@ def test(args, dag=None):
task.params.update(passed_in_params)
ti = TaskInstance(task, args.execution_date)
-if args.dry_run:
-ti.dry_run()
-else:
-ti.run(ignore_task_deps=True, ignore_ti_state=True, test_mode=True)
+try:
+if args.dry_run:
+ti.dry_run()
+else:
+ti.run(ignore_task_deps=True, ignore_ti_state=True, test_mode=True)
+except Exception:
+if args.post_mortem:
+debugger = get_debugger()
+debugger.post_mortem()
+else:
+raise
+
+
+def get_debugger():
+try:
+return importlib.import_module("ipdb")
+except ImportError:
+return importlib.import_module("pdb")
Review comment:
Agreed, I factored this back into the `test` function.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ryanyuan closed pull request #4847: [AIRFLOW-4023] Fix DataProcOperator templating field names
ryanyuan closed pull request #4847: [AIRFLOW-4023] Fix DataProcOperator templating field names URL: https://github.com/apache/airflow/pull/4847 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-4023) Fix DataProcOperator templating field names
[ https://issues.apache.org/jira/browse/AIRFLOW-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16785216#comment-16785216 ] ASF GitHub Bot commented on AIRFLOW-4023: - ryanyuan commented on pull request #4847: [AIRFLOW-4023] Fix DataProcOperator templating field names URL: https://github.com/apache/airflow/pull/4847 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Fix DataProcOperator templating field names > --- > > Key: AIRFLOW-4023 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4023 > Project: Apache Airflow > Issue Type: Bug >Affects Versions: 1.10.2 >Reporter: Ryan Yuan >Assignee: Ryan Yuan >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] XD-DENG commented on a change in pull request #4828: [AIRFLOW-4002] Optionally open debugger on errors in airflow test.
XD-DENG commented on a change in pull request #4828: [AIRFLOW-4002] Optionally
open debugger on errors in airflow test.
URL: https://github.com/apache/airflow/pull/4828#discussion_r262787268
##
File path: airflow/bin/cli.py
##
@@ -2032,6 +2047,11 @@ class CLIFactory(object):
'task_params': Arg(
("-tp", "--task_params"),
help="Sends a JSON params dict to the task"),
+'post_mortem': Arg(
+("-pm", "--post_mortem"),
+action="store_true",
Review comment:
Seems the default value here would be `True`, i.e. the post_mortem will be
invoked by default. But this is different from the previous behavior.
Personally I would suggest to use `False` as default value here, to make
behavior consistent.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] XD-DENG commented on a change in pull request #4828: [AIRFLOW-4002] Optionally open debugger on errors in airflow test.
XD-DENG commented on a change in pull request #4828: [AIRFLOW-4002] Optionally
open debugger on errors in airflow test.
URL: https://github.com/apache/airflow/pull/4828#discussion_r262787613
##
File path: airflow/bin/cli.py
##
@@ -686,10 +687,24 @@ def test(args, dag=None):
task.params.update(passed_in_params)
ti = TaskInstance(task, args.execution_date)
-if args.dry_run:
-ti.dry_run()
-else:
-ti.run(ignore_task_deps=True, ignore_ti_state=True, test_mode=True)
+try:
+if args.dry_run:
+ti.dry_run()
+else:
+ti.run(ignore_task_deps=True, ignore_ti_state=True, test_mode=True)
+except Exception:
+if args.post_mortem:
+debugger = get_debugger()
+debugger.post_mortem()
+else:
+raise
+
+
+def get_debugger():
+try:
+return importlib.import_module("ipdb")
+except ImportError:
+return importlib.import_module("pdb")
Review comment:
Do we need to have a separate function here?
I understand this is very "personal preference": but I think we can simply
put this part after `if args.post_mortem:` in the `try-except` block above,
given the logic inside this `get_debugger()` function is too simple.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-4023) Fix DataProcOperator templating field names
[
https://issues.apache.org/jira/browse/AIRFLOW-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16785188#comment-16785188
]
ASF GitHub Bot commented on AIRFLOW-4023:
-
ryanyuan commented on pull request #4847: [AIRFLOW-4023] Fix DataProcOperator
templating field names
URL: https://github.com/apache/airflow/pull/4847
Fix DataProcOperator templating field names
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following
[Airflow-4023](https://issues.apache.org/jira/browse/AIRFLOW-4023) issues and
references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement
Proposal([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Fix DataProcOperator templating field names
> ---
>
> Key: AIRFLOW-4023
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4023
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.10.2
>Reporter: Ryan Yuan
>Assignee: Ryan Yuan
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (AIRFLOW-4023) Fix DataProcOperator templating field names
[ https://issues.apache.org/jira/browse/AIRFLOW-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-4023 started by Ryan Yuan. -- > Fix DataProcOperator templating field names > --- > > Key: AIRFLOW-4023 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4023 > Project: Apache Airflow > Issue Type: Bug >Affects Versions: 1.10.2 >Reporter: Ryan Yuan >Assignee: Ryan Yuan >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] ryanyuan opened a new pull request #4847: [AIRFLOW-4023] Fix DataProcOperator templating field names
ryanyuan opened a new pull request #4847: [AIRFLOW-4023] Fix DataProcOperator
templating field names
URL: https://github.com/apache/airflow/pull/4847
Fix DataProcOperator templating field names
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following
[Airflow-4023](https://issues.apache.org/jira/browse/AIRFLOW-4023) issues and
references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement
Proposal([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-4023) Fix DataProcOperator templating field names
Ryan Yuan created AIRFLOW-4023: -- Summary: Fix DataProcOperator templating field names Key: AIRFLOW-4023 URL: https://issues.apache.org/jira/browse/AIRFLOW-4023 Project: Apache Airflow Issue Type: Bug Affects Versions: 1.10.2 Reporter: Ryan Yuan Assignee: Ryan Yuan -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] jmcarp commented on issue #4828: [AIRFLOW-4002] Optionally open debugger on errors in airflow test.
jmcarp commented on issue #4828: [AIRFLOW-4002] Optionally open debugger on errors in airflow test. URL: https://github.com/apache/airflow/pull/4828#issuecomment-469947705 Added `ipdb` detection. Ready for review when you have time @ashb @XD-DENG @feng-tao. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #4828: [AIRFLOW-4002] Optionally open debugger on errors in airflow test.
codecov-io edited a comment on issue #4828: [AIRFLOW-4002] Optionally open debugger on errors in airflow test. URL: https://github.com/apache/airflow/pull/4828#issuecomment-469055338 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4828?src=pr=h1) Report > Merging [#4828](https://codecov.io/gh/apache/airflow/pull/4828?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/dda309e66224346014ca42a0bcce4f53365f1118?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `40%`. [](https://codecov.io/gh/apache/airflow/pull/4828?src=pr=tree) ```diff @@Coverage Diff@@ ## master #4828 +/- ## = - Coverage 74.71% 74.7% -0.01% = Files 449 449 Lines 28961 28973 +12 = + Hits21638 21645 +7 - Misses 73237328 +5 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4828?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/4828/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `66.7% <40%> (-0.53%)` | :arrow_down: | | [airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/4828/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=) | `92.6% <0%> (-0.06%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/4828/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `93.43% <0%> (+0.36%)` | :arrow_up: | | [airflow/contrib/kubernetes/kube\_client.py](https://codecov.io/gh/apache/airflow/pull/4828/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2t1YmVybmV0ZXMva3ViZV9jbGllbnQucHk=) | `75% <0%> (+16.66%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4828?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4828?src=pr=footer). Last update [dda309e...a6404dd](https://codecov.io/gh/apache/airflow/pull/4828?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #4828: [AIRFLOW-4002] Optionally open debugger on errors in airflow test.
codecov-io edited a comment on issue #4828: [AIRFLOW-4002] Optionally open debugger on errors in airflow test. URL: https://github.com/apache/airflow/pull/4828#issuecomment-469055338 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4828?src=pr=h1) Report > Merging [#4828](https://codecov.io/gh/apache/airflow/pull/4828?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/dda309e66224346014ca42a0bcce4f53365f1118?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `40%`. [](https://codecov.io/gh/apache/airflow/pull/4828?src=pr=tree) ```diff @@Coverage Diff@@ ## master #4828 +/- ## = - Coverage 74.71% 74.7% -0.01% = Files 449 449 Lines 28961 28973 +12 = + Hits21638 21645 +7 - Misses 73237328 +5 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4828?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/4828/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `66.7% <40%> (-0.53%)` | :arrow_down: | | [airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/4828/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=) | `92.6% <0%> (-0.06%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/4828/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `93.43% <0%> (+0.36%)` | :arrow_up: | | [airflow/contrib/kubernetes/kube\_client.py](https://codecov.io/gh/apache/airflow/pull/4828/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2t1YmVybmV0ZXMva3ViZV9jbGllbnQucHk=) | `75% <0%> (+16.66%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4828?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4828?src=pr=footer). Last update [dda309e...a6404dd](https://codecov.io/gh/apache/airflow/pull/4828?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] aoen commented on issue #4396: [AIRFLOW-3585] - Add edges to database
aoen commented on issue #4396: [AIRFLOW-3585] - Add edges to database URL: https://github.com/apache/airflow/pull/4396#issuecomment-469942061 Sorry Ash, missed your reply. Kevin is correct, we should extend SimpleDag to contain edge information and store the serialized representation of this in the DB. My feeling is we should also store entire serialized representations of DAGs (e.g. JSON5 or have some discussion to agree on format) rather than as columns in the DB, both for performance reasons to avoid n+1 queries when we use the SimpleDAG representations for the Scheduler and Webserver, unless there is a good reason not to do this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] XD-DENG edited a comment on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details
XD-DENG edited a comment on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details URL: https://github.com/apache/airflow/pull/4569#issuecomment-469920279 > @XD-DENG , I don't think it will add 2000 entries to the db for admin role. It only adds the "all_dags" view-menu with the 2 permissions into the table. Hi @feng-tao please refer to https://github.com/apache/airflow/blob/dda309e66224346014ca42a0bcce4f53365f1118/airflow/www/security.py#L440-L453 By checking the code itself, seems it's simply adding all whatever permissions to Admin role. Below is the version before my `set`-related refactoring: https://github.com/apache/airflow/blob/c552996483f5aab5e31793317b26aba62c13332e/airflow/www/security.py#L440-L452 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] jyothsnapk17 commented on issue #4835: [AIRFLOW-XXXX] Improvements to formatted content in documentation
jyothsnapk17 commented on issue #4835: [AIRFLOW-] Improvements to formatted content in documentation URL: https://github.com/apache/airflow/pull/4835#issuecomment-469925927 @mik-laj , The build stalled on the Kubernetes tests. Does it necessarily have to be fixed with this PR? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] zhongjiajie commented on issue #4712: [AIRFLOW-3892] Create Redis pub sub sensor
zhongjiajie commented on issue #4712: [AIRFLOW-3892] Create Redis pub sub sensor URL: https://github.com/apache/airflow/pull/4712#issuecomment-469923984 @mans2singh The CI currently flaky, If you could not find detail error, maybe you could submit and restart CI test again. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mans2singh commented on issue #4712: [AIRFLOW-3892] Create Redis pub sub sensor
mans2singh commented on issue #4712: [AIRFLOW-3892] Create Redis pub sub sensor URL: https://github.com/apache/airflow/pull/4712#issuecomment-469920927 Hi @feng-tao - I've rebased the code and the integration unit/tests for redis sensor are passing locally. It looks like [two kubernetes tests are failing for python 3.5](https://travis-ci.org/apache/airflow/jobs/502318561) on travis-ci. Can you please let me know how to resolve this and if there is anything else is required ? Thanks for your advice and help. Mans This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] XD-DENG edited a comment on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details
XD-DENG edited a comment on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details URL: https://github.com/apache/airflow/pull/4569#issuecomment-469920279 > @XD-DENG , I don't think it will add 2000 entries to the db for admin role. It only adds the "all_dags" view-menu with the 2 permissions into the table. Hi @feng-tao please refer to https://github.com/apache/airflow/blob/dda309e66224346014ca42a0bcce4f53365f1118/airflow/www/security.py#L440-L453 By checking the code itself, seems it's simply adding all whatever permissions to Admin role. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] XD-DENG commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details
XD-DENG commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details URL: https://github.com/apache/airflow/pull/4569#issuecomment-469920279 > @XD-DENG , I don't think it will add 2000 entries to the db for admin role. It only adds the "all_dags" view-menu with the 2 permissions into the table. Hi @feng-tao please refer to https://github.com/apache/airflow/blob/dda309e66224346014ca42a0bcce4f53365f1118/airflow/www/security.py#L440-L453 . By checking the code itself, seems it's simply adding all whatever permissions to Admin role. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3981) Make Airflow UI timezone aware
[ https://issues.apache.org/jira/browse/AIRFLOW-3981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16785079#comment-16785079 ] Tao Feng commented on AIRFLOW-3981: --- cc [~verdan] our Airflow UI expect [~tronbabylove] I asked your question trough FAB issues(https://github.com/dpgaspar/Flask-AppBuilder/issues/920). > Make Airflow UI timezone aware > -- > > Key: AIRFLOW-3981 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3981 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Tao Feng >Assignee: Andrew Stahlman >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4022) Airflow on Kubernetes tutorial fails at finding ./docker/compile.sh
[
https://issues.apache.org/jira/browse/AIRFLOW-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rob Schoenbeck updated AIRFLOW-4022:
Description:
kubectl 1.11.8, client 1.13.3, minikube v0.34.1
Steps to reproduce: Simply follow the directions at
[https://github.com/apache/airflow/tree/master/scripts/ci/kubernetes] using
your own minikube (OS X 10.13.6 in this case)
# minikube start
# source ./docker/build.sh
# Fails with error:
{noformat}
dirname: illegal option -- b
usage: dirname path
Airflow directory /Users/robs/learning/airflow/scripts/ci/kubernetes/../../../..
Airflow Docker directory /Users/robs/learning/airflow/scripts/ci/kubernetes
docker: Error response from daemon: OCI runtime create failed:
container_linux.go:344: starting container process caused "exec:
\"./scripts/ci/kubernetes/docker/compile.sh\": stat
./scripts/ci/kubernetes/docker/compile.sh: no such file or directory":
unknown.{noformat}
was:
Steps to reproduce: Simply follow the directions at
[https://github.com/apache/airflow/tree/master/scripts/ci/kubernetes] using
your own minikube (OS X 10.13.6 in this case)
# minikube start
# source ./docker/build.sh
# Fails with error:
{noformat}
dirname: illegal option -- b
usage: dirname path
Airflow directory /Users/robs/learning/airflow/scripts/ci/kubernetes/../../../..
Airflow Docker directory /Users/robs/learning/airflow/scripts/ci/kubernetes
docker: Error response from daemon: OCI runtime create failed:
container_linux.go:344: starting container process caused "exec:
\"./scripts/ci/kubernetes/docker/compile.sh\": stat
./scripts/ci/kubernetes/docker/compile.sh: no such file or directory":
unknown.{noformat}
> Airflow on Kubernetes tutorial fails at finding ./docker/compile.sh
> ---
>
> Key: AIRFLOW-4022
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4022
> Project: Apache Airflow
> Issue Type: Bug
> Components: Documentation
>Reporter: Rob Schoenbeck
>Priority: Minor
>
> kubectl 1.11.8, client 1.13.3, minikube v0.34.1
> Steps to reproduce: Simply follow the directions at
> [https://github.com/apache/airflow/tree/master/scripts/ci/kubernetes] using
> your own minikube (OS X 10.13.6 in this case)
> # minikube start
> # source ./docker/build.sh
> # Fails with error:
> {noformat}
> dirname: illegal option -- b
> usage: dirname path
> Airflow directory
> /Users/robs/learning/airflow/scripts/ci/kubernetes/../../../..
> Airflow Docker directory /Users/robs/learning/airflow/scripts/ci/kubernetes
> docker: Error response from daemon: OCI runtime create failed:
> container_linux.go:344: starting container process caused "exec:
> \"./scripts/ci/kubernetes/docker/compile.sh\": stat
> ./scripts/ci/kubernetes/docker/compile.sh: no such file or directory":
> unknown.{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4022) Airflow on Kubernetes tutorial fails at finding ./docker/compile.sh
[
https://issues.apache.org/jira/browse/AIRFLOW-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16785052#comment-16785052
]
Rob Schoenbeck commented on AIRFLOW-4022:
-
Also apparently failing:
[https://github.com/apache/airflow/blob/master/scripts/ci/kubernetes/setup_kubernetes.sh]
steps to reproduce:
# source ./setup_kubernetes.sh
# Get error saying user that exists running the command cannot cannot be found
(robs)
{noformat}
+++ sudo -E
PATH=/Users/robs/learning/airflow/scripts/ci/kubernetes/minikube/bin:/Users/robs/.pyenv/shims:/anaconda3/bin:/anaconda3/condabin:/Users/robs/.local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/Library/TeX/texbin:/opt/X11/bin:/usr/local/git/bin:/Applications/Postgres.app/Contents/Versions/latest/bin
minikube start --kubernetes-version= --vm-driver=none
minikube v0.34.1 on darwin (amd64)
魯 Configuring local host environment ...
⚠️ The 'none' driver provides limited isolation and may reduce system security
and reliability.
⚠️ For more information, see:
https://github.com/kubernetes/minikube/blob/master/docs/vmdriver-none.md
Failed to chown /Users/robs/.minikube: Error looking up user: user: unknown
user robs{noformat}
> Airflow on Kubernetes tutorial fails at finding ./docker/compile.sh
> ---
>
> Key: AIRFLOW-4022
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4022
> Project: Apache Airflow
> Issue Type: Bug
> Components: Documentation
>Reporter: Rob Schoenbeck
>Priority: Minor
>
> kubectl 1.11.8, client 1.13.3, minikube v0.34.1
> Steps to reproduce: Simply follow the directions at
> [https://github.com/apache/airflow/tree/master/scripts/ci/kubernetes] using
> your own minikube (OS X 10.13.6 in this case)
> # minikube start
> # source ./docker/build.sh
> # Fails with error:
> {noformat}
> dirname: illegal option -- b
> usage: dirname path
> Airflow directory
> /Users/robs/learning/airflow/scripts/ci/kubernetes/../../../..
> Airflow Docker directory /Users/robs/learning/airflow/scripts/ci/kubernetes
> docker: Error response from daemon: OCI runtime create failed:
> container_linux.go:344: starting container process caused "exec:
> \"./scripts/ci/kubernetes/docker/compile.sh\": stat
> ./scripts/ci/kubernetes/docker/compile.sh: no such file or directory":
> unknown.{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] codecov-io edited a comment on issue #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run
codecov-io edited a comment on issue #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run URL: https://github.com/apache/airflow/pull/4751#issuecomment-466029246 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4751?src=pr=h1) Report > Merging [#4751](https://codecov.io/gh/apache/airflow/pull/4751?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/3bb0493106161506459e18632042c81f10f19946?src=pr=desc) will **decrease** coverage by `0.29%`. > The diff coverage is `40.9%`. [](https://codecov.io/gh/apache/airflow/pull/4751?src=pr=tree) ```diff @@Coverage Diff@@ ## master#4751 +/- ## = - Coverage 74.72% 74.43% -0.3% = Files 449 449 Lines 2895929085+126 = + Hits2164121648 +7 - Misses 7318 7437+119 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4751?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/state.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zdGF0ZS5weQ==) | `96.55% <ø> (ø)` | :arrow_up: | | [airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=) | `92.65% <ø> (ø)` | :arrow_up: | | [airflow/jobs.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5) | `76.94% <100%> (+0.06%)` | :arrow_up: | | [airflow/ti\_deps/dep\_context.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcF9jb250ZXh0LnB5) | `100% <100%> (ø)` | :arrow_up: | | [airflow/ti\_deps/deps/trigger\_rule\_dep.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvdHJpZ2dlcl9ydWxlX2RlcC5weQ==) | `77.01% <27.77%> (-13.13%)` | :arrow_down: | | [airflow/www/security.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvc2VjdXJpdHkucHk=) | `59.86% <0%> (-32.88%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4751?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4751?src=pr=footer). Last update [3bb0493...bb5c825](https://codecov.io/gh/apache/airflow/pull/4751?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] pgagnon commented on issue #4648: [AIRFLOW-3274] Add run_as_user and fs_group options for Kubernetes
pgagnon commented on issue #4648: [AIRFLOW-3274] Add run_as_user and fs_group options for Kubernetes URL: https://github.com/apache/airflow/pull/4648#issuecomment-469905909 @dmateusp In this PR I would be modifying slightly the functionality you added in #4777, specifically with regards to how you are handling security context. If you have a chance could you please take a look and let me know if you find any issues? Thanks in advance. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-4022) Airflow on Kubernetes fails at finding ./docker/compile.sh
Rob Schoenbeck created AIRFLOW-4022:
---
Summary: Airflow on Kubernetes fails at finding ./docker/compile.sh
Key: AIRFLOW-4022
URL: https://issues.apache.org/jira/browse/AIRFLOW-4022
Project: Apache Airflow
Issue Type: Bug
Components: Documentation
Reporter: Rob Schoenbeck
Steps to reproduce: Simply follow the directions at
[https://github.com/apache/airflow/tree/master/scripts/ci/kubernetes] using
your own minikube (OS X 10.13.6 in this case)
# minikube start
# source ./docker/build.sh
# Fails with error:
{noformat}
dirname: illegal option -- b
usage: dirname path
Airflow directory /Users/robs/learning/airflow/scripts/ci/kubernetes/../../../..
Airflow Docker directory /Users/robs/learning/airflow/scripts/ci/kubernetes
docker: Error response from daemon: OCI runtime create failed:
container_linux.go:344: starting container process caused "exec:
\"./scripts/ci/kubernetes/docker/compile.sh\": stat
./scripts/ci/kubernetes/docker/compile.sh: no such file or directory":
unknown.{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4022) Airflow on Kubernetes tutorial fails at finding ./docker/compile.sh
[
https://issues.apache.org/jira/browse/AIRFLOW-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rob Schoenbeck updated AIRFLOW-4022:
Summary: Airflow on Kubernetes tutorial fails at finding
./docker/compile.sh (was: Airflow on Kubernetes fails at finding
./docker/compile.sh)
> Airflow on Kubernetes tutorial fails at finding ./docker/compile.sh
> ---
>
> Key: AIRFLOW-4022
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4022
> Project: Apache Airflow
> Issue Type: Bug
> Components: Documentation
>Reporter: Rob Schoenbeck
>Priority: Minor
>
> Steps to reproduce: Simply follow the directions at
> [https://github.com/apache/airflow/tree/master/scripts/ci/kubernetes] using
> your own minikube (OS X 10.13.6 in this case)
> # minikube start
> # source ./docker/build.sh
> # Fails with error:
> {noformat}
> dirname: illegal option -- b
> usage: dirname path
> Airflow directory
> /Users/robs/learning/airflow/scripts/ci/kubernetes/../../../..
> Airflow Docker directory /Users/robs/learning/airflow/scripts/ci/kubernetes
> docker: Error response from daemon: OCI runtime create failed:
> container_linux.go:344: starting container process caused "exec:
> \"./scripts/ci/kubernetes/docker/compile.sh\": stat
> ./scripts/ci/kubernetes/docker/compile.sh: no such file or directory":
> unknown.{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] vsoch opened a new pull request #4846: [WIP] adding start to singularity for airflow
vsoch opened a new pull request #4846: [WIP] adding start to singularity for
airflow
URL: https://github.com/apache/airflow/pull/4846
Hey airflow maintainers! I'm working on an operator for [Singularity
containers](https://sylabs.io/guides/3.0/user-guide/), and I have a very basic
[example
working](https://github.com/researchapps/airflow-example/tree/master/singularity).
I'd like to do some more substantial tests beyond sleep (and I haven't written
proper tests yet) but wanted to open the pull request to start discussion about
tests and examples needed, etc. I'm new to airflow as of this afternoon, so
thank you (in the future) for teaching me all the things!
I haven't checked all the boxes below. I'm mostly afraid of Jira, so maybe
if someone wants to create an issue for this, would be greatly appreciated!
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement
Proposal([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-4020) remove viewer DAG edit permission
[ https://issues.apache.org/jira/browse/AIRFLOW-4020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16785026#comment-16785026 ] ASF GitHub Bot commented on AIRFLOW-4020: - feng-tao commented on pull request #4845: [AIRFLOW-4020] Remove viewer DAG edit permissions URL: https://github.com/apache/airflow/pull/4845 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > remove viewer DAG edit permission > - > > Key: AIRFLOW-4020 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4020 > Project: Apache Airflow > Issue Type: Improvement >Affects Versions: 1.10.2 >Reporter: Tao Feng >Assignee: Tao Feng >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4020) remove viewer DAG edit permission
[ https://issues.apache.org/jira/browse/AIRFLOW-4020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng resolved AIRFLOW-4020. --- Resolution: Fixed Fix Version/s: 1.10.3 > remove viewer DAG edit permission > - > > Key: AIRFLOW-4020 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4020 > Project: Apache Airflow > Issue Type: Improvement >Affects Versions: 1.10.2 >Reporter: Tao Feng >Assignee: Tao Feng >Priority: Major > Fix For: 1.10.3 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4020) remove viewer DAG edit permission
[ https://issues.apache.org/jira/browse/AIRFLOW-4020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16785027#comment-16785027 ] ASF subversion and git services commented on AIRFLOW-4020: -- Commit dda309e66224346014ca42a0bcce4f53365f1118 in airflow's branch refs/heads/master from Tao Feng [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=dda309e ] [AIRFLOW-4020] Remove viewer DAG edit permissions (#4845) > remove viewer DAG edit permission > - > > Key: AIRFLOW-4020 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4020 > Project: Apache Airflow > Issue Type: Improvement >Affects Versions: 1.10.2 >Reporter: Tao Feng >Assignee: Tao Feng >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] feng-tao merged pull request #4845: [AIRFLOW-4020] Remove viewer DAG edit permissions
feng-tao merged pull request #4845: [AIRFLOW-4020] Remove viewer DAG edit permissions URL: https://github.com/apache/airflow/pull/4845 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feng-tao edited a comment on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details
feng-tao edited a comment on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details URL: https://github.com/apache/airflow/pull/4569#issuecomment-469834959 @ashb , I see. Your deploy is based on 1.10.2 not on master? I think I fail to get this pr into 1.10.2 as the release timeline was pretty tight around the time. And I think someone said that just use UI to modify the permissions as a work around. Currently it only adds to those default roles to keep it the same behaviour as 1.10.1, but not those DAG level roles. And we do plan to write a blog on how we do DAG level access at Lyft, but just don't have enough time ATM. In terms of migrations from 1.10.1 to 1.10.2, I am not sure if it is easy to do as Airflow doesn't have the FAB model code which makes the db migration difficult. Let me know if you figure it out the right approach. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details
ashb commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details URL: https://github.com/apache/airflow/pull/4569#issuecomment-469896394 Oh thanks, I hadn't seen that diagram! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] XD-DENG commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details
XD-DENG commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details URL: https://github.com/apache/airflow/pull/4569#issuecomment-469896272 Thanks @feng-tao . let me double-check the code later today and will let you know if I still have confusion on it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feng-tao commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details
feng-tao commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details URL: https://github.com/apache/airflow/pull/4569#issuecomment-469895639 @XD-DENG , I don't think it will add 2000 entries to the db for admin role. It only adds the "all_dags" view-menu with the 2 permissions into the table. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] XD-DENG commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details
XD-DENG commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details URL: https://github.com/apache/airflow/pull/4569#issuecomment-469894456 I have a question to seek @feng-tao ‘s input given we have started the discussion on this: currently Admin’s permissions will always be updated so it has ALL permissions, including each DAG’s “can_dag_view” and “can_dag_edit”. Let’s say we have 1000 DAGs now: then 2000 permissions will be added to Admin role in DB, and all of them will also be shown in UI if we go to Security->Roles. This may not be necessary as Admin role already has “can_dag_view” and “can_dag_edit” on “all_dags”. (but from another side, I also understand it’s easier to just simply give Admin ALL permissions) May you let me know your thoughts on this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-4021) Log file location in email alert incorrect
[
https://issues.apache.org/jira/browse/AIRFLOW-4021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784983#comment-16784983
]
Kiran Gavali commented on AIRFLOW-4021:
---
Would it be possible to use the following property in airflow.cfg when
generating the email content:
{{log_filename_template = \{{ ti.dag_id }}/\{{ ti.task_id }}/\{{ ts }}/\{{
try_number }}.log}}
> Log file location in email alert incorrect
> --
>
> Key: AIRFLOW-4021
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4021
> Project: Apache Airflow
> Issue Type: Bug
> Components: logging
>Affects Versions: 1.10.1
>Reporter: Kiran Gavali
>Priority: Minor
>
> Log files for task tries are stored in
> {{base_log_dir/dag_id/task_id/execution_date_iso_format/try_number.log}}. On
> task failure, the email alert that is sent specifies "Log file:" path as
> {{base_log_dir/dag_id/task_id/execution_date_iso_format.log}}. Until
> AIRFLOW-3909 is fixed, one would have to use the log file path on the server
> mentioned in the email alert to view the log for the failed try. Related code
> line is
> [here|https://github.com/apache/airflow/blob/ad02179c28f1f93cbf55789b9ca8f3b9eeafeff7/airflow/models/__init__.py#L875].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] feng-tao edited a comment on issue #4627: [AIRFLOW-3258] K8S executor environment variables section.
feng-tao edited a comment on issue #4627: [AIRFLOW-3258] K8S executor environment variables section. URL: https://github.com/apache/airflow/pull/4627#issuecomment-469891035 @davlum , a n00b question: do you know if it is desirable to have both env variable defined and annotation defined(https://github.com/apache/airflow/pull/4589/files)? Or which one is the preferred approach? lgtm based on 101 k8s knowledge cc @dimberman in case he has cycle. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feng-tao commented on issue #4627: [AIRFLOW-3258] K8S executor environment variables section.
feng-tao commented on issue #4627: [AIRFLOW-3258] K8S executor environment variables section. URL: https://github.com/apache/airflow/pull/4627#issuecomment-469891035 @davlum , a n00b question: do you know if it is desirable to have both env variable defined and annotation defined(https://github.com/apache/airflow/pull/4589/files)? Or which one is a preferred approach? lgtm based on 101 k8s knowledge cc @dimberman in case he has cycle. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #4845: [AIRFLOW-4020] Remove viewer DAG edit permissions
codecov-io commented on issue #4845: [AIRFLOW-4020] Remove viewer DAG edit permissions URL: https://github.com/apache/airflow/pull/4845#issuecomment-469889220 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4845?src=pr=h1) Report > Merging [#4845](https://codecov.io/gh/apache/airflow/pull/4845?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/205ff174247244b44ad8bbaf170854b6fb747221?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/4845?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4845 +/- ## == - Coverage 74.73% 74.73% -0.01% == Files 449 449 Lines 2895928961 +2 == + Hits2164221643 +1 - Misses 7317 7318 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4845?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/www/security.py](https://codecov.io/gh/apache/airflow/pull/4845/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvc2VjdXJpdHkucHk=) | `92.82% <100%> (+0.07%)` | :arrow_up: | | [airflow/contrib/operators/ssh\_operator.py](https://codecov.io/gh/apache/airflow/pull/4845/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zc2hfb3BlcmF0b3IucHk=) | `82.27% <0%> (-1.27%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4845?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4845?src=pr=footer). Last update [205ff17...ccfd3af](https://codecov.io/gh/apache/airflow/pull/4845?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (AIRFLOW-3985) Adding linear DAG relationships with unary operator
[
https://issues.apache.org/jira/browse/AIRFLOW-3985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jarek Potiuk closed AIRFLOW-3985.
-
Resolution: Not A Problem
> Adding linear DAG relationships with unary operator
> ---
>
> Key: AIRFLOW-3985
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3985
> Project: Apache Airflow
> Issue Type: New Feature
>Reporter: Jarek Potiuk
>Priority: Major
>
> It would be nice for development and testing to be able to build linear DAG
> dependencies with unary + operator rather than with << with automated
> upstream dependency to the previously added task.
> Often during development we need to inject/remove tasks between two other
> tasks and with unary operator it might be very easy to comment out whole
> lines of code.
> Example:
> {code:java}
> +op1
> +op2
> +op3{code}
> might be equivalent to:
> {code:java}
> op1 >> op2 >> op3{code}
> This way we can comment out one line:
> {code:java}
> +op1
> # +op2
> +op3{code}
> equivalent to:
> {code:java}
> op1 >> op3{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4019) AWS Athena Sensor's object has no attribute 'mode'
[
https://issues.apache.org/jira/browse/AIRFLOW-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784971#comment-16784971
]
ASF GitHub Bot commented on AIRFLOW-4019:
-
feng-tao commented on pull request #4844: [AIRFLOW-4019] Fix AWS Athena Sensor
object has no attribute 'mode'
URL: https://github.com/apache/airflow/pull/4844
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> AWS Athena Sensor's object has no attribute 'mode'
> --
>
> Key: AIRFLOW-4019
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4019
> Project: Apache Airflow
> Issue Type: Bug
> Components: contrib
>Affects Versions: 1.10.2
>Reporter: Mariko Wakabayashi
>Assignee: Mariko Wakabayashi
>Priority: Major
> Fix For: 1.10.3
>
>
> *Bug*
> {code:java}
> [2019-03-05 18:52:32,317] {models.py:1788} ERROR - 'AthenaSensor' object has
> no attribute 'mode'
> Traceback (most recent call last):
> File "/usr/local/lib/python3.6/site-packages/airflow/models.py", line 1657,
> in _run_raw_task
> result = task_copy.execute(context=context)
> File
> "/usr/local/lib/python3.6/site-packages/airflow/sensors/base_sensor_operator.py",
> line 92, in execute
> if self.reschedule:
> File
> "/usr/local/lib/python3.6/site-packages/airflow/sensors/base_sensor_operator.py",
> line 123, in reschedule
> return self.mode == 'reschedule'{code}
> *Fix*
> * AWS Athena Sensor's first argument to super() is incorrect
> *
> [https://github.com/apache/airflow/blob/master/airflow/contrib/sensors/aws_athena_sensor.py#L59]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3985) Adding linear DAG relationships with unary operator
[
https://issues.apache.org/jira/browse/AIRFLOW-3985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784975#comment-16784975
]
Jarek Potiuk commented on AIRFLOW-3985:
---
Closing following the discussion in
[https://github.com/apache/airflow/pull/4810]
> Adding linear DAG relationships with unary operator
> ---
>
> Key: AIRFLOW-3985
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3985
> Project: Apache Airflow
> Issue Type: New Feature
>Reporter: Jarek Potiuk
>Priority: Major
>
> It would be nice for development and testing to be able to build linear DAG
> dependencies with unary + operator rather than with << with automated
> upstream dependency to the previously added task.
> Often during development we need to inject/remove tasks between two other
> tasks and with unary operator it might be very easy to comment out whole
> lines of code.
> Example:
> {code:java}
> +op1
> +op2
> +op3{code}
> might be equivalent to:
> {code:java}
> op1 >> op2 >> op3{code}
> This way we can comment out one line:
> {code:java}
> +op1
> # +op2
> +op3{code}
> equivalent to:
> {code:java}
> op1 >> op3{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] feng-tao commented on issue #4845: [AIRFLOW-4020] Remove viewer DAG edit permissions
feng-tao commented on issue #4845: [AIRFLOW-4020] Remove viewer DAG edit permissions URL: https://github.com/apache/airflow/pull/4845#issuecomment-469881537 PTAL @ashb Tests pass This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk closed pull request #4810: [AIRFLOW-3985] Added unary + operator for easier linear DAG building
potiuk closed pull request #4810: [AIRFLOW-3985] Added unary + operator for easier linear DAG building URL: https://github.com/apache/airflow/pull/4810 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3985) Adding linear DAG relationships with unary operator
[
https://issues.apache.org/jira/browse/AIRFLOW-3985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784974#comment-16784974
]
ASF GitHub Bot commented on AIRFLOW-3985:
-
potiuk commented on pull request #4810: [AIRFLOW-3985] Added unary + operator
for easier linear DAG building
URL: https://github.com/apache/airflow/pull/4810
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Adding linear DAG relationships with unary operator
> ---
>
> Key: AIRFLOW-3985
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3985
> Project: Apache Airflow
> Issue Type: New Feature
>Reporter: Jarek Potiuk
>Priority: Major
>
> It would be nice for development and testing to be able to build linear DAG
> dependencies with unary + operator rather than with << with automated
> upstream dependency to the previously added task.
> Often during development we need to inject/remove tasks between two other
> tasks and with unary operator it might be very easy to comment out whole
> lines of code.
> Example:
> {code:java}
> +op1
> +op2
> +op3{code}
> might be equivalent to:
> {code:java}
> op1 >> op2 >> op3{code}
> This way we can comment out one line:
> {code:java}
> +op1
> # +op2
> +op3{code}
> equivalent to:
> {code:java}
> op1 >> op3{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] potiuk commented on issue #4810: [AIRFLOW-3985] Added unary + operator for easier linear DAG building
potiuk commented on issue #4810: [AIRFLOW-3985] Added unary + operator for easier linear DAG building URL: https://github.com/apache/airflow/pull/4810#issuecomment-469881001 Closing as this could be misleading This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-4019) AWS Athena Sensor's object has no attribute 'mode'
[
https://issues.apache.org/jira/browse/AIRFLOW-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784972#comment-16784972
]
ASF subversion and git services commented on AIRFLOW-4019:
--
Commit 3bb0493106161506459e18632042c81f10f19946 in airflow's branch
refs/heads/master from Mariko Wakabayashi
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=3bb0493 ]
[AIRFLOW-4019] Fix AWS Athena Sensor object has no attribute 'mode' (#4844)
> AWS Athena Sensor's object has no attribute 'mode'
> --
>
> Key: AIRFLOW-4019
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4019
> Project: Apache Airflow
> Issue Type: Bug
> Components: contrib
>Affects Versions: 1.10.2
>Reporter: Mariko Wakabayashi
>Assignee: Mariko Wakabayashi
>Priority: Major
> Fix For: 1.10.3
>
>
> *Bug*
> {code:java}
> [2019-03-05 18:52:32,317] {models.py:1788} ERROR - 'AthenaSensor' object has
> no attribute 'mode'
> Traceback (most recent call last):
> File "/usr/local/lib/python3.6/site-packages/airflow/models.py", line 1657,
> in _run_raw_task
> result = task_copy.execute(context=context)
> File
> "/usr/local/lib/python3.6/site-packages/airflow/sensors/base_sensor_operator.py",
> line 92, in execute
> if self.reschedule:
> File
> "/usr/local/lib/python3.6/site-packages/airflow/sensors/base_sensor_operator.py",
> line 123, in reschedule
> return self.mode == 'reschedule'{code}
> *Fix*
> * AWS Athena Sensor's first argument to super() is incorrect
> *
> [https://github.com/apache/airflow/blob/master/airflow/contrib/sensors/aws_athena_sensor.py#L59]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] feng-tao merged pull request #4844: [AIRFLOW-4019] Fix AWS Athena Sensor object has no attribute 'mode'
feng-tao merged pull request #4844: [AIRFLOW-4019] Fix AWS Athena Sensor object has no attribute 'mode' URL: https://github.com/apache/airflow/pull/4844 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feng-tao commented on issue #4844: [AIRFLOW-4019] Fix AWS Athena Sensor object has no attribute 'mode'
feng-tao commented on issue #4844: [AIRFLOW-4019] Fix AWS Athena Sensor object has no attribute 'mode' URL: https://github.com/apache/airflow/pull/4844#issuecomment-469879613 lgtm This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3802) Correct HiveServer2Hook documentation formatting
[ https://issues.apache.org/jira/browse/AIRFLOW-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784931#comment-16784931 ] ASF subversion and git services commented on AIRFLOW-3802: -- Commit 66fef93138f7880e1e10b4459503a74277aa246c in airflow's branch refs/heads/v1-10-stable from mans2singh [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=66fef93 ] [AIRFLOW-3802] Updated documentation for HiveServer2Hook (#4647) > Correct HiveServer2Hook documentation formatting > > > Key: AIRFLOW-3802 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3802 > Project: Apache Airflow > Issue Type: Improvement > Components: Documentation >Affects Versions: 1.10.2 > Environment: All >Reporter: Mans Singh >Assignee: Mans Singh >Priority: Trivial > Labels: docuentation, hiveserver, hooks, newbie > Fix For: 1.10.3 > > Original Estimate: 1h > Remaining Estimate: 1h > > The documentation for the HiveServer2Hook is properly formatted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3801) DagBag always import examples
[
https://issues.apache.org/jira/browse/AIRFLOW-3801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784936#comment-16784936
]
ASF subversion and git services commented on AIRFLOW-3801:
--
Commit 351514905897e15e316b3e6ee812d085f4e3dcd8 in airflow's branch
refs/heads/v1-10-stable from Patouche
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=3515149 ]
[AIRFLOW-3801] Fix DagBag collect dags invocation to prevent examples to be
loaded (#4677)
> DagBag always import examples
> --
>
> Key: AIRFLOW-3801
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3801
> Project: Apache Airflow
> Issue Type: Bug
> Components: models
>Affects Versions: 1.10.2, 2.0.0
>Reporter: Patrick Allain
>Priority: Minor
> Fix For: 1.10.3
>
>
> Hello,
> Yesterday, I notice in my unit test that the example was always loaded even
> if I explicitly specified the parameters {{include_examples=False}} in the
> {{DagBag}} constructor.
> In fact, the DagBag init method invoke the {{collect_dags}} method without
> naming the parameters. Inside, the {{collect_dags}} method, the method
> {{list_py_file_paths}} was invoke without naming the parameters also.
> I tried to fix it in a pull request that will be made very soon.
>
> Regards,
> Patrick
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3821) Add replicas logic to GCP SQL example DAG
[ https://issues.apache.org/jira/browse/AIRFLOW-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784938#comment-16784938 ] ASF subversion and git services commented on AIRFLOW-3821: -- Commit 750ab5a4b114f3d226342749010325e49c7812c4 in airflow's branch refs/heads/v1-10-stable from Antoni Smoliński [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=750ab5a ] [AIRFLOW-3821] Add replicas logic to GCP SQL example DAG (#4662) > Add replicas logic to GCP SQL example DAG > - > > Key: AIRFLOW-3821 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3821 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Antoni Smoliński >Assignee: Antoni Smoliński >Priority: Minor > Fix For: 1.10.3 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2508) Handle non string types in render_template_from_field
[
https://issues.apache.org/jira/browse/AIRFLOW-2508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784941#comment-16784941
]
ASF subversion and git services commented on AIRFLOW-2508:
--
Commit 057258e3ec3f585eb7afb6dd2857e9bf5e262103 in airflow's branch
refs/heads/v1-10-stable from Géraud
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=057258e ]
[AIRFLOW-2508] Handle non string types in Operators templatized fields (#4292)
> Handle non string types in render_template_from_field
> -
>
> Key: AIRFLOW-2508
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2508
> Project: Apache Airflow
> Issue Type: Bug
> Components: models
>Affects Versions: 2.0.0
>Reporter: Eugene Brown
>Assignee: Galak
>Priority: Minor
> Labels: easyfix, newbie
> Fix For: 1.10.3
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> The render_template_from_field method of the BaseOperator class raises an
> exception when it encounters content that is not a string_type, list, tuple
> or dict.
> Example exception:
> {noformat}
> airflow.exceptions.AirflowException: Type '' used for parameter
> 'job_flow_overrides[Instances][InstanceGroups][InstanceCount]' is not
> supported for templating{noformat}
> I propose instead that when it encounters content of other types it returns
> the content unchanged, rather than raising an exception.
> Consider this case: I extended the EmrCreateJobFlowOperator to make the
> job_flow_overrides argument a templatable field. job_flow_overrides is a
> dictionary with a mix of strings, integers and booleans for values.
> When I extended the class as such:
> {code:java}
> class EmrCreateJobFlowOperatorTemplateOverrides(EmrCreateJobFlowOperator):
> template_fields = ['job_flow_overrides']{code}
> And added a task to my dag with this format:
> {code:java}
> step_create_cluster = EmrCreateJobFlowOperatorTemplateOverrides(
> task_id="create_cluster",
> job_flow_overrides={
> "Name": "my-cluster {{ dag_run.conf['run_date'] }}",
> "Instances": {
> "InstanceGroups": [
> {
> "Name": "Master nodes",
> "InstanceType": "c3.4xlarge",
> "InstanceCount": 1
> },
> {
> "Name": "Slave nodes",
> "InstanceType": "c3.4xlarge",
> "InstanceCount": 4
> },
> "TerminationProtected": False
> ]
> },
> "BootstrapActions": [{
> "Name": "Custom action",
> "ScriptBootstrapAction": {
> "Path": "s3://repo/{{ dag_run.conf['branch']
> }}/requirements.txt"
> }
> }],
>},
>aws_conn_id='aws_default',
>emr_conn_id='aws_default',
>dag=dag
> )
> {code}
> The exception I gave above was raised and the step failed. I think it would
> be preferable for the method to instead pass over numeric and boolean values
> as users may want to use template_fields in the way I have to template string
> values in dictionaries or lists of mixed types.
> Here is the render_template_from_field method from the BaseOperator:
> {code:java}
> def render_template_from_field(self, attr, content, context, jinja_env):
> """
> Renders a template from a field. If the field is a string, it will
> simply render the string and return the result. If it is a collection or
> nested set of collections, it will traverse the structure and render
> all strings in it.
> """
> rt = self.render_template
> if isinstance(content, six.string_types):
> result = jinja_env.from_string(content).render(**context)
> elif isinstance(content, (list, tuple)):
> result = [rt(attr, e, context) for e in content]
> elif isinstance(content, dict):
> result = {
> k: rt("{}[{}]".format(attr, k), v, context)
> for k, v in list(content.items())}
> else:
> param_type = type(content)
> msg = (
> "Type '{param_type}' used for parameter '{attr}' is "
> "not supported for templating").format(**locals())
> raise AirflowException(msg)
> return result{code}
> I propose that the method returns content unchanged if the content is of one
> of (int, float, complex, bool) types. So my solution would include an extra
> elif in the form:
> {code}
> elif isinstance(content, (int, float, complex, bool)):
> result = content
> {code}
> Are there any reasons this would be a bad idea?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3547) Jinja templating is not enabled for some SparkSubmitOperator parameters.
[
https://issues.apache.org/jira/browse/AIRFLOW-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784934#comment-16784934
]
ASF subversion and git services commented on AIRFLOW-3547:
--
Commit 09f7a89453b6a567290048f0115f50510c8c5f56 in airflow's branch
refs/heads/v1-10-stable from Mathew Wicks
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=09f7a89 ]
[AIRFLOW-3547] Fixed Jinja templating in SparkSubmitOperator (#4347)
* [AIRFLOW-3547] Fixed Jinja templating in SparkSubmitOperator
This is a minor change to allow Jinja templating in parameters where it
makes sense for SparkSubmitOperator.
* [AIRFLOW-3547] Fixed Jinja templating in SparkSubmitOperator
This is a minor change to allow Jinja templating in parameters where it
makes sense for SparkSubmitOperator.
> Jinja templating is not enabled for some SparkSubmitOperator parameters.
>
>
> Key: AIRFLOW-3547
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3547
> Project: Apache Airflow
> Issue Type: Improvement
> Components: contrib
>Affects Versions: 1.10.1
>Reporter: Mathew Wicks
>Assignee: Mathew Wicks
>Priority: Minor
> Fix For: 2.0.0, 1.10.3
>
>
> SparkSubmitOperator currently only supports Jinja templating in its 'name',
> 'application_args' and 'packages' parameters, this is problematic as a user
> might want to do something like:
> {code:python}
> application="{{ dag.folder }}/spark_code.py"
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3547) Jinja templating is not enabled for some SparkSubmitOperator parameters.
[
https://issues.apache.org/jira/browse/AIRFLOW-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784935#comment-16784935
]
ASF subversion and git services commented on AIRFLOW-3547:
--
Commit 09f7a89453b6a567290048f0115f50510c8c5f56 in airflow's branch
refs/heads/v1-10-stable from Mathew Wicks
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=09f7a89 ]
[AIRFLOW-3547] Fixed Jinja templating in SparkSubmitOperator (#4347)
* [AIRFLOW-3547] Fixed Jinja templating in SparkSubmitOperator
This is a minor change to allow Jinja templating in parameters where it
makes sense for SparkSubmitOperator.
* [AIRFLOW-3547] Fixed Jinja templating in SparkSubmitOperator
This is a minor change to allow Jinja templating in parameters where it
makes sense for SparkSubmitOperator.
> Jinja templating is not enabled for some SparkSubmitOperator parameters.
>
>
> Key: AIRFLOW-3547
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3547
> Project: Apache Airflow
> Issue Type: Improvement
> Components: contrib
>Affects Versions: 1.10.1
>Reporter: Mathew Wicks
>Assignee: Mathew Wicks
>Priority: Minor
> Fix For: 2.0.0, 1.10.3
>
>
> SparkSubmitOperator currently only supports Jinja templating in its 'name',
> 'application_args' and 'packages' parameters, this is problematic as a user
> might want to do something like:
> {code:python}
> application="{{ dag.folder }}/spark_code.py"
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3647) Contributed SparkSubmitOperator doesn't honor --archives configuration
[ https://issues.apache.org/jira/browse/AIRFLOW-3647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784932#comment-16784932 ] ASF subversion and git services commented on AIRFLOW-3647: -- Commit 9c11ac92fdda0e8ba623cb0be543ff5058cd9ce2 in airflow's branch refs/heads/v1-10-stable from Penumbra69 [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=9c11ac9 ] [AIRFLOW-3647] Add archives config option to SparkSubmitOperator (#4467) To enable to spark behavior of transporting and extracting an archive on job launch, making the _contents_ of the archive available to the driver as well as the workers (not just the jar or archive as a zip file) - this configuration attribute is necessary. This is required if you have no ability to modify the Python env on the worker / driver nodes, but you wish to use versions, modules, or features not installed. We transport a full Python 3.5 environment to our CDH cluster using this option and the alias "#PYTHON" paired an additional configuration to spark to use it: --archives "hdfs:///user/myuser/my_python_env.zip#PYTHON" --conf "spark.yarn.appMasterEnv.PYSPARK_PYTHON=./PYTHON/python35/bin/python3" > Contributed SparkSubmitOperator doesn't honor --archives configuration > -- > > Key: AIRFLOW-3647 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3647 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib >Affects Versions: 1.10.1 > Environment: Linux (RHEL 7) > Python 3.5 (using a virtual environment) > spark-2.1.3-bin-hadoop26 > Airflow 1.10.1 > CDH 5.14 Hadoop [Yarn] cluster (no end user / dev modifications allowed) >Reporter: Ken Melms >Priority: Minor > Labels: easyfix, newbie > Fix For: 1.10.3 > > Original Estimate: 1h > Remaining Estimate: 1h > > The contributed SparkSubmitOperator has no ability to honor the spark-submit > configuration field "--archives" which is treated subtly different than > "files" or "-py-files" in that it will unzip the archive into the > application's working directory, and can optionally add an alias to the > unzipped folder so that you can refer to it elsewhere in your submission. > EG: > spark-submit --archives=hdfs:user/someone/python35_venv.zip#PYTHON > --conf "spark.yarn.appMasterEnv.PYSPARK_PYTHON=./PYTHON/python35/bin/python3" > run_me.py > In our case - this behavior allows for multiple python virtual environments > to be sourced from HDFS without incurring the penalty of pushing the whole > python virtual env to the cluster each submission. This solves (for us) > using python-based spark jobs on a cluster that the end user has no ability > to define the python modules in use. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3547) Jinja templating is not enabled for some SparkSubmitOperator parameters.
[
https://issues.apache.org/jira/browse/AIRFLOW-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784933#comment-16784933
]
ASF subversion and git services commented on AIRFLOW-3547:
--
Commit 09f7a89453b6a567290048f0115f50510c8c5f56 in airflow's branch
refs/heads/v1-10-stable from Mathew Wicks
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=09f7a89 ]
[AIRFLOW-3547] Fixed Jinja templating in SparkSubmitOperator (#4347)
* [AIRFLOW-3547] Fixed Jinja templating in SparkSubmitOperator
This is a minor change to allow Jinja templating in parameters where it
makes sense for SparkSubmitOperator.
* [AIRFLOW-3547] Fixed Jinja templating in SparkSubmitOperator
This is a minor change to allow Jinja templating in parameters where it
makes sense for SparkSubmitOperator.
> Jinja templating is not enabled for some SparkSubmitOperator parameters.
>
>
> Key: AIRFLOW-3547
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3547
> Project: Apache Airflow
> Issue Type: Improvement
> Components: contrib
>Affects Versions: 1.10.1
>Reporter: Mathew Wicks
>Assignee: Mathew Wicks
>Priority: Minor
> Fix For: 2.0.0, 1.10.3
>
>
> SparkSubmitOperator currently only supports Jinja templating in its 'name',
> 'application_args' and 'packages' parameters, this is problematic as a user
> might want to do something like:
> {code:python}
> application="{{ dag.folder }}/spark_code.py"
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] Fokko commented on a change in pull request #4810: [AIRFLOW-3985] Added unary + operator for easier linear DAG building
Fokko commented on a change in pull request #4810: [AIRFLOW-3985] Added unary + operator for easier linear DAG building URL: https://github.com/apache/airflow/pull/4810#discussion_r262699628 ## File path: docs/concepts.rst ## @@ -263,6 +263,31 @@ and equivalent to: op1.set_downstream([op2, op3]) +Building linear DAGs with unary + operator +-- +Added in Airflow 10.1.3 (?) + +In simple, linear DAG case, operator relationships can be set with the unary + operator. The first time the ++ operator is used in DAG it adds no upstream dependency for the task, every subsequent time it adds upstream +dependency to the task that was most recently added with the + operator. Unlike with bitshift operators +the operators have to have the same dag already assigned (dag assignment does not carry through. + +.. code:: python + ++op1 ++op2 ++op3 Review comment: I'm not too confident about pushing this as a preferred way of chaining the operators. My preference would be using the `>>` which will translate into an arrow in the GUI. My main objection is that it feels a bit magic, and by the words of Python Zen: ``` MacBook-Pro-van-Fokko:incubator-airflow fokkodriesprong$ python3 Python 3.6.6 (v3.6.6:4cf1f54eb7, Jun 26 2018, 19:50:54) [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import this The Zen of Python, by Tim Peters Beautiful is better than ugly. Explicit is better than implicit. Simple is better than complex. Complex is better than complicated. Flat is better than nested. Sparse is better than dense. Readability counts. Special cases aren't special enough to break the rules. Although practicality beats purity. Errors should never pass silently. Unless explicitly silenced. In the face of ambiguity, refuse the temptation to guess. There should be one-- and preferably only one --obvious way to do it. Although that way may not be obvious at first unless you're Dutch. Now is better than never. Although never is often better than *right* now. If the implementation is hard to explain, it's a bad idea. If the implementation is easy to explain, it may be a good idea. Namespaces are one honking great idea -- let's do more of those! ``` I think explicit is better than implicit. I'm also not a big fan of the `.upstream_previous()`, my main preference would be the `>>` and `<<` operator (as in all the examples). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feng-tao commented on issue #4712: [AIRFLOW-3892] Create Redis pub sub sensor
feng-tao commented on issue #4712: [AIRFLOW-3892] Create Redis pub sub sensor URL: https://github.com/apache/airflow/pull/4712#issuecomment-469866240 @mans2singh sorry for the late review. Overall lgtm. We recently upgrade redis client(https://github.com/apache/airflow/pull/4834). Could you rebase your pr with master and do a sanity test and see if it still works as expected? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feng-tao merged pull request #4843: [AIRFLOW-XXX] Add note about backwards incompatible changes
feng-tao merged pull request #4843: [AIRFLOW-XXX] Add note about backwards incompatible changes URL: https://github.com/apache/airflow/pull/4843 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-4012) Upgrade upper bound of tabulate to 0.8.3
[ https://issues.apache.org/jira/browse/AIRFLOW-4012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784922#comment-16784922 ] ASF subversion and git services commented on AIRFLOW-4012: -- Commit f19538a319de4336f0cd90094dd9c2cb84650d50 in airflow's branch refs/heads/master from OmerJog [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=f19538a ] [AIRFLOW-4012] - Upgrade tabulate to 0.8.3 (#4838) Pin upper version bound to 0.9 too, so we don't have to update this again until 0.9 is out (when we can check for compatability) > Upgrade upper bound of tabulate to 0.8.3 > > > Key: AIRFLOW-4012 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4012 > Project: Apache Airflow > Issue Type: Improvement >Reporter: omerjog >Priority: Major > > Version 0.8.3 was released in January 2019 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] ashb commented on issue #4368: [AIRFLOW-3561] Improve queries
ashb commented on issue #4368: [AIRFLOW-3561] Improve queries URL: https://github.com/apache/airflow/pull/4368#issuecomment-469865539 We reverted this from master? Did it ever make it back in in any form? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feng-tao commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details
feng-tao commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details URL: https://github.com/apache/airflow/pull/4569#issuecomment-469865060 @ashb https://flask-appbuilder.readthedocs.io/en/latest/diagrams.html FYI This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-4016) Make job tests more stateless
[ https://issues.apache.org/jira/browse/AIRFLOW-4016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784926#comment-16784926 ] ASF subversion and git services commented on AIRFLOW-4016: -- Commit 6618dcb39b937db12d13ab44924a9440eecfa7ba in airflow's branch refs/heads/master from Peter van 't Hof [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=6618dcb ] [AIRFLOW-4016] Clear runs for BackfillJobTest (#4839) * Clear runs for BackfillJobTest * Fixing import * Fixing flake8 > Make job tests more stateless > - > > Key: AIRFLOW-4016 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4016 > Project: Apache Airflow > Issue Type: Bug > Components: tests >Reporter: Peter van 't Hof >Assignee: Peter van 't Hof >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4016) Make job tests more stateless
[ https://issues.apache.org/jira/browse/AIRFLOW-4016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784925#comment-16784925 ] ASF GitHub Bot commented on AIRFLOW-4016: - Fokko commented on pull request #4839: [AIRFLOW-4016] Clear runs for BackfillJobTest URL: https://github.com/apache/airflow/pull/4839 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Make job tests more stateless > - > > Key: AIRFLOW-4016 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4016 > Project: Apache Airflow > Issue Type: Bug > Components: tests >Reporter: Peter van 't Hof >Assignee: Peter van 't Hof >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4016) Make job tests more stateless
[ https://issues.apache.org/jira/browse/AIRFLOW-4016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-4016. --- Resolution: Fixed Fix Version/s: 2.0.0 > Make job tests more stateless > - > > Key: AIRFLOW-4016 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4016 > Project: Apache Airflow > Issue Type: Bug > Components: tests >Reporter: Peter van 't Hof >Assignee: Peter van 't Hof >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4012) Upgrade upper bound of tabulate to 0.8.3
[ https://issues.apache.org/jira/browse/AIRFLOW-4012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor resolved AIRFLOW-4012. Resolution: Fixed Fix Version/s: 1.10.3 > Upgrade upper bound of tabulate to 0.8.3 > > > Key: AIRFLOW-4012 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4012 > Project: Apache Airflow > Issue Type: Improvement >Reporter: omerjog >Priority: Major > Fix For: 1.10.3 > > > Version 0.8.3 was released in January 2019 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] Fokko merged pull request #4839: [AIRFLOW-4016] Clear runs for BackfillJobTest
Fokko merged pull request #4839: [AIRFLOW-4016] Clear runs for BackfillJobTest URL: https://github.com/apache/airflow/pull/4839 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-4012) Upgrade upper bound of tabulate to 0.8.3
[ https://issues.apache.org/jira/browse/AIRFLOW-4012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784921#comment-16784921 ] ASF GitHub Bot commented on AIRFLOW-4012: - ashb commented on pull request #4838: [AIRFLOW-4012] - Upgrade tabulate to 0.8.3 URL: https://github.com/apache/airflow/pull/4838 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Upgrade upper bound of tabulate to 0.8.3 > > > Key: AIRFLOW-4012 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4012 > Project: Apache Airflow > Issue Type: Improvement >Reporter: omerjog >Priority: Major > > Version 0.8.3 was released in January 2019 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] ashb merged pull request #4838: [AIRFLOW-4012] - Upgrade tabulate to 0.8.3
ashb merged pull request #4838: [AIRFLOW-4012] - Upgrade tabulate to 0.8.3 URL: https://github.com/apache/airflow/pull/4838 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #4838: [AIRFLOW-4012] - Upgrade tabulate to 0.8.3
codecov-io edited a comment on issue #4838: [AIRFLOW-4012] - Upgrade tabulate to 0.8.3 URL: https://github.com/apache/airflow/pull/4838#issuecomment-469630748 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4838?src=pr=h1) Report > Merging [#4838](https://codecov.io/gh/apache/airflow/pull/4838?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/0b5cc7127fde3b8cfbec881904ebcad7279b155d?src=pr=desc) will **decrease** coverage by `0.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/4838?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4838 +/- ## == - Coverage 74.64% 74.63% -0.02% == Files 449 449 Lines 2894028963 +23 == + Hits2160321616 +13 - Misses 7337 7347 +10 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4838?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/contrib/operators/qubole\_operator.py](https://codecov.io/gh/apache/airflow/pull/4838/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9xdWJvbGVfb3BlcmF0b3IucHk=) | `85.41% <0%> (-2.09%)` | :arrow_down: | | [airflow/contrib/hooks/qubole\_hook.py](https://codecov.io/gh/apache/airflow/pull/4838/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3F1Ym9sZV9ob29rLnB5) | `51.93% <0%> (-1.73%)` | :arrow_down: | | [airflow/utils/log/wasb\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/4838/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvd2FzYl90YXNrX2hhbmRsZXIucHk=) | `0% <0%> (ø)` | :arrow_up: | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/4838/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.47% <0%> (+0.01%)` | :arrow_up: | | [airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/4838/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=) | `92.65% <0%> (+0.06%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4838?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4838?src=pr=footer). Last update [0b5cc71...d1a4059](https://codecov.io/gh/apache/airflow/pull/4838?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #4838: [AIRFLOW-4012] - Upgrade tabulate to 0.8.3
codecov-io edited a comment on issue #4838: [AIRFLOW-4012] - Upgrade tabulate to 0.8.3 URL: https://github.com/apache/airflow/pull/4838#issuecomment-469630748 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4838?src=pr=h1) Report > Merging [#4838](https://codecov.io/gh/apache/airflow/pull/4838?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/0b5cc7127fde3b8cfbec881904ebcad7279b155d?src=pr=desc) will **decrease** coverage by `0.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/4838?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4838 +/- ## == - Coverage 74.64% 74.63% -0.02% == Files 449 449 Lines 2894028963 +23 == + Hits2160321616 +13 - Misses 7337 7347 +10 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4838?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/contrib/operators/qubole\_operator.py](https://codecov.io/gh/apache/airflow/pull/4838/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9xdWJvbGVfb3BlcmF0b3IucHk=) | `85.41% <0%> (-2.09%)` | :arrow_down: | | [airflow/contrib/hooks/qubole\_hook.py](https://codecov.io/gh/apache/airflow/pull/4838/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3F1Ym9sZV9ob29rLnB5) | `51.93% <0%> (-1.73%)` | :arrow_down: | | [airflow/utils/log/wasb\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/4838/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvd2FzYl90YXNrX2hhbmRsZXIucHk=) | `0% <0%> (ø)` | :arrow_up: | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/4838/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.47% <0%> (+0.01%)` | :arrow_up: | | [airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/4838/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=) | `92.65% <0%> (+0.06%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4838?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4838?src=pr=footer). Last update [0b5cc71...d1a4059](https://codecov.io/gh/apache/airflow/pull/4838?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feng-tao commented on a change in pull request #4845: [AIRFLOW-4020] Remove viewer DAG edit permissions
feng-tao commented on a change in pull request #4845: [AIRFLOW-4020] Remove
viewer DAG edit permissions
URL: https://github.com/apache/airflow/pull/4845#discussion_r262696111
##
File path: tests/www/test_views.py
##
@@ -1566,6 +1566,14 @@ def test_tree_view_for_viewer(self):
resp = self.client.get(url, follow_redirects=True)
self.check_content_in_response('runme_1', resp)
+def test_gantt_failure_for_viewer(self):
+url = 'gantt?dag_id=example_bash_operator'
+self.logout()
+self.login(username='test_viewer',
+ password='test_viewer')
+resp = self.client.get(url, follow_redirects=True)
+self.check_content_not_in_response('example_bash_operator', resp)
Review comment:
sorry, wrong unit test. I changed to use /refresh endpoint instead.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-4021) Log file location in email alert incorrect
Kiran Gavali created AIRFLOW-4021:
-
Summary: Log file location in email alert incorrect
Key: AIRFLOW-4021
URL: https://issues.apache.org/jira/browse/AIRFLOW-4021
Project: Apache Airflow
Issue Type: Bug
Components: logging
Affects Versions: 1.10.1
Reporter: Kiran Gavali
Log files for task tries are stored in
{{base_log_dir/dag_id/task_id/execution_date_iso_format/try_number.log}}. On
task failure, the email alert that is sent specifies "Log file:" path as
{{base_log_dir/dag_id/task_id/execution_date_iso_format.log}}. Until
AIRFLOW-3909 is fixed, one would have to use the log file path on the server
mentioned in the email alert to view the log for the failed try. Related code
line is
[here|https://github.com/apache/airflow/blob/ad02179c28f1f93cbf55789b9ca8f3b9eeafeff7/airflow/models/__init__.py#L875].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] ashb commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details
ashb commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag
details
URL: https://github.com/apache/airflow/pull/4569#issuecomment-469861187
I'm sort of picturing something like (very pseudo code):
```python
from something import role_migrations as rm
def up():
p = rm.add_permission('all_dags', 'can_dag_edit')
rm.add_to_role('Op', p)
def down():
rm.remove_from_role('Op', 'all_dags', 'can_dag_edit')
```
I haven't worked out the details yet, still a bit hazy (and I'm not 100% on
the FAB data model around permissions, view menus, roles and all the
many-to-many tables between them.)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on issue #4819: [AIRFLOW-3997] Add getter to Variable that returns None instead of throwing
ashb commented on issue #4819: [AIRFLOW-3997] Add getter to Variable that returns None instead of throwing URL: https://github.com/apache/airflow/pull/4819#issuecomment-469859676 Could you update your commit message subject to "[AIRFLOW-3997] Add getter to Variable that returns None instead of throwing" please? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
