[GitHub] xnuinside commented on a change in pull request #4238: [AIRFLOW-987] pass kerberos cli args keytab and principal to kerberos…
xnuinside commented on a change in pull request #4238: [AIRFLOW-987] pass
kerberos cli args keytab and principal to kerberos…
URL: https://github.com/apache/incubator-airflow/pull/4238#discussion_r236970400
##
File path: tests/security/test_kerberos.py
##
@@ -32,13 +33,26 @@ def setUp(self):
if not configuration.conf.has_section("kerberos"):
configuration.conf.add_section("kerberos")
-
-configuration.conf.set("kerberos",
- "keytab",
+configuration.conf.set("kerberos", "keytab",
os.environ['KRB5_KTNAME'])
+keytab_from_cfg = configuration.conf.get("kerberos", "keytab")
+self.args = Namespace(keytab=keytab_from_cfg, principal=None, pid=None,
+ daemon=None, stdout=None, stderr=None,
log_file=None)
def test_renew_from_kt(self):
"""
We expect no result, but a successful run. No more TypeError
"""
-self.assertIsNone(renew_from_kt())
+self.assertIsNone(renew_from_kt(args=self.args))
+
+def test_args_from_cli(self):
+"""
+We expect no result, but a run with sys.exit(1) because keytab not
exist.
+"""
+configuration.conf.set("kerberos", "keytab", "")
+self.args.keytab = "test_keytab"
+
+with self.assertRaises(SystemExit) as se:
+renew_from_kt(args=self.args)
Review comment:
@ashb New in version 3.4. method added only in 3.4, at 2.7 will fail with
`with self.assertLogs(LoggingMixin().log) as log:
AttributeError: 'KerberosTest' object has no attribute 'assertLogs'`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] xnuinside commented on a change in pull request #4238: [AIRFLOW-987] pass kerberos cli args keytab and principal to kerberos…
xnuinside commented on a change in pull request #4238: [AIRFLOW-987] pass kerberos cli args keytab and principal to kerberos… URL: https://github.com/apache/incubator-airflow/pull/4238#discussion_r236967908 ## File path: airflow/security/kerberos.py ## @@ -80,7 +82,7 @@ def perform_krb181_workaround(): ret = subprocess.call(cmdv, close_fds=True) if ret != 0: -principal = "%s/%s" % ( +principal = args.principal or "%s/%s" % ( Review comment: oh, I understand what you talking about, I think you right, yes. My mistake. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4250: [AIRFLOW-XXX] Add VeeR VR to org list in README
codecov-io edited a comment on issue #4250: [AIRFLOW-XXX] Add VeeR VR to org list in README URL: https://github.com/apache/incubator-airflow/pull/4250#issuecomment-442318148 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4250?src=pr=h1) Report > Merging [#4250](https://codecov.io/gh/apache/incubator-airflow/pull/4250?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/2fd409d194b8a8dea73f8a62636795940407e84e?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4250?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#4250 +/- ## === Coverage 77.82% 77.82% === Files 201 201 Lines 1636716367 === Hits1273812738 Misses 3629 3629 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4250?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4250?src=pr=footer). Last update [2fd409d...7f73382](https://codecov.io/gh/apache/incubator-airflow/pull/4250?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] XD-DENG commented on issue #4250: [AIRFLOW-XXX] Add VeeR VR to org list in README
XD-DENG commented on issue #4250: [AIRFLOW-XXX] Add VeeR VR to org list in README URL: https://github.com/apache/incubator-airflow/pull/4250#issuecomment-442330411 @pishilong no worries, I forget it all the time too ;-) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] pishilong commented on issue #4250: [AIRFLOW-XXX] Add VeeR VR to org list in README
pishilong commented on issue #4250: [AIRFLOW-XXX] Add VeeR VR to org list in README URL: https://github.com/apache/incubator-airflow/pull/4250#issuecomment-442328750 @XD-DENG Sorry, I forgot add [ci skip] in commit message. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3405) Task instance fail intermittently due to MySQL error
[ https://issues.apache.org/jira/browse/AIRFLOW-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701411#comment-16701411 ] Yuvaraj commented on AIRFLOW-3405: -- Version 1.8.1 > Task instance fail intermittently due to MySQL error > > > Key: AIRFLOW-3405 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3405 > Project: Apache Airflow > Issue Type: Improvement > Environment: MySQL, Redhat Linux >Reporter: Yuvaraj >Priority: Major > Labels: performance, usability > > Dags are getting failed intermittently due to below error. > OperationalError: (_mysql_exceptions.OperationalError) (1040, 'Too many > connections') > [2018-11-25 12:24:16,952] - Heartbeat time limited exceeded! > We have max_connections defined as 2000 in DB. > Below are the setting in cfg. > sql_alchemy_pool_size = 1980 > sql_alchemy_pool_recycle = 3600 > As per DBA, The airflow scheduler keeps opening connections to the database, > these connections are mostly idle, they get reset whenever the scheduler > restarts but with max_connections at 2000 and scheduler holding on to 1600 of > these, other apps trying to connect might start running out of connections. > How do we remediate these idle connections. What should be the optimal value > for these configs and max_connections that to be set at DB. Consider we need > to build a large environment serving 500+ definitions with 1+ runs per > day. Need suggestions... > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] codecov-io edited a comment on issue #4250: [AIRFLOW-XXX] Add VeeR VR to org list in README
codecov-io edited a comment on issue #4250: [AIRFLOW-XXX] Add VeeR VR to org list in README URL: https://github.com/apache/incubator-airflow/pull/4250#issuecomment-442318148 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4250?src=pr=h1) Report > Merging [#4250](https://codecov.io/gh/apache/incubator-airflow/pull/4250?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/2fd409d194b8a8dea73f8a62636795940407e84e?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4250?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4250 +/- ## == - Coverage 77.82% 77.82% -0.01% == Files 201 201 Lines 1636716367 == - Hits1273812737 -1 - Misses 3629 3630 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4250?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4250/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `92.25% <0%> (-0.05%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4250?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4250?src=pr=footer). Last update [2fd409d...eca8cb4](https://codecov.io/gh/apache/incubator-airflow/pull/4250?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #4250: [AIRFLOW-XXX] Add VeeR VR to org list in README
codecov-io commented on issue #4250: [AIRFLOW-XXX] Add VeeR VR to org list in README URL: https://github.com/apache/incubator-airflow/pull/4250#issuecomment-442318148 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4250?src=pr=h1) Report > Merging [#4250](https://codecov.io/gh/apache/incubator-airflow/pull/4250?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/2fd409d194b8a8dea73f8a62636795940407e84e?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4250?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4250 +/- ## == - Coverage 77.82% 77.82% -0.01% == Files 201 201 Lines 1636716367 == - Hits1273812737 -1 - Misses 3629 3630 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4250?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4250/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `92.25% <0%> (-0.05%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4250?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4250?src=pr=footer). Last update [2fd409d...eca8cb4](https://codecov.io/gh/apache/incubator-airflow/pull/4250?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] XD-DENG commented on issue #4250: [AIRFLOW-XXX] Add VeeR VR to org list in README
XD-DENG commented on issue #4250: [AIRFLOW-XXX] Add VeeR VR to org list in README URL: https://github.com/apache/incubator-airflow/pull/4250#issuecomment-442315454 Hi @pishilong , the company list is following alphabetic order. In addition, given it's a simple doc change, you can add "[ci skip]" in your commit msg to skip Travis CI to save a few cents for ASF. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] pishilong opened a new pull request #4250: [AIRFLOW-XXX] Aadd VeeR VR to org list in README
pishilong opened a new pull request #4250: [AIRFLOW-XXX] Aadd VeeR VR to org
list in README
URL: https://github.com/apache/incubator-airflow/pull/4250
Make sure you have checked _all_ steps below.
### Jira
- [X ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [X ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ X] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ X] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [X ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [X ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-3410) Remote Host Identification Has Changed Error while using SSHOperator
[
https://issues.apache.org/jira/browse/AIRFLOW-3410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701220#comment-16701220
]
ASF GitHub Bot commented on AIRFLOW-3410:
-
kaxil opened a new pull request #4249: [AIRFLOW-3410] Add feature to allow Host
Key Change for SSH Op
URL: https://github.com/apache/incubator-airflow/pull/4249
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title.
- https://issues.apache.org/jira/browse/AIRFLOW-3410
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
This PR adds a feature to allow connecting to host (that already has an
entry in Known Host file with different key) that has a new IP. Currently, it
fails with `Remote Identification Changed`.
I have also added documentation on adding extras field to SSH Connection
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Remote Host Identification Has Changed Error while using SSHOperator
>
>
> Key: AIRFLOW-3410
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3410
> Project: Apache Airflow
> Issue Type: New Feature
> Components: contrib
>Reporter: Kaxil Naik
>Assignee: Kaxil Naik
>Priority: Minor
> Fix For: 1.10.2
>
>
> Currently, there is no provision in Airflow's implementation of SSHOperator
> to disable Host Checking and use a Null Known Host file to allow connecting
> to host that has a new IP.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] kaxil opened a new pull request #4249: [AIRFLOW-3410] Add feature to allow Host Key Change for SSH Op
kaxil opened a new pull request #4249: [AIRFLOW-3410] Add feature to allow Host
Key Change for SSH Op
URL: https://github.com/apache/incubator-airflow/pull/4249
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title.
- https://issues.apache.org/jira/browse/AIRFLOW-3410
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
This PR adds a feature to allow connecting to host (that already has an
entry in Known Host file with different key) that has a new IP. Currently, it
fails with `Remote Identification Changed`.
I have also added documentation on adding extras field to SSH Connection
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] kppullin opened a new pull request #4248: [AIRFLOW-3409] Build docker iamge with 'compile_assets' defined
kppullin opened a new pull request #4248: [AIRFLOW-3409] Build docker iamge
with 'compile_assets' defined
URL: https://github.com/apache/incubator-airflow/pull/4248
Make sure you have checked _all_ steps below.
### Jira
- [X] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3409
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [X] Here are some details about my PR, including screenshots of any UI
changes:
- Update Dockerfile to build the www assets so that the UI is functional.
### Tests
- [X] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
- N/A
### Commits
- [X] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [X] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- N/A
### Code Quality
- [X] Passes `flake8`
- N/A
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-3410) Remote Host Identification Has Changed Error while using SSHOperator
Kaxil Naik created AIRFLOW-3410: --- Summary: Remote Host Identification Has Changed Error while using SSHOperator Key: AIRFLOW-3410 URL: https://issues.apache.org/jira/browse/AIRFLOW-3410 Project: Apache Airflow Issue Type: New Feature Components: contrib Reporter: Kaxil Naik Assignee: Kaxil Naik Fix For: 1.10.2 Currently, there is no provision in Airflow's implementation of SSHOperator to disable Host Checking and use a Null Known Host file to allow connecting to host that has a new IP. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3409) Docker build script should build web assets (compile_assets).
Kevin Pullin created AIRFLOW-3409: - Summary: Docker build script should build web assets (compile_assets). Key: AIRFLOW-3409 URL: https://issues.apache.org/jira/browse/AIRFLOW-3409 Project: Apache Airflow Issue Type: Improvement Reporter: Kevin Pullin The docker container does not build the web assets and renders the UI unusable when deployed to a local minikube k8s instance. A minor patch to the Dockerfile is needed to build these assets: -python setup.py sdist -q +python setup.py compile_assets sdist -q -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] kppullin opened a new pull request #4247: [AIRFLOW-3402] Support global k8s affinity and toleration configs
kppullin opened a new pull request #4247: [AIRFLOW-3402] Support global k8s
affinity and toleration configs
URL: https://github.com/apache/incubator-airflow/pull/4247
Make sure you have checked _all_ steps below.
### Jira
- [X] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3402
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [X] Here are some details about my PR, including screenshots of any UI
changes:
- In the airflow.cfg file, adds new `affinity` and `tolerations` config keys
to the `kubernetes` section.
- The values for these are required to be in JSON format, as they are
dense.
- Wires up support for `tolerations` for pods generated via the
`KubernetesExecutor`.
- Minor updates to function documentation.
### Tests
- [X ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
- Tests added to `test_kubernetes_executor.py` to validate the generated
pod contains the defined affinity and tolerations config, for config coming
from both the `kube_config` and `kube_executor_config` dicts.
### Commits
- [X] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [X] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- Documentation added to the config file and inline in tests.
### Code Quality
- [X] Passes `flake8`
- These changes at least don't make things worse...
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-3402) Set default kubernetes affinity and toleration settings in airflow.cfg
[
https://issues.apache.org/jira/browse/AIRFLOW-3402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701190#comment-16701190
]
ASF GitHub Bot commented on AIRFLOW-3402:
-
kppullin opened a new pull request #4247: [AIRFLOW-3402] Support global k8s
affinity and toleration configs
URL: https://github.com/apache/incubator-airflow/pull/4247
Make sure you have checked _all_ steps below.
### Jira
- [X] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3402
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [X] Here are some details about my PR, including screenshots of any UI
changes:
- In the airflow.cfg file, adds new `affinity` and `tolerations` config keys
to the `kubernetes` section.
- The values for these are required to be in JSON format, as they are
dense.
- Wires up support for `tolerations` for pods generated via the
`KubernetesExecutor`.
- Minor updates to function documentation.
### Tests
- [X ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
- Tests added to `test_kubernetes_executor.py` to validate the generated
pod contains the defined affinity and tolerations config, for config coming
from both the `kube_config` and `kube_executor_config` dicts.
### Commits
- [X] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [X] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- Documentation added to the config file and inline in tests.
### Code Quality
- [X] Passes `flake8`
- These changes at least don't make things worse...
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Set default kubernetes affinity and toleration settings in airflow.cfg
> --
>
> Key: AIRFLOW-3402
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3402
> Project: Apache Airflow
> Issue Type: Improvement
> Components: kubernetes
>Reporter: Kevin Pullin
>Priority: Major
>
> Currently airflow supports setting kubernetes `affinity` and `toleration`
> configuration inside dags using either a `KubernetesExecutorConfig`
> definition or using the `KubernetesPodOperator`.
> In order to reduce having to set and maintain this configuration in every
> dag, it'd be useful to have the ability to set these globally in the
> airflow.cfg file. One use case is to force all kubernetes pods to run on a
> particular set of dedicated airflow nodes, which requires both affinity rules
> and tolerations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3407) BaseOperator and LoggingMixin do not call super().__init__
[
https://issues.apache.org/jira/browse/AIRFLOW-3407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
adam hitchcock updated AIRFLOW-3407:
Summary: BaseOperator and LoggingMixin do not call super().__init__ (was:
BaseOperator does not call super().__init__)
> BaseOperator and LoggingMixin do not call super().__init__
> --
>
> Key: AIRFLOW-3407
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3407
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: 1.10.1
>Reporter: adam hitchcock
>Priority: Major
>
> The {{BaseOperator}} is not necessarily the last class in the MRO; usually it
> is best practice to always call {{super().__init__(*args, **kwargs)}}
> to make sure that every class gets it chance to {{__init__}}.
> Is there a specific reason {{BaseOperator}} doesn't call super?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] codecov-io edited a comment on issue #4234: [AIRFLOW-2761] Parallelize enqueue in celery executor
codecov-io edited a comment on issue #4234: [AIRFLOW-2761] Parallelize enqueue in celery executor URL: https://github.com/apache/incubator-airflow/pull/4234#issuecomment-441964683 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4234?src=pr=h1) Report > Merging [#4234](https://codecov.io/gh/apache/incubator-airflow/pull/4234?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/2fd409d194b8a8dea73f8a62636795940407e84e?src=pr=desc) will **increase** coverage by `0.01%`. > The diff coverage is `82.48%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4234?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4234 +/- ## == + Coverage 77.82% 77.84% +0.01% == Files 201 201 Lines 1636716455 +88 == + Hits1273812809 +71 - Misses 3629 3646 +17 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4234?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/timeout.py](https://codecov.io/gh/apache/incubator-airflow/pull/4234/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy90aW1lb3V0LnB5) | `78.57% <100%> (+0.79%)` | :arrow_up: | | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4234/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `92.29% <100%> (ø)` | :arrow_up: | | [airflow/executors/base\_executor.py](https://codecov.io/gh/apache/incubator-airflow/pull/4234/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvYmFzZV9leGVjdXRvci5weQ==) | `95.16% <100%> (+1.41%)` | :arrow_up: | | [airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/incubator-airflow/pull/4234/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5) | `76.92% <68.51%> (-3.69%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/incubator-airflow/pull/4234/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `59.53% <86.11%> (+1.67%)` | :arrow_up: | | [airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/4234/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5) | `77.66% <93.93%> (+0.29%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4234?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4234?src=pr=footer). Last update [2fd409d...554b4cf](https://codecov.io/gh/apache/incubator-airflow/pull/4234?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] KevinYang21 commented on issue #3214: [AIRFLOW-1325] Add ElasticSearch log handler and reader
KevinYang21 commented on issue #3214: [AIRFLOW-1325] Add ElasticSearch log handler and reader URL: https://github.com/apache/incubator-airflow/pull/3214#issuecomment-442234773 @ashb sorry somehow I lost track to this conversation. I should have provided more details there my bad. So the compatibility changes needed are basically [these two method](https://github.com/apache/incubator-airflow/blob/master/tests/utils/log/elasticmock/fake_elasticsearch.py#L283-L295). It looks like the doc_type type issue has been fixed tho in [this commit](https://github.com/vrcmarcos/elasticmock/commit/8895aed57562f6ad7df817c352035eafc12aad60). I can experiment if we can just use the lib w/o modifications( maybe later this week or weekend). Tho this would mean we still depend on this package, is this what you wanted? Or you actually meant something else when saying `remove this vendored dep`? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] KevinYang21 edited a comment on issue #3214: [AIRFLOW-1325] Add ElasticSearch log handler and reader
KevinYang21 edited a comment on issue #3214: [AIRFLOW-1325] Add ElasticSearch log handler and reader URL: https://github.com/apache/incubator-airflow/pull/3214#issuecomment-442234773 @ashb sorry somehow I lost track to this conversation. I should have provided more details there my bad. So the compatibility changes needed are basically [this method](https://github.com/apache/incubator-airflow/blob/master/tests/utils/log/elasticmock/fake_elasticsearch.py#L283-L295). It looks like the doc_type type issue has been fixed tho in [this commit](https://github.com/vrcmarcos/elasticmock/commit/8895aed57562f6ad7df817c352035eafc12aad60). I can experiment if we can just use the lib w/o modifications( maybe later this week or weekend). Tho this would mean we still depend on this package, is this what you wanted? Or you actually meant something else when saying `remove this vendored dep`? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3164) verify certificate of LDAP server
[ https://issues.apache.org/jira/browse/AIRFLOW-3164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701011#comment-16701011 ] Bolke de Bruin commented on AIRFLOW-3164: - I agree that the backwards compatibility could have been handled better. We probably targeted 2.0 around the times this happened, but instead went for 1.10.1. However, we have given you the option. If you want to be insecure. Go ahead copy the auth and disable the check. It's not a big change. If big companies can find the time to maintain an insecure LDAP setup they can also delegate some time to add this to their setup (oh and my big company is a good client of yours if i'm correct ;) ). > verify certificate of LDAP server > - > > Key: AIRFLOW-3164 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3164 > Project: Apache Airflow > Issue Type: Bug >Reporter: Bolke de Bruin >Priority: Blocker > Fix For: 1.10.1 > > > Currently we dont verify the certificate of the Ldap server this can lead to > security incidents. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-3164) verify certificate of LDAP server
[ https://issues.apache.org/jira/browse/AIRFLOW-3164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700987#comment-16700987 ] Maciej Bryński edited comment on AIRFLOW-3164 at 11/27/18 8:47 PM: --- Still you can enable TLS option as default and give the ability to disable it. Auth backend it's not an option. It's a hack. Difficult to implement and even more difficult to maintain. Sometimes if you're working in bigger company changes like enabling TLS are not easy to process. Long story short: give users options and don't break backward compatibility in bugfix versions. was (Author: maver1ck): Still you can enable TLS option as default and give the ability to disable it. Auth backend it's not an option. It's a hack. Difficult to implement and even more difficult to maintain. Sometimes if you're working in bigger company changes like enabling TLS are not easy to process. Long story short: give users options and don't break your code in bugfix versions. > verify certificate of LDAP server > - > > Key: AIRFLOW-3164 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3164 > Project: Apache Airflow > Issue Type: Bug >Reporter: Bolke de Bruin >Priority: Blocker > Fix For: 1.10.1 > > > Currently we dont verify the certificate of the Ldap server this can lead to > security incidents. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-3164) verify certificate of LDAP server
[ https://issues.apache.org/jira/browse/AIRFLOW-3164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700987#comment-16700987 ] Maciej Bryński edited comment on AIRFLOW-3164 at 11/27/18 8:45 PM: --- Still you can enable TLS option as default and give the ability to disable it. Auth backend it's not an option. It's a hack. Difficult to implement and even more difficult to maintain. Sometimes if you're working in bigger company changes like enabling TLS are not easy to process. Long story short: give users options and don't break your code in bugfix versions. was (Author: maver1ck): Still you can enable TLS option as default and give the option to disable it. Auth backend it's not an option. It's a hack. Difficult to implement and even more difficult to maintain. Sometimes if you're working in bigger company changes like enabling TLS are not easy to process. Long story short: give users options and don't break your code in bugfix versions. > verify certificate of LDAP server > - > > Key: AIRFLOW-3164 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3164 > Project: Apache Airflow > Issue Type: Bug >Reporter: Bolke de Bruin >Priority: Blocker > Fix For: 1.10.1 > > > Currently we dont verify the certificate of the Ldap server this can lead to > security incidents. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3164) verify certificate of LDAP server
[ https://issues.apache.org/jira/browse/AIRFLOW-3164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700987#comment-16700987 ] Maciej Bryński commented on AIRFLOW-3164: - Still you can enable TLS option as default and give the option to disable it. Auth backend it's not an option. It's a hack. Difficult to implement and even more difficult to maintain. Sometimes if you're working in bigger company changes like enabling TLS are not easy to process. Long story short: give users options and don't break your code in bugfix versions. > verify certificate of LDAP server > - > > Key: AIRFLOW-3164 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3164 > Project: Apache Airflow > Issue Type: Bug >Reporter: Bolke de Bruin >Priority: Blocker > Fix For: 1.10.1 > > > Currently we dont verify the certificate of the Ldap server this can lead to > security incidents. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-3164) verify certificate of LDAP server
[ https://issues.apache.org/jira/browse/AIRFLOW-3164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700923#comment-16700923 ] Maciej Bryński edited comment on AIRFLOW-3164 at 11/27/18 8:01 PM: --- I know that LDAP without TLS transmits ever users password in plain text over the network where it could be sniffed. But I can have this transfer in my private intranet. If you want to have this secure - OK. But leave the option. >From my side I'll be telling stories how bugfix upgrade of Airflow can kill >your env. And this don't bring the trust into Airflow. And maybe you don't "officially" use SemVer but from user perspective it looks like SemVer. (anyway why you want to have 2.0.0 as next version if you're not using SemVer ?) About FAB If we move to FAB then I can use LDAP without TLS ? Because FAB has option if I want to use TLS for LDAP connection. https://github.com/dpgaspar/Flask-AppBuilder/blob/master/flask_appbuilder/security/manager.py#L190 So this is another point for leaving a choise to user. PS. I'm pretty sure that currently new UI have plenty of bugs and I don't want to be a guinea pig testing it. was (Author: maver1ck): I know that LDAP without TLS transmits ever users password in plain text over the network where it could be sniffed. But I can have this transfer in my private intranet. If you want to have this secure - OK. But leave the option. About FAB If we move to FAB then I can use LDAP without TLS ? Because FAB has option if I want to use TLS for LDAP connection. https://github.com/dpgaspar/Flask-AppBuilder/blob/master/flask_appbuilder/security/manager.py#L190 So this is another point for leaving a choise to user. PS. I'm pretty sure that currently new UI have plenty of bugs and I don't want to be a guinea pig testing it. > verify certificate of LDAP server > - > > Key: AIRFLOW-3164 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3164 > Project: Apache Airflow > Issue Type: Bug >Reporter: Bolke de Bruin >Priority: Blocker > Fix For: 1.10.1 > > > Currently we dont verify the certificate of the Ldap server this can lead to > security incidents. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3164) verify certificate of LDAP server
[ https://issues.apache.org/jira/browse/AIRFLOW-3164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700940#comment-16700940 ] Bolke de Bruin commented on AIRFLOW-3164: - Because the user in this respect can't really be trusted. We have already had reports of people leaving their Airflow installations wide open. We give you the choice by implementing your own auth backend but then you are really on your own. On your note on FAB's usage in Airflow. FAB still supports non TLS indeed, but we should maybe consider suggesting a patch that disables it. You have plenty of time to test it without being required to use it and you are not required to upgrade if you don't want. We just don't want to maintain two UIs side by side. Long story short: enable TLS on your LDAP server it is not hard to do and it is best practice. There is no reason not to. > verify certificate of LDAP server > - > > Key: AIRFLOW-3164 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3164 > Project: Apache Airflow > Issue Type: Bug >Reporter: Bolke de Bruin >Priority: Blocker > Fix For: 1.10.1 > > > Currently we dont verify the certificate of the Ldap server this can lead to > security incidents. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-3164) verify certificate of LDAP server
[ https://issues.apache.org/jira/browse/AIRFLOW-3164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700923#comment-16700923 ] Maciej Bryński edited comment on AIRFLOW-3164 at 11/27/18 8:02 PM: --- I know that LDAP without TLS transmits ever users password in plain text over the network where it could be sniffed. But I can have this transfer in my private intranet. If you want to have this secure - OK. But leave the option. >From my side I'll be telling stories how bugfix upgrade of Airflow can kill >your env. And this don't bring the trust into Airflow. And maybe you don't "officially" use SemVer but from user perspective it looks like SemVer. (anyway why you want to have 2.0.0 as next version if you're not using SemVer ?) About FAB If we move to FAB then I can use LDAP without TLS ? Because FAB has option if I want to use TLS for LDAP connection. https://github.com/dpgaspar/Flask-AppBuilder/blob/master/flask_appbuilder/security/manager.py#L190 So this is another point for leaving a choise to user. PS. I'm pretty sure that currently new FAB UI have plenty of bugs and I don't want to be a guinea pig testing it. was (Author: maver1ck): I know that LDAP without TLS transmits ever users password in plain text over the network where it could be sniffed. But I can have this transfer in my private intranet. If you want to have this secure - OK. But leave the option. >From my side I'll be telling stories how bugfix upgrade of Airflow can kill >your env. And this don't bring the trust into Airflow. And maybe you don't "officially" use SemVer but from user perspective it looks like SemVer. (anyway why you want to have 2.0.0 as next version if you're not using SemVer ?) About FAB If we move to FAB then I can use LDAP without TLS ? Because FAB has option if I want to use TLS for LDAP connection. https://github.com/dpgaspar/Flask-AppBuilder/blob/master/flask_appbuilder/security/manager.py#L190 So this is another point for leaving a choise to user. PS. I'm pretty sure that currently new UI have plenty of bugs and I don't want to be a guinea pig testing it. > verify certificate of LDAP server > - > > Key: AIRFLOW-3164 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3164 > Project: Apache Airflow > Issue Type: Bug >Reporter: Bolke de Bruin >Priority: Blocker > Fix For: 1.10.1 > > > Currently we dont verify the certificate of the Ldap server this can lead to > security incidents. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3164) verify certificate of LDAP server
[ https://issues.apache.org/jira/browse/AIRFLOW-3164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700923#comment-16700923 ] Maciej Bryński commented on AIRFLOW-3164: - I know that LDAP without TLS transmits ever users password in plain text over the network where it could be sniffed. But I can have this transfer in my private intranet. If you want to have this secure - OK. But leave the option. About FAB If we move to FAB then I can use LDAP without TLS ? Because FAB has option if I want to use TLS for LDAP connection. https://github.com/dpgaspar/Flask-AppBuilder/blob/master/flask_appbuilder/security/manager.py#L190 So this is another point for leaving a choise to user. PS. I'm pretty sure that currently new UI have plenty of bugs and I don't want to be a guinea pig testing it. > verify certificate of LDAP server > - > > Key: AIRFLOW-3164 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3164 > Project: Apache Airflow > Issue Type: Bug >Reporter: Bolke de Bruin >Priority: Blocker > Fix For: 1.10.1 > > > Currently we dont verify the certificate of the Ldap server this can lead to > security incidents. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] kaxil closed pull request #4246: [AIRFLOW-XXX] Replace airflow with apache-airflow
kaxil closed pull request #4246: [AIRFLOW-XXX] Replace airflow with apache-airflow URL: https://github.com/apache/incubator-airflow/pull/4246 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/README.md b/README.md index 6903bf2a8a..1176031984 100644 --- a/README.md +++ b/README.md @@ -27,8 +27,6 @@ under the License. [](https://pypi.org/project/apache-airflow/) [](https://twitter.com/ApacheAirflow) -_NOTE: The transition from 1.8.0 (or before) to 1.8.1 (or after) requires uninstalling Apache Airflow before installing the new version. The package name was changed from `airflow` to `apache-airflow` as of version 1.8.1._ - Apache Airflow (or simply Airflow) is a platform to programmatically author, schedule, and monitor workflows. When workflows are defined as code, they become more maintainable, diff --git a/docs/security.rst b/docs/security.rst index 1adefcd834..ea745836ac 100644 --- a/docs/security.rst +++ b/docs/security.rst @@ -258,7 +258,7 @@ To use kerberos authentication, you must install Airflow with the `kerberos` ext .. code-block:: base - pip install airflow[kerberos] + pip install apache-airflow[kerberos] OAuth Authentication @@ -291,7 +291,7 @@ To use GHE authentication, you must install Airflow with the `github_enterprise` .. code-block:: base - pip install airflow[github_enterprise] + pip install apache-airflow[github_enterprise] Setting up GHE Authentication ^ @@ -339,7 +339,7 @@ To use Google authentication, you must install Airflow with the `google_auth` ex .. code-block:: base - pip install airflow[google_auth] + pip install apache-airflow[google_auth] Setting up Google Authentication This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-3407) BaseOperator does not call super().__init__
[
https://issues.apache.org/jira/browse/AIRFLOW-3407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
adam hitchcock updated AIRFLOW-3407:
Description:
The {{BaseOperator}} is not necessarily the last class in the MRO; usually it
is best practice to always call {{super().__init__(*args, **kwargs)}}
to make sure that every class gets it chance to {{__init__}}.
Is there a specific reason {{BaseOperator}} doesn't call super?
was:
The {{BaseOperator}} is not necessarily the last class in the MRO; usually it
is best practice to always call {{super().__init__(*args, **kwargs)}} to make
sure that every class gets it chance to {{__init__}}.
Is there a specific reason {{BaseOperator}} doesn't call super?
> BaseOperator does not call super().__init__
> ---
>
> Key: AIRFLOW-3407
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3407
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: 1.10.1
>Reporter: adam hitchcock
>Priority: Major
>
> The {{BaseOperator}} is not necessarily the last class in the MRO; usually it
> is best practice to always call {{super().__init__(*args, **kwargs)}}
> to make sure that every class gets it chance to {{__init__}}.
> Is there a specific reason {{BaseOperator}} doesn't call super?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-3407) BaseOperator does not call super().__init__
adam hitchcock created AIRFLOW-3407:
---
Summary: BaseOperator does not call super().__init__
Key: AIRFLOW-3407
URL: https://issues.apache.org/jira/browse/AIRFLOW-3407
Project: Apache Airflow
Issue Type: Bug
Components: operators
Affects Versions: 1.10.1
Reporter: adam hitchcock
The {{BaseOperator}} is not necessarily the last class in the MRO; usually it
is best practice to always call {{super().__init__(*args, **kwargs)}} to make
sure that every class gets it chance to {{__init__}}.
Is there a specific reason {{BaseOperator}} doesn't call super?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] ashb commented on issue #4174: [AIRFLOW-571] Airflow CLI: add gunicorn_config param and refactor webserver cli function
ashb commented on issue #4174: [AIRFLOW-571] Airflow CLI: add gunicorn_config param and refactor webserver cli function URL: https://github.com/apache/incubator-airflow/pull/4174#issuecomment-442169985 Sorry - got way-laid by the release management of 1.10.1. I'll try and take another look at this tomorrow or Thursday This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb commented on a change in pull request #4238: [AIRFLOW-987] pass kerberos cli args keytab and principal to kerberos…
ashb commented on a change in pull request #4238: [AIRFLOW-987] pass kerberos cli args keytab and principal to kerberos… URL: https://github.com/apache/incubator-airflow/pull/4238#discussion_r236793244 ## File path: airflow/security/kerberos.py ## @@ -80,7 +82,7 @@ def perform_krb181_workaround(): ret = subprocess.call(cmdv, close_fds=True) if ret != 0: -principal = "%s/%s" % ( +principal = args.principal or "%s/%s" % ( Review comment: This is a change in behaviour. Previously: ``` principal == "the-actual-principal/my-fqdn" ``` After if I specify the arg: ``` principal == "the-actual-principal" ``` i.e. if arg.principal is provided the format is never called, and I think it should be? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb commented on a change in pull request #4238: [AIRFLOW-987] pass kerberos cli args keytab and principal to kerberos…
ashb commented on a change in pull request #4238: [AIRFLOW-987] pass kerberos
cli args keytab and principal to kerberos…
URL: https://github.com/apache/incubator-airflow/pull/4238#discussion_r236791874
##
File path: airflow/security/kerberos.py
##
@@ -110,11 +112,11 @@ def detect_conf_var():
return b'X-CACHECONF:' in f.read()
-def run():
-if configuration.conf.get('kerberos', 'keytab') is None:
+def run(args):
+if not args.keytab:
Review comment:
Ah I missed that, thanks.
Does it come out as None or `""`? (Oh shouldn't matter for `if not`)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-3406) Implement an Azure CosmosDB operator
Tom Miller created AIRFLOW-3406: --- Summary: Implement an Azure CosmosDB operator Key: AIRFLOW-3406 URL: https://issues.apache.org/jira/browse/AIRFLOW-3406 Project: Apache Airflow Issue Type: New Feature Reporter: Tom Miller Assignee: Tom Miller -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Issue Comment Deleted] (AIRFLOW-3351) Airflow webserver intermitent broken
[
https://issues.apache.org/jira/browse/AIRFLOW-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor updated AIRFLOW-3351:
---
Comment: was deleted
(was: [harik@gmail.com|mailto:harik@gmail.com] is an email id, please
send invitation.)
> Airflow webserver intermitent broken
>
>
> Key: AIRFLOW-3351
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3351
> Project: Apache Airflow
> Issue Type: Bug
> Components: authentication
>Affects Versions: 1.10.0
>Reporter: Hari Krishna ADDEPALLI LN
>Priority: Blocker
>
> After completing the airflow 1.10.0 integration with LDAP anonymously
> (AIRFLOW-3270), we started to hit "Internal Server Error" with below
> exception stack, we tried to clean up the browser cache, it sometimes works
> and sometimes error our. Please advise the resolution to avoid this issue.
>
> {code:java}
> During handling of the above exception, another exception occurred: Traceback
> (most recent call last): File
> "/usr/local/lib/python3.5/site-packages/gunicorn/workers/sync.py", line 135,
> in handle self.handle_request(listener, req, client, addr) File
> "/usr/local/lib/python3.5/site-packages/gunicorn/workers/sync.py", line 176,
> in handle_request respiter = self.wsgi(environ, resp.start_response)
> File "/usr/local/lib/python3.5/site-packages/werkzeug/wsgi.py", line 826, in
> __call__ return app(environ, start_response) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1997, in __call__
> return self.wsgi_app(environ, start_response) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1985, in wsgi_app
> response = self.handle_exception(e) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1547, in
> handle_exception return self.finalize_request(handler(e),
> from_error_handler=True) File
> "/usr/local/lib/python3.5/site-packages/airflow/www/views.py", line 716, in
> show_traceback info=traceback.format_exc()), 500 File
> "/usr/local/lib/python3.5/site-packages/flask/templating.py", line 132, in
> render_template ctx.app.update_template_context(context) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 764, in
> update_template_context context.update(func()) File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 825, in
> _user_context_processor return dict(current_user=_get_user()) File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 794, in
> _get_user current_app.login_manager._load_user() File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 363, in
> _load_user return self.reload_user() File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 325, in
> reload_user user = self.user_callback(user_id) File
> "/usr/local/lib/python3.5/site-packages/airflow/utils/db.py", line 74, in
> wrapper return func(*args, **kwargs) File
> "/usr/local/lib/python3.5/site-packages/airflow/contrib/auth/backends/ldap_auth.py",
> line 278, in load_user return LdapUser(user) File "", line 4,
> in __init__ File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/state.py", line 414,
> in _initialize_instance manager.dispatch.init_failure(self, args, kwargs)
> File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/util/langhelpers.py", line
> 66, in __exit__ compat.reraise(exc_type, exc_value, exc_tb) File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 187,
> in reraise raise value File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/state.py", line 411,
> in _initialize_instance return manager.original_init(*mixed[1:],
> **kwargs) File
> "/usr/local/lib/python3.5/site-packages/airflow/contrib/auth/backends/ldap_auth.py",
> line 157, in __init__ user.username) AttributeError: 'NoneType' object
> has no attribute 'username' 127.0.0.1 - - [15/Nov/2018:10:47:31 +] "GET
> /admin/ HTTP/1.1" 500 0 "-" "-" [2018-11-15 10:47:38,590] ERROR in app:
> Exception on /favicon.ico [GET] Traceback (most recent call last): File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1612, in
> full_dispatch_request rv = self.dispatch_request() File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1590, in
> dispatch_request self.raise_routing_exception(req) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1573, in
> raise_routing_exception raise request.routing_exception File
> "/usr/local/lib/python3.5/site-packages/flask/ctx.py", line 294, in
> match_request self.url_adapter.match(return_rule=True) File
> "/usr/local/lib/python3.5/site-packages/werkzeug/routing.py", line 1581, in
> match
[jira] [Commented] (AIRFLOW-3351) Airflow webserver intermitent broken
[
https://issues.apache.org/jira/browse/AIRFLOW-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700810#comment-16700810
]
Ash Berlin-Taylor commented on AIRFLOW-3351:
Sent.
> Airflow webserver intermitent broken
>
>
> Key: AIRFLOW-3351
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3351
> Project: Apache Airflow
> Issue Type: Bug
> Components: authentication
>Affects Versions: 1.10.0
>Reporter: Hari Krishna ADDEPALLI LN
>Priority: Blocker
>
> After completing the airflow 1.10.0 integration with LDAP anonymously
> (AIRFLOW-3270), we started to hit "Internal Server Error" with below
> exception stack, we tried to clean up the browser cache, it sometimes works
> and sometimes error our. Please advise the resolution to avoid this issue.
>
> {code:java}
> During handling of the above exception, another exception occurred: Traceback
> (most recent call last): File
> "/usr/local/lib/python3.5/site-packages/gunicorn/workers/sync.py", line 135,
> in handle self.handle_request(listener, req, client, addr) File
> "/usr/local/lib/python3.5/site-packages/gunicorn/workers/sync.py", line 176,
> in handle_request respiter = self.wsgi(environ, resp.start_response)
> File "/usr/local/lib/python3.5/site-packages/werkzeug/wsgi.py", line 826, in
> __call__ return app(environ, start_response) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1997, in __call__
> return self.wsgi_app(environ, start_response) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1985, in wsgi_app
> response = self.handle_exception(e) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1547, in
> handle_exception return self.finalize_request(handler(e),
> from_error_handler=True) File
> "/usr/local/lib/python3.5/site-packages/airflow/www/views.py", line 716, in
> show_traceback info=traceback.format_exc()), 500 File
> "/usr/local/lib/python3.5/site-packages/flask/templating.py", line 132, in
> render_template ctx.app.update_template_context(context) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 764, in
> update_template_context context.update(func()) File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 825, in
> _user_context_processor return dict(current_user=_get_user()) File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 794, in
> _get_user current_app.login_manager._load_user() File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 363, in
> _load_user return self.reload_user() File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 325, in
> reload_user user = self.user_callback(user_id) File
> "/usr/local/lib/python3.5/site-packages/airflow/utils/db.py", line 74, in
> wrapper return func(*args, **kwargs) File
> "/usr/local/lib/python3.5/site-packages/airflow/contrib/auth/backends/ldap_auth.py",
> line 278, in load_user return LdapUser(user) File "", line 4,
> in __init__ File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/state.py", line 414,
> in _initialize_instance manager.dispatch.init_failure(self, args, kwargs)
> File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/util/langhelpers.py", line
> 66, in __exit__ compat.reraise(exc_type, exc_value, exc_tb) File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 187,
> in reraise raise value File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/state.py", line 411,
> in _initialize_instance return manager.original_init(*mixed[1:],
> **kwargs) File
> "/usr/local/lib/python3.5/site-packages/airflow/contrib/auth/backends/ldap_auth.py",
> line 157, in __init__ user.username) AttributeError: 'NoneType' object
> has no attribute 'username' 127.0.0.1 - - [15/Nov/2018:10:47:31 +] "GET
> /admin/ HTTP/1.1" 500 0 "-" "-" [2018-11-15 10:47:38,590] ERROR in app:
> Exception on /favicon.ico [GET] Traceback (most recent call last): File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1612, in
> full_dispatch_request rv = self.dispatch_request() File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1590, in
> dispatch_request self.raise_routing_exception(req) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1573, in
> raise_routing_exception raise request.routing_exception File
> "/usr/local/lib/python3.5/site-packages/flask/ctx.py", line 294, in
> match_request self.url_adapter.match(return_rule=True) File
> "/usr/local/lib/python3.5/site-packages/werkzeug/routing.py", line 1581, in
> match raise NotFound() werkzeug.exceptions.NotFound: 404 Not Found: The
>
[jira] [Commented] (AIRFLOW-3351) Airflow webserver intermitent broken
[
https://issues.apache.org/jira/browse/AIRFLOW-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700801#comment-16700801
]
Ash Berlin-Taylor commented on AIRFLOW-3351:
I will need an email address to send the invite too.
> Airflow webserver intermitent broken
>
>
> Key: AIRFLOW-3351
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3351
> Project: Apache Airflow
> Issue Type: Bug
> Components: authentication
>Affects Versions: 1.10.0
>Reporter: Hari Krishna ADDEPALLI LN
>Priority: Blocker
>
> After completing the airflow 1.10.0 integration with LDAP anonymously
> (AIRFLOW-3270), we started to hit "Internal Server Error" with below
> exception stack, we tried to clean up the browser cache, it sometimes works
> and sometimes error our. Please advise the resolution to avoid this issue.
>
> {code:java}
> During handling of the above exception, another exception occurred: Traceback
> (most recent call last): File
> "/usr/local/lib/python3.5/site-packages/gunicorn/workers/sync.py", line 135,
> in handle self.handle_request(listener, req, client, addr) File
> "/usr/local/lib/python3.5/site-packages/gunicorn/workers/sync.py", line 176,
> in handle_request respiter = self.wsgi(environ, resp.start_response)
> File "/usr/local/lib/python3.5/site-packages/werkzeug/wsgi.py", line 826, in
> __call__ return app(environ, start_response) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1997, in __call__
> return self.wsgi_app(environ, start_response) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1985, in wsgi_app
> response = self.handle_exception(e) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1547, in
> handle_exception return self.finalize_request(handler(e),
> from_error_handler=True) File
> "/usr/local/lib/python3.5/site-packages/airflow/www/views.py", line 716, in
> show_traceback info=traceback.format_exc()), 500 File
> "/usr/local/lib/python3.5/site-packages/flask/templating.py", line 132, in
> render_template ctx.app.update_template_context(context) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 764, in
> update_template_context context.update(func()) File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 825, in
> _user_context_processor return dict(current_user=_get_user()) File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 794, in
> _get_user current_app.login_manager._load_user() File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 363, in
> _load_user return self.reload_user() File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 325, in
> reload_user user = self.user_callback(user_id) File
> "/usr/local/lib/python3.5/site-packages/airflow/utils/db.py", line 74, in
> wrapper return func(*args, **kwargs) File
> "/usr/local/lib/python3.5/site-packages/airflow/contrib/auth/backends/ldap_auth.py",
> line 278, in load_user return LdapUser(user) File "", line 4,
> in __init__ File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/state.py", line 414,
> in _initialize_instance manager.dispatch.init_failure(self, args, kwargs)
> File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/util/langhelpers.py", line
> 66, in __exit__ compat.reraise(exc_type, exc_value, exc_tb) File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 187,
> in reraise raise value File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/state.py", line 411,
> in _initialize_instance return manager.original_init(*mixed[1:],
> **kwargs) File
> "/usr/local/lib/python3.5/site-packages/airflow/contrib/auth/backends/ldap_auth.py",
> line 157, in __init__ user.username) AttributeError: 'NoneType' object
> has no attribute 'username' 127.0.0.1 - - [15/Nov/2018:10:47:31 +] "GET
> /admin/ HTTP/1.1" 500 0 "-" "-" [2018-11-15 10:47:38,590] ERROR in app:
> Exception on /favicon.ico [GET] Traceback (most recent call last): File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1612, in
> full_dispatch_request rv = self.dispatch_request() File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1590, in
> dispatch_request self.raise_routing_exception(req) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1573, in
> raise_routing_exception raise request.routing_exception File
> "/usr/local/lib/python3.5/site-packages/flask/ctx.py", line 294, in
> match_request self.url_adapter.match(return_rule=True) File
> "/usr/local/lib/python3.5/site-packages/werkzeug/routing.py", line 1581, in
> match raise NotFound()
[jira] [Commented] (AIRFLOW-3351) Airflow webserver intermitent broken
[
https://issues.apache.org/jira/browse/AIRFLOW-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700798#comment-16700798
]
Hari Krishna ADDEPALLI LN commented on AIRFLOW-3351:
I am still unable to sign in, can you please send me invitation [~ashb] ?
> Airflow webserver intermitent broken
>
>
> Key: AIRFLOW-3351
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3351
> Project: Apache Airflow
> Issue Type: Bug
> Components: authentication
>Affects Versions: 1.10.0
>Reporter: Hari Krishna ADDEPALLI LN
>Priority: Blocker
>
> After completing the airflow 1.10.0 integration with LDAP anonymously
> (AIRFLOW-3270), we started to hit "Internal Server Error" with below
> exception stack, we tried to clean up the browser cache, it sometimes works
> and sometimes error our. Please advise the resolution to avoid this issue.
>
> {code:java}
> During handling of the above exception, another exception occurred: Traceback
> (most recent call last): File
> "/usr/local/lib/python3.5/site-packages/gunicorn/workers/sync.py", line 135,
> in handle self.handle_request(listener, req, client, addr) File
> "/usr/local/lib/python3.5/site-packages/gunicorn/workers/sync.py", line 176,
> in handle_request respiter = self.wsgi(environ, resp.start_response)
> File "/usr/local/lib/python3.5/site-packages/werkzeug/wsgi.py", line 826, in
> __call__ return app(environ, start_response) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1997, in __call__
> return self.wsgi_app(environ, start_response) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1985, in wsgi_app
> response = self.handle_exception(e) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1547, in
> handle_exception return self.finalize_request(handler(e),
> from_error_handler=True) File
> "/usr/local/lib/python3.5/site-packages/airflow/www/views.py", line 716, in
> show_traceback info=traceback.format_exc()), 500 File
> "/usr/local/lib/python3.5/site-packages/flask/templating.py", line 132, in
> render_template ctx.app.update_template_context(context) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 764, in
> update_template_context context.update(func()) File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 825, in
> _user_context_processor return dict(current_user=_get_user()) File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 794, in
> _get_user current_app.login_manager._load_user() File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 363, in
> _load_user return self.reload_user() File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 325, in
> reload_user user = self.user_callback(user_id) File
> "/usr/local/lib/python3.5/site-packages/airflow/utils/db.py", line 74, in
> wrapper return func(*args, **kwargs) File
> "/usr/local/lib/python3.5/site-packages/airflow/contrib/auth/backends/ldap_auth.py",
> line 278, in load_user return LdapUser(user) File "", line 4,
> in __init__ File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/state.py", line 414,
> in _initialize_instance manager.dispatch.init_failure(self, args, kwargs)
> File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/util/langhelpers.py", line
> 66, in __exit__ compat.reraise(exc_type, exc_value, exc_tb) File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 187,
> in reraise raise value File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/state.py", line 411,
> in _initialize_instance return manager.original_init(*mixed[1:],
> **kwargs) File
> "/usr/local/lib/python3.5/site-packages/airflow/contrib/auth/backends/ldap_auth.py",
> line 157, in __init__ user.username) AttributeError: 'NoneType' object
> has no attribute 'username' 127.0.0.1 - - [15/Nov/2018:10:47:31 +] "GET
> /admin/ HTTP/1.1" 500 0 "-" "-" [2018-11-15 10:47:38,590] ERROR in app:
> Exception on /favicon.ico [GET] Traceback (most recent call last): File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1612, in
> full_dispatch_request rv = self.dispatch_request() File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1590, in
> dispatch_request self.raise_routing_exception(req) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1573, in
> raise_routing_exception raise request.routing_exception File
> "/usr/local/lib/python3.5/site-packages/flask/ctx.py", line 294, in
> match_request self.url_adapter.match(return_rule=True) File
> "/usr/local/lib/python3.5/site-packages/werkzeug/routing.py", line 1581,
[jira] [Commented] (AIRFLOW-3351) Airflow webserver intermitent broken
[
https://issues.apache.org/jira/browse/AIRFLOW-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700803#comment-16700803
]
Hari Krishna ADDEPALLI LN commented on AIRFLOW-3351:
[harik@gmail.com|mailto:harik@gmail.com] is an email id, please send
invitation.
> Airflow webserver intermitent broken
>
>
> Key: AIRFLOW-3351
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3351
> Project: Apache Airflow
> Issue Type: Bug
> Components: authentication
>Affects Versions: 1.10.0
>Reporter: Hari Krishna ADDEPALLI LN
>Priority: Blocker
>
> After completing the airflow 1.10.0 integration with LDAP anonymously
> (AIRFLOW-3270), we started to hit "Internal Server Error" with below
> exception stack, we tried to clean up the browser cache, it sometimes works
> and sometimes error our. Please advise the resolution to avoid this issue.
>
> {code:java}
> During handling of the above exception, another exception occurred: Traceback
> (most recent call last): File
> "/usr/local/lib/python3.5/site-packages/gunicorn/workers/sync.py", line 135,
> in handle self.handle_request(listener, req, client, addr) File
> "/usr/local/lib/python3.5/site-packages/gunicorn/workers/sync.py", line 176,
> in handle_request respiter = self.wsgi(environ, resp.start_response)
> File "/usr/local/lib/python3.5/site-packages/werkzeug/wsgi.py", line 826, in
> __call__ return app(environ, start_response) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1997, in __call__
> return self.wsgi_app(environ, start_response) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1985, in wsgi_app
> response = self.handle_exception(e) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1547, in
> handle_exception return self.finalize_request(handler(e),
> from_error_handler=True) File
> "/usr/local/lib/python3.5/site-packages/airflow/www/views.py", line 716, in
> show_traceback info=traceback.format_exc()), 500 File
> "/usr/local/lib/python3.5/site-packages/flask/templating.py", line 132, in
> render_template ctx.app.update_template_context(context) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 764, in
> update_template_context context.update(func()) File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 825, in
> _user_context_processor return dict(current_user=_get_user()) File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 794, in
> _get_user current_app.login_manager._load_user() File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 363, in
> _load_user return self.reload_user() File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 325, in
> reload_user user = self.user_callback(user_id) File
> "/usr/local/lib/python3.5/site-packages/airflow/utils/db.py", line 74, in
> wrapper return func(*args, **kwargs) File
> "/usr/local/lib/python3.5/site-packages/airflow/contrib/auth/backends/ldap_auth.py",
> line 278, in load_user return LdapUser(user) File "", line 4,
> in __init__ File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/state.py", line 414,
> in _initialize_instance manager.dispatch.init_failure(self, args, kwargs)
> File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/util/langhelpers.py", line
> 66, in __exit__ compat.reraise(exc_type, exc_value, exc_tb) File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 187,
> in reraise raise value File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/state.py", line 411,
> in _initialize_instance return manager.original_init(*mixed[1:],
> **kwargs) File
> "/usr/local/lib/python3.5/site-packages/airflow/contrib/auth/backends/ldap_auth.py",
> line 157, in __init__ user.username) AttributeError: 'NoneType' object
> has no attribute 'username' 127.0.0.1 - - [15/Nov/2018:10:47:31 +] "GET
> /admin/ HTTP/1.1" 500 0 "-" "-" [2018-11-15 10:47:38,590] ERROR in app:
> Exception on /favicon.ico [GET] Traceback (most recent call last): File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1612, in
> full_dispatch_request rv = self.dispatch_request() File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1590, in
> dispatch_request self.raise_routing_exception(req) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1573, in
> raise_routing_exception raise request.routing_exception File
> "/usr/local/lib/python3.5/site-packages/flask/ctx.py", line 294, in
> match_request self.url_adapter.match(return_rule=True) File
> "/usr/local/lib/python3.5/site-packages/werkzeug/routing.py",
[jira] [Commented] (AIRFLOW-3351) Airflow webserver intermitent broken
[
https://issues.apache.org/jira/browse/AIRFLOW-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700793#comment-16700793
]
Ash Berlin-Taylor commented on AIRFLOW-3351:
You can get an invite sent via http://apache-airflow.slack.com/
> Airflow webserver intermitent broken
>
>
> Key: AIRFLOW-3351
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3351
> Project: Apache Airflow
> Issue Type: Bug
> Components: authentication
>Affects Versions: 1.10.0
>Reporter: Hari Krishna ADDEPALLI LN
>Priority: Blocker
>
> After completing the airflow 1.10.0 integration with LDAP anonymously
> (AIRFLOW-3270), we started to hit "Internal Server Error" with below
> exception stack, we tried to clean up the browser cache, it sometimes works
> and sometimes error our. Please advise the resolution to avoid this issue.
>
> {code:java}
> During handling of the above exception, another exception occurred: Traceback
> (most recent call last): File
> "/usr/local/lib/python3.5/site-packages/gunicorn/workers/sync.py", line 135,
> in handle self.handle_request(listener, req, client, addr) File
> "/usr/local/lib/python3.5/site-packages/gunicorn/workers/sync.py", line 176,
> in handle_request respiter = self.wsgi(environ, resp.start_response)
> File "/usr/local/lib/python3.5/site-packages/werkzeug/wsgi.py", line 826, in
> __call__ return app(environ, start_response) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1997, in __call__
> return self.wsgi_app(environ, start_response) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1985, in wsgi_app
> response = self.handle_exception(e) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1547, in
> handle_exception return self.finalize_request(handler(e),
> from_error_handler=True) File
> "/usr/local/lib/python3.5/site-packages/airflow/www/views.py", line 716, in
> show_traceback info=traceback.format_exc()), 500 File
> "/usr/local/lib/python3.5/site-packages/flask/templating.py", line 132, in
> render_template ctx.app.update_template_context(context) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 764, in
> update_template_context context.update(func()) File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 825, in
> _user_context_processor return dict(current_user=_get_user()) File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 794, in
> _get_user current_app.login_manager._load_user() File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 363, in
> _load_user return self.reload_user() File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 325, in
> reload_user user = self.user_callback(user_id) File
> "/usr/local/lib/python3.5/site-packages/airflow/utils/db.py", line 74, in
> wrapper return func(*args, **kwargs) File
> "/usr/local/lib/python3.5/site-packages/airflow/contrib/auth/backends/ldap_auth.py",
> line 278, in load_user return LdapUser(user) File "", line 4,
> in __init__ File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/state.py", line 414,
> in _initialize_instance manager.dispatch.init_failure(self, args, kwargs)
> File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/util/langhelpers.py", line
> 66, in __exit__ compat.reraise(exc_type, exc_value, exc_tb) File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 187,
> in reraise raise value File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/state.py", line 411,
> in _initialize_instance return manager.original_init(*mixed[1:],
> **kwargs) File
> "/usr/local/lib/python3.5/site-packages/airflow/contrib/auth/backends/ldap_auth.py",
> line 157, in __init__ user.username) AttributeError: 'NoneType' object
> has no attribute 'username' 127.0.0.1 - - [15/Nov/2018:10:47:31 +] "GET
> /admin/ HTTP/1.1" 500 0 "-" "-" [2018-11-15 10:47:38,590] ERROR in app:
> Exception on /favicon.ico [GET] Traceback (most recent call last): File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1612, in
> full_dispatch_request rv = self.dispatch_request() File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1590, in
> dispatch_request self.raise_routing_exception(req) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1573, in
> raise_routing_exception raise request.routing_exception File
> "/usr/local/lib/python3.5/site-packages/flask/ctx.py", line 294, in
> match_request self.url_adapter.match(return_rule=True) File
> "/usr/local/lib/python3.5/site-packages/werkzeug/routing.py", line 1581, in
> match raise
[jira] [Comment Edited] (AIRFLOW-3351) Airflow webserver intermitent broken
[
https://issues.apache.org/jira/browse/AIRFLOW-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700786#comment-16700786
]
Hari Krishna ADDEPALLI LN edited comment on AIRFLOW-3351 at 11/27/18 5:45 PM:
--
Well [~ashb], slack is not available when i tried to sign in. Is there any
instruction to join/sign in?
was (Author: alnhk):
Well, slack is not available when i tried to sign in. Is there any instruction
to join/sign in?
> Airflow webserver intermitent broken
>
>
> Key: AIRFLOW-3351
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3351
> Project: Apache Airflow
> Issue Type: Bug
> Components: authentication
>Affects Versions: 1.10.0
>Reporter: Hari Krishna ADDEPALLI LN
>Priority: Blocker
>
> After completing the airflow 1.10.0 integration with LDAP anonymously
> (AIRFLOW-3270), we started to hit "Internal Server Error" with below
> exception stack, we tried to clean up the browser cache, it sometimes works
> and sometimes error our. Please advise the resolution to avoid this issue.
>
> {code:java}
> During handling of the above exception, another exception occurred: Traceback
> (most recent call last): File
> "/usr/local/lib/python3.5/site-packages/gunicorn/workers/sync.py", line 135,
> in handle self.handle_request(listener, req, client, addr) File
> "/usr/local/lib/python3.5/site-packages/gunicorn/workers/sync.py", line 176,
> in handle_request respiter = self.wsgi(environ, resp.start_response)
> File "/usr/local/lib/python3.5/site-packages/werkzeug/wsgi.py", line 826, in
> __call__ return app(environ, start_response) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1997, in __call__
> return self.wsgi_app(environ, start_response) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1985, in wsgi_app
> response = self.handle_exception(e) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1547, in
> handle_exception return self.finalize_request(handler(e),
> from_error_handler=True) File
> "/usr/local/lib/python3.5/site-packages/airflow/www/views.py", line 716, in
> show_traceback info=traceback.format_exc()), 500 File
> "/usr/local/lib/python3.5/site-packages/flask/templating.py", line 132, in
> render_template ctx.app.update_template_context(context) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 764, in
> update_template_context context.update(func()) File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 825, in
> _user_context_processor return dict(current_user=_get_user()) File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 794, in
> _get_user current_app.login_manager._load_user() File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 363, in
> _load_user return self.reload_user() File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 325, in
> reload_user user = self.user_callback(user_id) File
> "/usr/local/lib/python3.5/site-packages/airflow/utils/db.py", line 74, in
> wrapper return func(*args, **kwargs) File
> "/usr/local/lib/python3.5/site-packages/airflow/contrib/auth/backends/ldap_auth.py",
> line 278, in load_user return LdapUser(user) File "", line 4,
> in __init__ File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/state.py", line 414,
> in _initialize_instance manager.dispatch.init_failure(self, args, kwargs)
> File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/util/langhelpers.py", line
> 66, in __exit__ compat.reraise(exc_type, exc_value, exc_tb) File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 187,
> in reraise raise value File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/state.py", line 411,
> in _initialize_instance return manager.original_init(*mixed[1:],
> **kwargs) File
> "/usr/local/lib/python3.5/site-packages/airflow/contrib/auth/backends/ldap_auth.py",
> line 157, in __init__ user.username) AttributeError: 'NoneType' object
> has no attribute 'username' 127.0.0.1 - - [15/Nov/2018:10:47:31 +] "GET
> /admin/ HTTP/1.1" 500 0 "-" "-" [2018-11-15 10:47:38,590] ERROR in app:
> Exception on /favicon.ico [GET] Traceback (most recent call last): File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1612, in
> full_dispatch_request rv = self.dispatch_request() File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1590, in
> dispatch_request self.raise_routing_exception(req) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1573, in
> raise_routing_exception raise request.routing_exception File
>
[jira] [Commented] (AIRFLOW-3351) Airflow webserver intermitent broken
[
https://issues.apache.org/jira/browse/AIRFLOW-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700786#comment-16700786
]
Hari Krishna ADDEPALLI LN commented on AIRFLOW-3351:
Well, slack is not available when i tried to sign in. Is there any instruction
to join/sign in?
> Airflow webserver intermitent broken
>
>
> Key: AIRFLOW-3351
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3351
> Project: Apache Airflow
> Issue Type: Bug
> Components: authentication
>Affects Versions: 1.10.0
>Reporter: Hari Krishna ADDEPALLI LN
>Priority: Blocker
>
> After completing the airflow 1.10.0 integration with LDAP anonymously
> (AIRFLOW-3270), we started to hit "Internal Server Error" with below
> exception stack, we tried to clean up the browser cache, it sometimes works
> and sometimes error our. Please advise the resolution to avoid this issue.
>
> {code:java}
> During handling of the above exception, another exception occurred: Traceback
> (most recent call last): File
> "/usr/local/lib/python3.5/site-packages/gunicorn/workers/sync.py", line 135,
> in handle self.handle_request(listener, req, client, addr) File
> "/usr/local/lib/python3.5/site-packages/gunicorn/workers/sync.py", line 176,
> in handle_request respiter = self.wsgi(environ, resp.start_response)
> File "/usr/local/lib/python3.5/site-packages/werkzeug/wsgi.py", line 826, in
> __call__ return app(environ, start_response) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1997, in __call__
> return self.wsgi_app(environ, start_response) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1985, in wsgi_app
> response = self.handle_exception(e) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1547, in
> handle_exception return self.finalize_request(handler(e),
> from_error_handler=True) File
> "/usr/local/lib/python3.5/site-packages/airflow/www/views.py", line 716, in
> show_traceback info=traceback.format_exc()), 500 File
> "/usr/local/lib/python3.5/site-packages/flask/templating.py", line 132, in
> render_template ctx.app.update_template_context(context) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 764, in
> update_template_context context.update(func()) File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 825, in
> _user_context_processor return dict(current_user=_get_user()) File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 794, in
> _get_user current_app.login_manager._load_user() File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 363, in
> _load_user return self.reload_user() File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 325, in
> reload_user user = self.user_callback(user_id) File
> "/usr/local/lib/python3.5/site-packages/airflow/utils/db.py", line 74, in
> wrapper return func(*args, **kwargs) File
> "/usr/local/lib/python3.5/site-packages/airflow/contrib/auth/backends/ldap_auth.py",
> line 278, in load_user return LdapUser(user) File "", line 4,
> in __init__ File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/state.py", line 414,
> in _initialize_instance manager.dispatch.init_failure(self, args, kwargs)
> File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/util/langhelpers.py", line
> 66, in __exit__ compat.reraise(exc_type, exc_value, exc_tb) File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 187,
> in reraise raise value File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/state.py", line 411,
> in _initialize_instance return manager.original_init(*mixed[1:],
> **kwargs) File
> "/usr/local/lib/python3.5/site-packages/airflow/contrib/auth/backends/ldap_auth.py",
> line 157, in __init__ user.username) AttributeError: 'NoneType' object
> has no attribute 'username' 127.0.0.1 - - [15/Nov/2018:10:47:31 +] "GET
> /admin/ HTTP/1.1" 500 0 "-" "-" [2018-11-15 10:47:38,590] ERROR in app:
> Exception on /favicon.ico [GET] Traceback (most recent call last): File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1612, in
> full_dispatch_request rv = self.dispatch_request() File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1590, in
> dispatch_request self.raise_routing_exception(req) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1573, in
> raise_routing_exception raise request.routing_exception File
> "/usr/local/lib/python3.5/site-packages/flask/ctx.py", line 294, in
> match_request self.url_adapter.match(return_rule=True) File
>
[GitHub] ashb commented on issue #4241: [AIRFLOW-XXX] Correct typos in UPDATING.md
ashb commented on issue #4241: [AIRFLOW-XXX] Correct typos in UPDATING.md URL: https://github.com/apache/incubator-airflow/pull/4241#issuecomment-442151509 And yes, PyPI releases are entirely a manual process. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] feng-tao commented on issue #4241: [AIRFLOW-XXX] Correct typos in UPDATING.md
feng-tao commented on issue #4241: [AIRFLOW-XXX] Correct typos in UPDATING.md URL: https://github.com/apache/incubator-airflow/pull/4241#issuecomment-442151426 thanks @kaxil This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3351) Airflow webserver intermitent broken
[
https://issues.apache.org/jira/browse/AIRFLOW-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700781#comment-16700781
]
Ash Berlin-Taylor commented on AIRFLOW-3351:
This is an open source project and that is predominately volunteer lead, and as
such there is support is best efforts.
If you need that level of support grab me on Slack or email (a...@apache.org)
and we can discuss something more formal.
> Airflow webserver intermitent broken
>
>
> Key: AIRFLOW-3351
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3351
> Project: Apache Airflow
> Issue Type: Bug
> Components: authentication
>Affects Versions: 1.10.0
>Reporter: Hari Krishna ADDEPALLI LN
>Priority: Blocker
>
> After completing the airflow 1.10.0 integration with LDAP anonymously
> (AIRFLOW-3270), we started to hit "Internal Server Error" with below
> exception stack, we tried to clean up the browser cache, it sometimes works
> and sometimes error our. Please advise the resolution to avoid this issue.
>
> {code:java}
> During handling of the above exception, another exception occurred: Traceback
> (most recent call last): File
> "/usr/local/lib/python3.5/site-packages/gunicorn/workers/sync.py", line 135,
> in handle self.handle_request(listener, req, client, addr) File
> "/usr/local/lib/python3.5/site-packages/gunicorn/workers/sync.py", line 176,
> in handle_request respiter = self.wsgi(environ, resp.start_response)
> File "/usr/local/lib/python3.5/site-packages/werkzeug/wsgi.py", line 826, in
> __call__ return app(environ, start_response) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1997, in __call__
> return self.wsgi_app(environ, start_response) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1985, in wsgi_app
> response = self.handle_exception(e) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1547, in
> handle_exception return self.finalize_request(handler(e),
> from_error_handler=True) File
> "/usr/local/lib/python3.5/site-packages/airflow/www/views.py", line 716, in
> show_traceback info=traceback.format_exc()), 500 File
> "/usr/local/lib/python3.5/site-packages/flask/templating.py", line 132, in
> render_template ctx.app.update_template_context(context) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 764, in
> update_template_context context.update(func()) File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 825, in
> _user_context_processor return dict(current_user=_get_user()) File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 794, in
> _get_user current_app.login_manager._load_user() File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 363, in
> _load_user return self.reload_user() File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 325, in
> reload_user user = self.user_callback(user_id) File
> "/usr/local/lib/python3.5/site-packages/airflow/utils/db.py", line 74, in
> wrapper return func(*args, **kwargs) File
> "/usr/local/lib/python3.5/site-packages/airflow/contrib/auth/backends/ldap_auth.py",
> line 278, in load_user return LdapUser(user) File "", line 4,
> in __init__ File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/state.py", line 414,
> in _initialize_instance manager.dispatch.init_failure(self, args, kwargs)
> File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/util/langhelpers.py", line
> 66, in __exit__ compat.reraise(exc_type, exc_value, exc_tb) File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 187,
> in reraise raise value File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/state.py", line 411,
> in _initialize_instance return manager.original_init(*mixed[1:],
> **kwargs) File
> "/usr/local/lib/python3.5/site-packages/airflow/contrib/auth/backends/ldap_auth.py",
> line 157, in __init__ user.username) AttributeError: 'NoneType' object
> has no attribute 'username' 127.0.0.1 - - [15/Nov/2018:10:47:31 +] "GET
> /admin/ HTTP/1.1" 500 0 "-" "-" [2018-11-15 10:47:38,590] ERROR in app:
> Exception on /favicon.ico [GET] Traceback (most recent call last): File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1612, in
> full_dispatch_request rv = self.dispatch_request() File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1590, in
> dispatch_request self.raise_routing_exception(req) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1573, in
> raise_routing_exception raise request.routing_exception File
> "/usr/local/lib/python3.5/site-packages/flask/ctx.py", line 294,
[jira] [Commented] (AIRFLOW-3351) Airflow webserver intermitent broken
[
https://issues.apache.org/jira/browse/AIRFLOW-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700774#comment-16700774
]
Hari Krishna ADDEPALLI LN commented on AIRFLOW-3351:
[~ashb] [~sanand] : Can any one please share the telephone detail in order to
resolve this ? We are blocked with 1 step away from production go live!
> Airflow webserver intermitent broken
>
>
> Key: AIRFLOW-3351
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3351
> Project: Apache Airflow
> Issue Type: Bug
> Components: authentication
>Affects Versions: 1.10.0
>Reporter: Hari Krishna ADDEPALLI LN
>Priority: Blocker
>
> After completing the airflow 1.10.0 integration with LDAP anonymously
> (AIRFLOW-3270), we started to hit "Internal Server Error" with below
> exception stack, we tried to clean up the browser cache, it sometimes works
> and sometimes error our. Please advise the resolution to avoid this issue.
>
> {code:java}
> During handling of the above exception, another exception occurred: Traceback
> (most recent call last): File
> "/usr/local/lib/python3.5/site-packages/gunicorn/workers/sync.py", line 135,
> in handle self.handle_request(listener, req, client, addr) File
> "/usr/local/lib/python3.5/site-packages/gunicorn/workers/sync.py", line 176,
> in handle_request respiter = self.wsgi(environ, resp.start_response)
> File "/usr/local/lib/python3.5/site-packages/werkzeug/wsgi.py", line 826, in
> __call__ return app(environ, start_response) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1997, in __call__
> return self.wsgi_app(environ, start_response) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1985, in wsgi_app
> response = self.handle_exception(e) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1547, in
> handle_exception return self.finalize_request(handler(e),
> from_error_handler=True) File
> "/usr/local/lib/python3.5/site-packages/airflow/www/views.py", line 716, in
> show_traceback info=traceback.format_exc()), 500 File
> "/usr/local/lib/python3.5/site-packages/flask/templating.py", line 132, in
> render_template ctx.app.update_template_context(context) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 764, in
> update_template_context context.update(func()) File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 825, in
> _user_context_processor return dict(current_user=_get_user()) File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 794, in
> _get_user current_app.login_manager._load_user() File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 363, in
> _load_user return self.reload_user() File
> "/usr/local/lib/python3.5/site-packages/flask_login.py", line 325, in
> reload_user user = self.user_callback(user_id) File
> "/usr/local/lib/python3.5/site-packages/airflow/utils/db.py", line 74, in
> wrapper return func(*args, **kwargs) File
> "/usr/local/lib/python3.5/site-packages/airflow/contrib/auth/backends/ldap_auth.py",
> line 278, in load_user return LdapUser(user) File "", line 4,
> in __init__ File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/state.py", line 414,
> in _initialize_instance manager.dispatch.init_failure(self, args, kwargs)
> File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/util/langhelpers.py", line
> 66, in __exit__ compat.reraise(exc_type, exc_value, exc_tb) File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 187,
> in reraise raise value File
> "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/state.py", line 411,
> in _initialize_instance return manager.original_init(*mixed[1:],
> **kwargs) File
> "/usr/local/lib/python3.5/site-packages/airflow/contrib/auth/backends/ldap_auth.py",
> line 157, in __init__ user.username) AttributeError: 'NoneType' object
> has no attribute 'username' 127.0.0.1 - - [15/Nov/2018:10:47:31 +] "GET
> /admin/ HTTP/1.1" 500 0 "-" "-" [2018-11-15 10:47:38,590] ERROR in app:
> Exception on /favicon.ico [GET] Traceback (most recent call last): File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1612, in
> full_dispatch_request rv = self.dispatch_request() File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1590, in
> dispatch_request self.raise_routing_exception(req) File
> "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1573, in
> raise_routing_exception raise request.routing_exception File
> "/usr/local/lib/python3.5/site-packages/flask/ctx.py", line 294, in
> match_request self.url_adapter.match(return_rule=True) File
>
[GitHub] feng-tao commented on issue #4241: [AIRFLOW-XXX] Correct typos in UPDATING.md
feng-tao commented on issue #4241: [AIRFLOW-XXX] Correct typos in UPDATING.md URL: https://github.com/apache/incubator-airflow/pull/4241#issuecomment-442148956 @kaxil , so this won't go into pypi unless we manually push so, right? So what is the purpose of merging this change into both master branch and v10 branch? And have we had consensus on whether we are going to maintain/ update all the release branch? And what pr should go to both release branch and master branch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jj-ian edited a comment on issue #4083: [AIRFLOW-3211] Reattach to GCP Dataproc jobs upon Airflow restart
jj-ian edited a comment on issue #4083: [AIRFLOW-3211] Reattach to GCP Dataproc jobs upon Airflow restart URL: https://github.com/apache/incubator-airflow/pull/4083#issuecomment-441752788 Hi @bolkedebruin sorry for the delay; things got busy at work. Will try to make some headway on the requested changes this week. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4121: [AIRFLOW-2568] Azure Container Instances operator
codecov-io edited a comment on issue #4121: [AIRFLOW-2568] Azure Container Instances operator URL: https://github.com/apache/incubator-airflow/pull/4121#issuecomment-436818600 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4121?src=pr=h1) Report > Merging [#4121](https://codecov.io/gh/apache/incubator-airflow/pull/4121?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/d8c0b639743009dcfdc0bbe513319e0ed8e7de6e?src=pr=desc) will **decrease** coverage by `0.01%`. > The diff coverage is `50%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4121?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4121 +/- ## == - Coverage 77.82% 77.81% -0.02% == Files 201 201 Lines 1636716371 +4 == + Hits1273812739 +1 - Misses 3629 3632 +3 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4121?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/db.py](https://codecov.io/gh/apache/incubator-airflow/pull/4121/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `33.33% <0%> (-0.27%)` | :arrow_down: | | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4121/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `92.22% <66.66%> (-0.08%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4121?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4121?src=pr=footer). Last update [d8c0b63...a4f27cb](https://codecov.io/gh/apache/incubator-airflow/pull/4121?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] omusavi commented on issue #4121: [AIRFLOW-2568] Azure Container Instances operator
omusavi commented on issue #4121: [AIRFLOW-2568] Azure Container Instances operator URL: https://github.com/apache/incubator-airflow/pull/4121#issuecomment-44285 Rebased again to get a new build. @Fokko, please take a look when you get a chance, hope it is ready to go. Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4244: [AIRFLOW-3403] Create Athena sensor
codecov-io edited a comment on issue #4244: [AIRFLOW-3403] Create Athena sensor URL: https://github.com/apache/incubator-airflow/pull/4244#issuecomment-442078711 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4244?src=pr=h1) Report > Merging [#4244](https://codecov.io/gh/apache/incubator-airflow/pull/4244?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/d8c0b639743009dcfdc0bbe513319e0ed8e7de6e?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4244?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4244 +/- ## == - Coverage 77.82% 77.82% -0.01% == Files 201 201 Lines 1636716367 == - Hits1273812737 -1 - Misses 3629 3630 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4244?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4244/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `92.25% <0%> (-0.05%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4244?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4244?src=pr=footer). Last update [d8c0b63...9caab0e](https://codecov.io/gh/apache/incubator-airflow/pull/4244?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3164) verify certificate of LDAP server
[ https://issues.apache.org/jira/browse/AIRFLOW-3164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700551#comment-16700551 ] Ash Berlin-Taylor commented on AIRFLOW-3164: Because LDAP without TLS transmits ever users password in plain text over the network where it could be sniffed. In following releases (2.0.0 onwards) this version of the webserver is going to be removed and replaced with Flask-AppBuilder so login will need changing anyway. https://flask-appbuilder.readthedocs.io/en/latest/security.html#authentication-ldap Thinking about it you could switch to this new UI already (since 1.10.0. See https://github.com/apache/incubator-airflow/blob/master/UPDATING.md#new-webserver-ui-with-role-based-access-control) - it may be less work and would future proof you more. > verify certificate of LDAP server > - > > Key: AIRFLOW-3164 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3164 > Project: Apache Airflow > Issue Type: Bug >Reporter: Bolke de Bruin >Priority: Blocker > Fix For: 1.10.1 > > > Currently we dont verify the certificate of the Ldap server this can lead to > security incidents. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] tal181 commented on a change in pull request #4244: [AIRFLOW-3403] Create Athena sensor
tal181 commented on a change in pull request #4244: [AIRFLOW-3403] Create Athena sensor URL: https://github.com/apache/incubator-airflow/pull/4244#discussion_r236693598 ## File path: airflow/contrib/sensors/aws_athena_senseor.py ## @@ -0,0 +1,65 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + + +from airflow.exceptions import AirflowException +from airflow.utils.decorators import apply_defaults +from airflow.contrib.hooks.aws_athena_hook import AWSAthenaHook +from airflow.sensors.base_sensor_operator import BaseSensorOperator + + +# our own implementation of athena step sensor, including multiple failure states. Review comment: No,removing the comment This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (AIRFLOW-987) `airflow kerberos` ignores --keytab and --principal arguments
[
https://issues.apache.org/jira/browse/AIRFLOW-987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anonymous reassigned AIRFLOW-987:
-
Assignee: (was: Pratap20)
> `airflow kerberos` ignores --keytab and --principal arguments
> -
>
> Key: AIRFLOW-987
> URL: https://issues.apache.org/jira/browse/AIRFLOW-987
> Project: Apache Airflow
> Issue Type: Bug
> Components: security
>Affects Versions: 1.8.0
> Environment: 1.8-rc5
>Reporter: Ruslan Dautkhanov
>Priority: Major
> Labels: easyfix, kerberos, security
>
> No matter which arguments I pass to `airflow kerberos`,
> it always executes as `kinit -r 3600m -k -t airflow.keytab -c
> /tmp/airflow_krb5_ccache airflow`
> So it failes with expected "kinit: Keytab contains no suitable keys for
> airf...@corp.some.com while getting initial credentials"
> Tried different arguments, -kt and --keytab, here's one of the runs (some
> lines wrapped for readability):
> {noformat}
> $ airflow kerberos -kt /home/rdautkha/.keytab rdautkha...@corp.some.com
> [2017-03-14 23:50:11,523] {__init__.py:57} INFO - Using executor LocalExecutor
> [2017-03-14 23:50:12,069] {kerberos.py:43} INFO - Reinitting kerberos from
> keytab:
> kinit -r 3600m -k -t airflow.keytab -c /tmp/airflow_krb5_ccache airflow
> [2017-03-14 23:50:12,080] {kerberos.py:55} ERROR -
> Couldn't reinit from keytab! `kinit' exited with 1.
> kinit: Keytab contains no suitable keys for airf...@corp.some.com
> while getting initial credentials
> {noformat}
> 1.8-rc5
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] XD-DENG commented on a change in pull request #4234: [AIRFLOW-2761] Parallelize enqueue in celery executor
XD-DENG commented on a change in pull request #4234: [AIRFLOW-2761] Parallelize enqueue in celery executor URL: https://github.com/apache/incubator-airflow/pull/4234#discussion_r236628518 ## File path: airflow/jobs.py ## @@ -1374,24 +1352,65 @@ def _execute_task_instances(self, :type simple_dag_bag: SimpleDagBag :param states: Execute TaskInstances in these states :type states: Tuple[State] -:return: None +:return: Number of task instance with state changed. """ executable_tis = self._find_executable_task_instances(simple_dag_bag, states, session=session) def query(result, items): -tis_with_state_changed = self._change_state_for_executable_task_instances( -items, -states, -session=session) +simple_tis_with_state_changed = \ +self._change_state_for_executable_task_instances(items, + states, + session=session) self._enqueue_task_instances_with_queued_state( simple_dag_bag, -tis_with_state_changed) +simple_tis_with_state_changed) session.commit() -return result + len(tis_with_state_changed) +return result + len(simple_tis_with_state_changed) return helpers.reduce_in_chunks(query, executable_tis, 0, self.max_tis_per_query) +@provide_session +def _change_state_for_tasks_failed_to_execute(self, session): +""" +If there are tasks left over in the executor, +we set them back to SCHEDULED to avoid creating hanging tasks. +:param session: +:return: +""" Review comment: Minor: shall these two lines be removed if we don’t give details for them? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] tal181 commented on a change in pull request #4244: [AIRFLOW-3403] Create Athena sensor
tal181 commented on a change in pull request #4244: [AIRFLOW-3403] Create Athena sensor URL: https://github.com/apache/incubator-airflow/pull/4244#discussion_r236693598 ## File path: airflow/contrib/sensors/aws_athena_senseor.py ## @@ -0,0 +1,65 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + + +from airflow.exceptions import AirflowException +from airflow.utils.decorators import apply_defaults +from airflow.contrib.hooks.aws_athena_hook import AWSAthenaHook +from airflow.sensors.base_sensor_operator import BaseSensorOperator + + +# our own implementation of athena step sensor, including multiple failure states. Review comment: removing the comment This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] XD-DENG commented on a change in pull request #4244: [AIRFLOW-3403] Create Athena sensor
XD-DENG commented on a change in pull request #4244: [AIRFLOW-3403] Create Athena sensor URL: https://github.com/apache/incubator-airflow/pull/4244#discussion_r236691235 ## File path: airflow/contrib/sensors/aws_athena_senseor.py ## @@ -0,0 +1,65 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + + +from airflow.exceptions import AirflowException +from airflow.utils.decorators import apply_defaults +from airflow.contrib.hooks.aws_athena_hook import AWSAthenaHook +from airflow.sensors.base_sensor_operator import BaseSensorOperator + + +# our own implementation of athena step sensor, including multiple failure states. Review comment: Is this line necessary? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] XD-DENG commented on a change in pull request #4244: [AIRFLOW-3403] Create Athena sensor
XD-DENG commented on a change in pull request #4244: [AIRFLOW-3403] Create
Athena sensor
URL: https://github.com/apache/incubator-airflow/pull/4244#discussion_r236691440
##
File path: airflow/contrib/sensors/aws_athena_senseor.py
##
@@ -0,0 +1,65 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+from airflow.exceptions import AirflowException
+from airflow.utils.decorators import apply_defaults
+from airflow.contrib.hooks.aws_athena_hook import AWSAthenaHook
+from airflow.sensors.base_sensor_operator import BaseSensorOperator
+
+
+# our own implementation of athena step sensor, including multiple failure
states.
+
+class AthenaSensor(BaseSensorOperator):
+INTERMEDIATE_STATES = ('QUEUED', 'RUNNING',)
Review comment:
Docstring is outstanding.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] codecov-io commented on issue #4244: [AIRFLOW-3403] Create Athena sensor
codecov-io commented on issue #4244: [AIRFLOW-3403] Create Athena sensor URL: https://github.com/apache/incubator-airflow/pull/4244#issuecomment-442078711 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4244?src=pr=h1) Report > Merging [#4244](https://codecov.io/gh/apache/incubator-airflow/pull/4244?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/d8c0b639743009dcfdc0bbe513319e0ed8e7de6e?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4244?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#4244 +/- ## === Coverage 77.82% 77.82% === Files 201 201 Lines 1636716367 === Hits1273812738 Misses 3629 3629 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4244?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4244?src=pr=footer). Last update [d8c0b63...9efed44](https://codecov.io/gh/apache/incubator-airflow/pull/4244?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4244: [AIRFLOW-3403] Create Athena sensor
codecov-io edited a comment on issue #4244: [AIRFLOW-3403] Create Athena sensor URL: https://github.com/apache/incubator-airflow/pull/4244#issuecomment-442078711 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4244?src=pr=h1) Report > Merging [#4244](https://codecov.io/gh/apache/incubator-airflow/pull/4244?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/d8c0b639743009dcfdc0bbe513319e0ed8e7de6e?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4244?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#4244 +/- ## === Coverage 77.82% 77.82% === Files 201 201 Lines 1636716367 === Hits1273812738 Misses 3629 3629 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4244?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4244?src=pr=footer). Last update [d8c0b63...9efed44](https://codecov.io/gh/apache/incubator-airflow/pull/4244?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3164) verify certificate of LDAP server
[ https://issues.apache.org/jira/browse/AIRFLOW-3164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700487#comment-16700487 ] Maciej Bryński commented on AIRFLOW-3164: - OK. But why we don't give option to use LDAP without TLS ? Creating of copy of authentication backend just for this is IMHO not the best programming practice. And I'm pretty sure that this will break in following releases (because I will need to follow changes in auth_ldap) > verify certificate of LDAP server > - > > Key: AIRFLOW-3164 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3164 > Project: Apache Airflow > Issue Type: Bug >Reporter: Bolke de Bruin >Priority: Blocker > Fix For: 1.10.1 > > > Currently we dont verify the certificate of the Ldap server this can lead to > security incidents. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-3404) Add support for Amazon SES
[ https://issues.apache.org/jira/browse/AIRFLOW-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Loic Antoine-Gombeaud reassigned AIRFLOW-3404: -- Assignee: Loic Antoine-Gombeaud > Add support for Amazon SES > -- > > Key: AIRFLOW-3404 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3404 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Loic Antoine-Gombeaud >Assignee: Loic Antoine-Gombeaud >Priority: Major > > Sendgrid is currently implemented for e-mail sending, it would be great to > have SES as well -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3405) Task instance fail intermittently due to MySQL error
[ https://issues.apache.org/jira/browse/AIRFLOW-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700466#comment-16700466 ] Ash Berlin-Taylor commented on AIRFLOW-3405: If you are up for trying the bleeding edge version of Airflow (warning! there may be bugs in there!) then this PR[1] may help by reducing the number of pool slots you need - it should be possible to run with a much much smaller SQLA pool size in theory. [1]: https://github.com/apache/incubator-airflow/pull/4234 > Task instance fail intermittently due to MySQL error > > > Key: AIRFLOW-3405 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3405 > Project: Apache Airflow > Issue Type: Improvement > Environment: MySQL, Redhat Linux >Reporter: Yuvaraj >Priority: Major > Labels: performance, usability > > Dags are getting failed intermittently due to below error. > OperationalError: (_mysql_exceptions.OperationalError) (1040, 'Too many > connections') > [2018-11-25 12:24:16,952] - Heartbeat time limited exceeded! > We have max_connections defined as 2000 in DB. > Below are the setting in cfg. > sql_alchemy_pool_size = 1980 > sql_alchemy_pool_recycle = 3600 > As per DBA, The airflow scheduler keeps opening connections to the database, > these connections are mostly idle, they get reset whenever the scheduler > restarts but with max_connections at 2000 and scheduler holding on to 1600 of > these, other apps trying to connect might start running out of connections. > How do we remediate these idle connections. What should be the optimal value > for these configs and max_connections that to be set at DB. Consider we need > to build a large environment serving 500+ definitions with 1+ runs per > day. Need suggestions... > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3405) Task instance fail intermittently due to MySQL error
[ https://issues.apache.org/jira/browse/AIRFLOW-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700460#comment-16700460 ] Ash Berlin-Taylor commented on AIRFLOW-3405: That is a huge number of connections and is something we'd want to fix. What version of Airflow are you running on? > Task instance fail intermittently due to MySQL error > > > Key: AIRFLOW-3405 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3405 > Project: Apache Airflow > Issue Type: Improvement > Environment: MySQL, Redhat Linux >Reporter: Yuvaraj >Priority: Major > Labels: performance, usability > > Dags are getting failed intermittently due to below error. > OperationalError: (_mysql_exceptions.OperationalError) (1040, 'Too many > connections') > [2018-11-25 12:24:16,952] - Heartbeat time limited exceeded! > We have max_connections defined as 2000 in DB. > Below are the setting in cfg. > sql_alchemy_pool_size = 1980 > sql_alchemy_pool_recycle = 3600 > As per DBA, The airflow scheduler keeps opening connections to the database, > these connections are mostly idle, they get reset whenever the scheduler > restarts but with max_connections at 2000 and scheduler holding on to 1600 of > these, other apps trying to connect might start running out of connections. > How do we remediate these idle connections. What should be the optimal value > for these configs and max_connections that to be set at DB. Consider we need > to build a large environment serving 500+ definitions with 1+ runs per > day. Need suggestions... > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3164) verify certificate of LDAP server
[ https://issues.apache.org/jira/browse/AIRFLOW-3164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700458#comment-16700458 ] Ash Berlin-Taylor commented on AIRFLOW-3164: Airflow doesn't follow SemVer, and doesn't claim to. I'm sorry this broke your install, but we have decided that sending passwords in plain text over the network is bad security practice and security should win over not breaking some installs in this case. If you need help setting up the custom backend let me know and I can run you through it. > verify certificate of LDAP server > - > > Key: AIRFLOW-3164 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3164 > Project: Apache Airflow > Issue Type: Bug >Reporter: Bolke de Bruin >Priority: Blocker > Fix For: 1.10.1 > > > Currently we dont verify the certificate of the Ldap server this can lead to > security incidents. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ashb commented on a change in pull request #4006: [AIRFLOW-3164] Verify server certificate when connecting to LDAP
ashb commented on a change in pull request #4006: [AIRFLOW-3164] Verify server certificate when connecting to LDAP URL: https://github.com/apache/incubator-airflow/pull/4006#discussion_r236673953 ## File path: UPDATING.md ## @@ -87,6 +87,17 @@ configuration, so creating EMR clusters might fail until your connection is upda Ec2SubnetId, TerminationProtection and KeepJobFlowAliveWhenNoSteps were all top-level keys when they should be inside the "Instances" dict) +### LDAP Auth Backend now requires TLS + +Connecting to an LDAP serever over plain text is not supported anymore. The +certificate presented by the LDAP server must be signed by a trusted +certificiate, or you must provide the `cacert` option under `[ldap]` in the +config file. + +If you want to use LDAP auth backend without TLS then you will habe to create a +custom-auth backend based on +https://github.com/apache/incubator-airflow/blob/1.10.0/airflow/contrib/auth/backends/ldap_auth.py Review comment: (Continuing the conversation on Jira now) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb commented on a change in pull request #2418: [AIRFLOW-1381] Specify host temporary directory
ashb commented on a change in pull request #2418: [AIRFLOW-1381] Specify host temporary directory URL: https://github.com/apache/incubator-airflow/pull/2418#discussion_r236671899 ## File path: airflow/operators/docker_operator.py ## @@ -46,6 +46,9 @@ class DockerOperator(BaseOperator): :type environment: dict :param force_pull: Pull the docker image on every run. :type force_pull: bool +:param host_tmp_dir: Specify the location of the temporary directory on the host which will +be mapped to tmp_dir Review comment: ```suggestion be mapped to tmp_dir. If not provided defaults to using the standard system temp directory ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] maver1ck commented on a change in pull request #4006: [AIRFLOW-3164] Verify server certificate when connecting to LDAP
maver1ck commented on a change in pull request #4006: [AIRFLOW-3164] Verify server certificate when connecting to LDAP URL: https://github.com/apache/incubator-airflow/pull/4006#discussion_r236671478 ## File path: UPDATING.md ## @@ -87,6 +87,17 @@ configuration, so creating EMR clusters might fail until your connection is upda Ec2SubnetId, TerminationProtection and KeepJobFlowAliveWhenNoSteps were all top-level keys when they should be inside the "Instances" dict) +### LDAP Auth Backend now requires TLS + +Connecting to an LDAP serever over plain text is not supported anymore. The +certificate presented by the LDAP server must be signed by a trusted +certificiate, or you must provide the `cacert` option under `[ldap]` in the +config file. + +If you want to use LDAP auth backend without TLS then you will habe to create a +custom-auth backend based on +https://github.com/apache/incubator-airflow/blob/1.10.0/airflow/contrib/auth/backends/ldap_auth.py Review comment: Why we did a breaking change without option to configure this ? Not everyone is using TLS enabled LDAP. Upgrade from 1.10.0 to 1.10.1 completely broke my Airflow instalation. And this should be bugfix release. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ron819 commented on issue #2418: [AIRFLOW-1381] Specify host temporary directory
ron819 commented on issue #2418: [AIRFLOW-1381] Specify host temporary directory URL: https://github.com/apache/incubator-airflow/pull/2418#issuecomment-442064419 @benjaminsims can you rebase? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3164) verify certificate of LDAP server
[ https://issues.apache.org/jira/browse/AIRFLOW-3164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700442#comment-16700442 ] Maciej Bryński commented on AIRFLOW-3164: - Nothing. I just upgraded Airflow thinking it's bugfix upgrade. And did a rollback after reading this PR. https://github.com/apache/incubator-airflow/pull/4006#discussion_r236643279 Still curious about introducing breaking changes in bugfix version. > verify certificate of LDAP server > - > > Key: AIRFLOW-3164 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3164 > Project: Apache Airflow > Issue Type: Bug >Reporter: Bolke de Bruin >Priority: Blocker > Fix For: 1.10.1 > > > Currently we dont verify the certificate of the Ldap server this can lead to > security incidents. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-2104) Recommendation for Airflow design/architecture
[ https://issues.apache.org/jira/browse/AIRFLOW-2104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik closed AIRFLOW-2104. --- Resolution: Not A Problem > Recommendation for Airflow design/architecture > -- > > Key: AIRFLOW-2104 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2104 > Project: Apache Airflow > Issue Type: Improvement > Components: celery, configuration, DagRun >Affects Versions: 1.9.0 >Reporter: Anil Kumar >Priority: Major > Fix For: 1.9.0 > > > Hello, > We are in the implementation phase for Apache Airflow and looking for > Recommendation on Airflow configuration/structure. > Which one will be best option > # Airflow standalone installation with multi worker nodes or > # Airflow running as docker image on Kubernetes > Requirement :- > We are suppose to schedule a long running ETL process that includes complex > unzip/decrypt task, execution of data wrangling UDF's. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] maver1ck commented on a change in pull request #4006: [AIRFLOW-3164] Verify server certificate when connecting to LDAP
maver1ck commented on a change in pull request #4006: [AIRFLOW-3164] Verify server certificate when connecting to LDAP URL: https://github.com/apache/incubator-airflow/pull/4006#discussion_r236668195 ## File path: UPDATING.md ## @@ -87,6 +87,17 @@ configuration, so creating EMR clusters might fail until your connection is upda Ec2SubnetId, TerminationProtection and KeepJobFlowAliveWhenNoSteps were all top-level keys when they should be inside the "Instances" dict) +### LDAP Auth Backend now requires TLS + +Connecting to an LDAP serever over plain text is not supported anymore. The +certificate presented by the LDAP server must be signed by a trusted +certificiate, or you must provide the `cacert` option under `[ldap]` in the +config file. + +If you want to use LDAP auth backend without TLS then you will habe to create a +custom-auth backend based on +https://github.com/apache/incubator-airflow/blob/1.10.0/airflow/contrib/auth/backends/ldap_auth.py Review comment: So my question one more time. Why we created not backward compatible change in bugfix upgrade ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] maver1ck edited a comment on issue #4006: [AIRFLOW-3164] Verify server certificate when connecting to LDAP
maver1ck edited a comment on issue #4006: [AIRFLOW-3164] Verify server certificate when connecting to LDAP URL: https://github.com/apache/incubator-airflow/pull/4006#issuecomment-442063152 Thanks. I remove typo comments. Still curious about introducing breaking changes in bugfix version. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] maver1ck commented on issue #4006: [AIRFLOW-3164] Verify server certificate when connecting to LDAP
maver1ck commented on issue #4006: [AIRFLOW-3164] Verify server certificate when connecting to LDAP URL: https://github.com/apache/incubator-airflow/pull/4006#issuecomment-442063152 Thanks. I remove typo comments. Still curious about breaking changes in bugfix version. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] maver1ck commented on a change in pull request #4006: [AIRFLOW-3164] Verify server certificate when connecting to LDAP
maver1ck commented on a change in pull request #4006: [AIRFLOW-3164] Verify server certificate when connecting to LDAP URL: https://github.com/apache/incubator-airflow/pull/4006#discussion_r236668195 ## File path: UPDATING.md ## @@ -87,6 +87,17 @@ configuration, so creating EMR clusters might fail until your connection is upda Ec2SubnetId, TerminationProtection and KeepJobFlowAliveWhenNoSteps were all top-level keys when they should be inside the "Instances" dict) +### LDAP Auth Backend now requires TLS + +Connecting to an LDAP serever over plain text is not supported anymore. The +certificate presented by the LDAP server must be signed by a trusted +certificiate, or you must provide the `cacert` option under `[ldap]` in the +config file. + +If you want to use LDAP auth backend without TLS then you will habe to create a +custom-auth backend based on +https://github.com/apache/incubator-airflow/blob/1.10.0/airflow/contrib/auth/backends/ldap_auth.py Review comment: So my question one more time. Why we created not backward compatible change in bugfix upgrade ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] maver1ck commented on a change in pull request #4006: [AIRFLOW-3164] Verify server certificate when connecting to LDAP
maver1ck commented on a change in pull request #4006: [AIRFLOW-3164] Verify server certificate when connecting to LDAP URL: https://github.com/apache/incubator-airflow/pull/4006#discussion_r236633963 ## File path: UPDATING.md ## @@ -87,6 +87,17 @@ configuration, so creating EMR clusters might fail until your connection is upda Ec2SubnetId, TerminationProtection and KeepJobFlowAliveWhenNoSteps were all top-level keys when they should be inside the "Instances" dict) +### LDAP Auth Backend now requires TLS + +Connecting to an LDAP serever over plain text is not supported anymore. The +certificate presented by the LDAP server must be signed by a trusted +certificiate, or you must provide the `cacert` option under `[ldap]` in the Review comment: certificate This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] maver1ck commented on a change in pull request #4006: [AIRFLOW-3164] Verify server certificate when connecting to LDAP
maver1ck commented on a change in pull request #4006: [AIRFLOW-3164] Verify server certificate when connecting to LDAP URL: https://github.com/apache/incubator-airflow/pull/4006#discussion_r236633899 ## File path: UPDATING.md ## @@ -87,6 +87,17 @@ configuration, so creating EMR clusters might fail until your connection is upda Ec2SubnetId, TerminationProtection and KeepJobFlowAliveWhenNoSteps were all top-level keys when they should be inside the "Instances" dict) +### LDAP Auth Backend now requires TLS + +Connecting to an LDAP serever over plain text is not supported anymore. The Review comment: server This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kaxil commented on issue #4006: [AIRFLOW-3164] Verify server certificate when connecting to LDAP
kaxil commented on issue #4006: [AIRFLOW-3164] Verify server certificate when connecting to LDAP URL: https://github.com/apache/incubator-airflow/pull/4006#issuecomment-442062384 @maver1ck All the typos have already been corrected in the master: https://github.com/apache/incubator-airflow/pull/4242 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-3405) Task instance fail intermittently due to MySQL error
Yuvaraj created AIRFLOW-3405: Summary: Task instance fail intermittently due to MySQL error Key: AIRFLOW-3405 URL: https://issues.apache.org/jira/browse/AIRFLOW-3405 Project: Apache Airflow Issue Type: Improvement Environment: MySQL, Redhat Linux Reporter: Yuvaraj Dags are getting failed intermittently due to below error. OperationalError: (_mysql_exceptions.OperationalError) (1040, 'Too many connections') [2018-11-25 12:24:16,952] - Heartbeat time limited exceeded! We have max_connections defined as 2000 in DB. Below are the setting in cfg. sql_alchemy_pool_size = 1980 sql_alchemy_pool_recycle = 3600 As per DBA, The airflow scheduler keeps opening connections to the database, these connections are mostly idle, they get reset whenever the scheduler restarts but with max_connections at 2000 and scheduler holding on to 1600 of these, other apps trying to connect might start running out of connections. How do we remediate these idle connections. What should be the optimal value for these configs and max_connections that to be set at DB. Consider we need to build a large environment serving 500+ definitions with 1+ runs per day. Need suggestions... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] codecov-io commented on issue #4245: [AIRFLOW-3404] Add SES support
codecov-io commented on issue #4245: [AIRFLOW-3404] Add SES support URL: https://github.com/apache/incubator-airflow/pull/4245#issuecomment-442060768 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4245?src=pr=h1) Report > Merging [#4245](https://codecov.io/gh/apache/incubator-airflow/pull/4245?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/d8c0b639743009dcfdc0bbe513319e0ed8e7de6e?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4245?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#4245 +/- ## === Coverage 77.82% 77.82% === Files 201 201 Lines 1636716367 === Hits1273812738 Misses 3629 3629 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4245?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4245?src=pr=footer). Last update [d8c0b63...a5b1a55](https://codecov.io/gh/apache/incubator-airflow/pull/4245?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-1703) Airflow LocalExecutor crashes after 3 hours of work. Database is locked
[
https://issues.apache.org/jira/browse/AIRFLOW-1703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor resolved AIRFLOW-1703.
Resolution: Fixed
That was possibly the problem, and we now don't allow sqlite with anything
other than the Sequential Executor.
> Airflow LocalExecutor crashes after 3 hours of work. Database is locked
> ---
>
> Key: AIRFLOW-1703
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1703
> Project: Apache Airflow
> Issue Type: Bug
> Components: db, worker
>Affects Versions: 1.8.0
> Environment: Single CentOS virtual server
>Reporter: Kirill Dubovikov
>Priority: Major
> Attachments: nohup.out
>
>
> Airflow consistently crashes after working several hours on a single node
> when using SQLite DB. Our DAG is scheduled to run {{@daily}}. We launch
> airflow using the following commands
> {code:sh}
> airflow scheduler
> airflow webserver -p 8080
> {code}
> After a while worker and webserver crash with the following error:
> {{sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is
> locked [SQL: 'SELECT connection.conn_id AS connection_conn_id \nFROM
> connection GROUP BY connection.conn_id']}}
> I've attached full logs for further investigation
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3164) verify certificate of LDAP server
[ https://issues.apache.org/jira/browse/AIRFLOW-3164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700407#comment-16700407 ] Ash Berlin-Taylor commented on AIRFLOW-3164: > invalid CA public key file What are you providing to the cacert config option? It sounds like it is not a PEM-encoded certificate. (Security should not be optional) > verify certificate of LDAP server > - > > Key: AIRFLOW-3164 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3164 > Project: Apache Airflow > Issue Type: Bug >Reporter: Bolke de Bruin >Priority: Blocker > Fix For: 1.10.1 > > > Currently we dont verify the certificate of the Ldap server this can lead to > security incidents. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3364) Change Ooops Exception to something meanningfull
[
https://issues.apache.org/jira/browse/AIRFLOW-3364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700408#comment-16700408
]
jack commented on AIRFLOW-3364:
---
This could be related to:
[https://issues.apache.org/jira/projects/AIRFLOW/issues/AIRFLOW-419]
I'm also running PostgreSQL and the issue occurs when trying to change task
status.
could be more than a mere coincidence.
> Change Ooops Exception to something meanningfull
> -
>
> Key: AIRFLOW-3364
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3364
> Project: Apache Airflow
> Issue Type: Improvement
>Affects Versions: 1.9.0
>Reporter: jack
>Priority: Major
> Attachments: oops.PNG
>
>
> When I try to manually change tasks state (from up_for_retry) to success or
> whatever.
> I often get:
> h1.
> {code:java}
> Ooops. Traceback (most recent call last): File
> "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1988, in wsgi_app
> response = self.full_dispatch_request() File
> "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1641, in
> full_dispatch_request rv = self.handle_user_exception(e) File
> "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1544, in
> handle_user_exception reraise(exc_type, exc_value, tb) File
> "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1639, in
> full_dispatch_request rv = self.dispatch_request() File
> "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1625, in
> dispatch_request return self.view_functions[rule.endpoint](**req.view_args)
> File "/usr/local/lib/python2.7/dist-packages/flask_admin/base.py", line 69,
> in inner return self._run_view(f, *args, **kwargs) File
> "/usr/local/lib/python2.7/dist-packages/flask_admin/base.py", line 368, in
> _run_view return fn(self, *args, **kwargs) File
> "/usr/local/lib/python2.7/dist-packages/flask_admin/model/base.py", line
> 2068, in action_view return self.handle_action() File
> "/usr/local/lib/python2.7/dist-packages/flask_admin/actions.py", line 113, in
> handle_action response = handler[0](ids) File
> "/usr/local/lib/python2.7/dist-packages/airflow/www/views.py", line 2502, in
> action_set_success self.set_task_instance_state(ids, State.SUCCESS) File
> "/usr/local/lib/python2.7/dist-packages/airflow/utils/db.py", line 50, in
> wrapper result = func(*args, **kwargs) File
> "/usr/local/lib/python2.7/dist-packages/airflow/www/views.py", line 2540, in
> set_task_instance_state raise Exception("Ooops") Exception: Ooops{code}
>
>
> It's unclear what this oops is and why it's needed.
> If there is exception it should shown the actual error.
>
> Usually I simply delete the task to solve this.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1703) Airflow LocalExecutor crashes after 3 hours of work. Database is locked
[
https://issues.apache.org/jira/browse/AIRFLOW-1703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700393#comment-16700393
]
jack commented on AIRFLOW-1703:
---
SQLite with local executor? This doesn't make a lot of sense
> Airflow LocalExecutor crashes after 3 hours of work. Database is locked
> ---
>
> Key: AIRFLOW-1703
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1703
> Project: Apache Airflow
> Issue Type: Bug
> Components: db, worker
>Affects Versions: 1.8.0
> Environment: Single CentOS virtual server
>Reporter: Kirill Dubovikov
>Priority: Major
> Attachments: nohup.out
>
>
> Airflow consistently crashes after working several hours on a single node
> when using SQLite DB. Our DAG is scheduled to run {{@daily}}. We launch
> airflow using the following commands
> {code:sh}
> airflow scheduler
> airflow webserver -p 8080
> {code}
> After a while worker and webserver crash with the following error:
> {{sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is
> locked [SQL: 'SELECT connection.conn_id AS connection_conn_id \nFROM
> connection GROUP BY connection.conn_id']}}
> I've attached full logs for further investigation
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2076) [2017-11-21 16:12:40,961] {jobs.py:187} DEBUG - [heart] Boom.
[
https://issues.apache.org/jira/browse/AIRFLOW-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700382#comment-16700382
]
jack commented on AIRFLOW-2076:
---
you are using old version of Airflow. can you check it against master?
> [2017-11-21 16:12:40,961] {jobs.py:187} DEBUG - [heart] Boom.
> -
>
> Key: AIRFLOW-2076
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2076
> Project: Apache Airflow
> Issue Type: Bug
> Components: api
>Affects Versions: 1.8.0, 1.8.2, 2.0.0
>Reporter: fann
>Priority: Critical
> Labels: features
>
> I used airflow submit spark application, the airflow DAG always print DEBUG -
> [heart] Boom. When my spark application is finished, the print is still
> on，and the DAG cannot finished!
> I was used the LocalExecutor.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] codecov-io commented on issue #4246: [AIRFLOW-XXX] Replace airflow with apache-airflow
codecov-io commented on issue #4246: [AIRFLOW-XXX] Replace airflow with apache-airflow URL: https://github.com/apache/incubator-airflow/pull/4246#issuecomment-442052708 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4246?src=pr=h1) Report > Merging [#4246](https://codecov.io/gh/apache/incubator-airflow/pull/4246?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/d8c0b639743009dcfdc0bbe513319e0ed8e7de6e?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4246?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#4246 +/- ## === Coverage 77.82% 77.82% === Files 201 201 Lines 1636716367 === Hits1273812738 Misses 3629 3629 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4246?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4246?src=pr=footer). Last update [d8c0b63...7e2d8b6](https://codecov.io/gh/apache/incubator-airflow/pull/4246?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb commented on a change in pull request #4006: [AIRFLOW-3164] Verify server certificate when connecting to LDAP
ashb commented on a change in pull request #4006: [AIRFLOW-3164] Verify server certificate when connecting to LDAP URL: https://github.com/apache/incubator-airflow/pull/4006#discussion_r236643279 ## File path: UPDATING.md ## @@ -87,6 +87,17 @@ configuration, so creating EMR clusters might fail until your connection is upda Ec2SubnetId, TerminationProtection and KeepJobFlowAliveWhenNoSteps were all top-level keys when they should be inside the "Instances" dict) +### LDAP Auth Backend now requires TLS + +Connecting to an LDAP serever over plain text is not supported anymore. The +certificate presented by the LDAP server must be signed by a trusted +certificiate, or you must provide the `cacert` option under `[ldap]` in the +config file. + +If you want to use LDAP auth backend without TLS then you will habe to create a +custom-auth backend based on +https://github.com/apache/incubator-airflow/blob/1.10.0/airflow/contrib/auth/backends/ldap_auth.py Review comment: It is not configurable - if you want the old behaviour back you need to take this code, make it available in your install and configure the auth backend for the webserver to use it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko opened a new pull request #4246: [AIRFLOW-XXX] Replace airflow with apache-airflow
Fokko opened a new pull request #4246: [AIRFLOW-XXX] Replace airflow with
apache-airflow
URL: https://github.com/apache/incubator-airflow/pull/4246
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-3164) verify certificate of LDAP server
[
https://issues.apache.org/jira/browse/AIRFLOW-3164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700308#comment-16700308
]
Maciej Bryński commented on AIRFLOW-3164:
-
This PR broke my Airflow.
{code}
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/sync.py", line
135, in handle
self.handle_request(listener, req, client, addr)
File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/sync.py", line
176, in handle_request
respiter = self.wsgi(environ, resp.start_response)
File "/usr/local/lib/python3.6/site-packages/werkzeug/wsgi.py", line 826, in
__call__
return app(environ, start_response)
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1997, in
__call__
return self.wsgi_app(environ, start_response)
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1985, in
wsgi_app
response = self.handle_exception(e)
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1547, in
handle_exception
return self.finalize_request(handler(e), from_error_handler=True)
File "/usr/local/lib/python3.6/site-packages/airflow/www/views.py", line 708,
in show_traceback
info=traceback.format_exc()), 500
File "/usr/local/lib/python3.6/site-packages/flask/templating.py", line 132,
in render_template
ctx.app.update_template_context(context)
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 764, in
update_template_context
context.update(func())
File "/usr/local/lib/python3.6/site-packages/flask_login/utils.py", line 368,
in _user_context_processor
return dict(current_user=_get_user())
File "/usr/local/lib/python3.6/site-packages/flask_login/utils.py", line 335,
in _get_user
current_app.login_manager._load_user()
File "/usr/local/lib/python3.6/site-packages/flask_login/login_manager.py",
line 359, in _load_user
return self.reload_user()
File "/usr/local/lib/python3.6/site-packages/flask_login/login_manager.py",
line 321, in reload_user
user = self.user_callback(user_id)
File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 74,
in wrapper
return func(*args, **kwargs)
File
"/usr/local/lib/python3.6/site-packages/airflow/contrib/auth/backends/ldap_auth.py",
line 277, in load_user
return LdapUser(user)
File "", line 4, in __init__
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/state.py", line
414, in _initialize_instance
manager.dispatch.init_failure(self, args, kwargs)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py",
line 66, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line
187, in reraise
raise value
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/state.py", line
411, in _initialize_instance
return manager.original_init(*mixed[1:], **kwargs)
File
"/usr/local/lib/python3.6/site-packages/airflow/contrib/auth/backends/ldap_auth.py",
line 135, in __init__
configuration.conf.get("ldap", "bind_password"))
File
"/usr/local/lib/python3.6/site-packages/airflow/contrib/auth/backends/ldap_auth.py",
line 65, in get_ldap_connection
ca_certs_file=cacert)
File "/usr/local/lib/python3.6/site-packages/ldap3/core/tls.py", line 93, in
__init__
raise LDAPSSLConfigurationError('invalid CA public key file')
ldap3.core.exceptions.LDAPSSLConfigurationError: invalid CA public key file
{code}
Can we make this functionality optional?
> verify certificate of LDAP server
> -
>
> Key: AIRFLOW-3164
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3164
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Bolke de Bruin
>Priority: Blocker
> Fix For: 1.10.1
>
>
> Currently we dont verify the certificate of the Ldap server this can lead to
> security incidents.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] maver1ck commented on a change in pull request #4006: [AIRFLOW-3164] Verify server certificate when connecting to LDAP
maver1ck commented on a change in pull request #4006: [AIRFLOW-3164] Verify server certificate when connecting to LDAP URL: https://github.com/apache/incubator-airflow/pull/4006#discussion_r236633963 ## File path: UPDATING.md ## @@ -87,6 +87,17 @@ configuration, so creating EMR clusters might fail until your connection is upda Ec2SubnetId, TerminationProtection and KeepJobFlowAliveWhenNoSteps were all top-level keys when they should be inside the "Instances" dict) +### LDAP Auth Backend now requires TLS + +Connecting to an LDAP serever over plain text is not supported anymore. The +certificate presented by the LDAP server must be signed by a trusted +certificiate, or you must provide the `cacert` option under `[ldap]` in the Review comment: certificate This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] maver1ck commented on a change in pull request #4006: [AIRFLOW-3164] Verify server certificate when connecting to LDAP
maver1ck commented on a change in pull request #4006: [AIRFLOW-3164] Verify server certificate when connecting to LDAP URL: https://github.com/apache/incubator-airflow/pull/4006#discussion_r236634155 ## File path: UPDATING.md ## @@ -87,6 +87,17 @@ configuration, so creating EMR clusters might fail until your connection is upda Ec2SubnetId, TerminationProtection and KeepJobFlowAliveWhenNoSteps were all top-level keys when they should be inside the "Instances" dict) +### LDAP Auth Backend now requires TLS + +Connecting to an LDAP serever over plain text is not supported anymore. The +certificate presented by the LDAP server must be signed by a trusted +certificiate, or you must provide the `cacert` option under `[ldap]` in the +config file. + +If you want to use LDAP auth backend without TLS then you will habe to create a +custom-auth backend based on +https://github.com/apache/incubator-airflow/blob/1.10.0/airflow/contrib/auth/backends/ldap_auth.py Review comment: How to configure this backend ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] maver1ck commented on a change in pull request #4006: [AIRFLOW-3164] Verify server certificate when connecting to LDAP
maver1ck commented on a change in pull request #4006: [AIRFLOW-3164] Verify server certificate when connecting to LDAP URL: https://github.com/apache/incubator-airflow/pull/4006#discussion_r236634094 ## File path: UPDATING.md ## @@ -87,6 +87,17 @@ configuration, so creating EMR clusters might fail until your connection is upda Ec2SubnetId, TerminationProtection and KeepJobFlowAliveWhenNoSteps were all top-level keys when they should be inside the "Instances" dict) +### LDAP Auth Backend now requires TLS + +Connecting to an LDAP serever over plain text is not supported anymore. The +certificate presented by the LDAP server must be signed by a trusted +certificiate, or you must provide the `cacert` option under `[ldap]` in the +config file. + +If you want to use LDAP auth backend without TLS then you will habe to create a Review comment: have This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3404) Add support for Amazon SES
[
https://issues.apache.org/jira/browse/AIRFLOW-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700310#comment-16700310
]
ASF GitHub Bot commented on AIRFLOW-3404:
-
LoicAG opened a new pull request #4245: [AIRFLOW-3404] Add SES support
URL: https://github.com/apache/incubator-airflow/pull/4245
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3404
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
Similarly to `sendgrid.py`, `ses.py` defines an e-mail backend using Amazon
SES.
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
I don't know how to test this code, and would appreciate any support in that
direction
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add support for Amazon SES
> --
>
> Key: AIRFLOW-3404
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3404
> Project: Apache Airflow
> Issue Type: New Feature
>Reporter: Loic Antoine-Gombeaud
>Priority: Major
>
> Sendgrid is currently implemented for e-mail sending, it would be great to
> have SES as well
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-3404) Add support for Amazon SES
Loic Antoine-Gombeaud created AIRFLOW-3404: -- Summary: Add support for Amazon SES Key: AIRFLOW-3404 URL: https://issues.apache.org/jira/browse/AIRFLOW-3404 Project: Apache Airflow Issue Type: New Feature Reporter: Loic Antoine-Gombeaud Sendgrid is currently implemented for e-mail sending, it would be great to have SES as well -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3403) create athena sensor
[ https://issues.apache.org/jira/browse/AIRFLOW-3403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700301#comment-16700301 ] Ash Berlin-Taylor commented on AIRFLOW-3403: Having a sensor is no bad thing and we'll review and merge the PR, but if the operator is not failing when the query fails (sorry missed that detail) that is a bug in the operator that should be fixed too. > create athena sensor > > > Key: AIRFLOW-3403 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3403 > Project: Apache Airflow > Issue Type: Improvement >Reporter: tal >Assignee: tal >Priority: Minor > > It will be nice to have an Athena sensor to monitor the progress of the query -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] XD-DENG commented on issue #4234: [AIRFLOW-2761] Parallelize enqueue in celery executor
XD-DENG commented on issue #4234: [AIRFLOW-2761] Parallelize enqueue in celery executor URL: https://github.com/apache/incubator-airflow/pull/4234#issuecomment-442030964 Hi @KevinYang21 , haven’t got time to check details now. A few very minor points above first. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] XD-DENG commented on a change in pull request #4234: [AIRFLOW-2761] Parallelize enqueue in celery executor
XD-DENG commented on a change in pull request #4234: [AIRFLOW-2761] Parallelize enqueue in celery executor URL: https://github.com/apache/incubator-airflow/pull/4234#discussion_r236628644 ## File path: airflow/utils/dag_processing.py ## @@ -175,6 +190,48 @@ def try_number(self): def state(self): return self._state +@property +def pool(self): +return self._pool + +@property +def priority_weight(self): +return self._priority_weight + +@property +def queue(self): +return self._queue + +@property +def key(self): +return self._key + +@property +def executor_config(self): +return self._executor_config + +@provide_session +def construct_task_instance(self, session=None, lock_for_update=False): +""" +Construct a TaskInstance from the database based on the primary key +:param session: DB session. Review comment: Minor: One empty line is expected before this line to render Sphinx properly This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] XD-DENG commented on a change in pull request #4234: [AIRFLOW-2761] Parallelize enqueue in celery executor
XD-DENG commented on a change in pull request #4234: [AIRFLOW-2761] Parallelize enqueue in celery executor URL: https://github.com/apache/incubator-airflow/pull/4234#discussion_r236628518 ## File path: airflow/jobs.py ## @@ -1374,24 +1352,65 @@ def _execute_task_instances(self, :type simple_dag_bag: SimpleDagBag :param states: Execute TaskInstances in these states :type states: Tuple[State] -:return: None +:return: Number of task instance with state changed. """ executable_tis = self._find_executable_task_instances(simple_dag_bag, states, session=session) def query(result, items): -tis_with_state_changed = self._change_state_for_executable_task_instances( -items, -states, -session=session) +simple_tis_with_state_changed = \ +self._change_state_for_executable_task_instances(items, + states, + session=session) self._enqueue_task_instances_with_queued_state( simple_dag_bag, -tis_with_state_changed) +simple_tis_with_state_changed) session.commit() -return result + len(tis_with_state_changed) +return result + len(simple_tis_with_state_changed) return helpers.reduce_in_chunks(query, executable_tis, 0, self.max_tis_per_query) +@provide_session +def _change_state_for_tasks_failed_to_execute(self, session): +""" +If there are tasks left over in the executor, +we set them back to SCHEDULED to avoid creating hanging tasks. +:param session: +:return: +""" Review comment: Minor: shall these two lines to be removed if we don’t give details for them? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
