[GitHub] [airflow] yegeniy opened a new pull request #6281: Run batches of (self-terminating) EMR JobFlows [AIRFLOW-XXX]
yegeniy opened a new pull request #6281: Run batches of (self-terminating) EMR
JobFlows [AIRFLOW-XXX]
URL: https://github.com/apache/airflow/pull/6281
What is this:
-
This commit introduces an implementation of an operator that allows
Airflow to manage the execution of multiple EMR clusters. The clusters
are all configured through a Jinja template ahead of time, but submitted
in batches, allowing for workflows that require both sequences and
parallel clusters. The operator assumes that all the clusters it manages
auto-terminate.
As demonstrated in the unit test, with this operator the following
batches of EMR clusters could be set up: First, cluster1 runs its steps
until they all complete and the cluster terminates. Then, cluster2a and
cluster2b run their steps in parallel until they both terminate.
Finally, cluster3 runs its steps.
Why use this:
-
1) The primary appeal of this operator is for submitting one-off tasks
to EMR. The most straight-forward way to do this is to set up a DAG
consisting of a single `EmrRunJobFlows` operator which derives its
`job_flows` value from the DagRun's conf attribute ([`airflow dags
trigger --conf`][1]). While the actual EMR steps are obfuscated, one
small benefit from using this operator over modifying the DAG using the
existing EMR operators is that, while the resulting work is dynamic at
DagRun-time, the structure of the DAG stays constant between wildly
different runs, so we can see the execution history even when a typical
DAG would have been modified and lost its history in the Web UI.
[1]: https://airflow.readthedocs.io/en/latest/cli-ref.html#trigger
2) It can also simplify the retry logic wherever
`EmrCreateJobFlowOperator` and `EmrJobFlowSensor` operator pairs are
used. The `retry_handler` of an `EmrJobFlowSensor` task would need to
run an `airflow clear` on the preceding `EmrCreateJobFlowOperator` task
if it detects that the cluster failed, while somehow maintaining the
total count of attempts. Using the `EmrJobFlowSensor`, a cluster can be
restarted on failure without a custom `retry_handler` - simply by
setting `retries > 0`. Please note that, as described in its docstring,
the current implementation of `EmrRunJobFlows` limits the retry logic to
just this basic use case. This is because we don't expose any
information for a `retry_handler` to reason about at this time. This
could be improved by pushing step- and cluster- level status information
to an XCom, but that is not implemented here at this time because the
operator implementation is complex enough as is. If they are to be
retried, please make sure that all steps be idempotent.
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
[jira] [Commented] (AIRFLOW-5603) Add MLEngine version operators
[ https://issues.apache.org/jira/browse/AIRFLOW-5603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946457#comment-16946457 ] ASF subversion and git services commented on AIRFLOW-5603: -- Commit 0d71f335561800f117be41c575b20770f2345d50 in airflow's branch refs/heads/master from Kamil Breguła [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=0d71f33 ] [AIRFLOW-5603] Add MLEngine version operators (#6271) > Add MLEngine version operators > -- > > Key: AIRFLOW-5603 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5603 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5603) Add MLEngine version operators
[ https://issues.apache.org/jira/browse/AIRFLOW-5603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946455#comment-16946455 ] ASF GitHub Bot commented on AIRFLOW-5603: - mik-laj commented on pull request #6271: [AIRFLOW-5603] Add MLEngine version operators URL: https://github.com/apache/airflow/pull/6271 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add MLEngine version operators > -- > > Key: AIRFLOW-5603 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5603 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] mik-laj merged pull request #6271: [AIRFLOW-5603] Add MLEngine version operators
mik-laj merged pull request #6271: [AIRFLOW-5603] Add MLEngine version operators URL: https://github.com/apache/airflow/pull/6271 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6271: [AIRFLOW-5603] Add MLEngine version operators
codecov-io edited a comment on issue #6271: [AIRFLOW-5603] Add MLEngine version operators URL: https://github.com/apache/airflow/pull/6271#issuecomment-539172069 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6271?src=pr=h1) Report > Merging [#6271](https://codecov.io/gh/apache/airflow/pull/6271?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/b2dc1636855dcd3ad6957745e5426cd4f5f10c70?src=pr=desc) will **decrease** coverage by `0.43%`. > The diff coverage is `93.12%`. [](https://codecov.io/gh/apache/airflow/pull/6271?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6271 +/- ## == - Coverage 80.09% 79.66% -0.44% == Files 612 612 Lines 3532035399 +79 == - Hits2828928199 -90 - Misses 7031 7200 +169 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6271?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/gcp/operators/mlengine.py](https://codecov.io/gh/apache/airflow/pull/6271/diff?src=pr=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL21sZW5naW5lLnB5) | `85.48% <100%> (+6.29%)` | :arrow_up: | | [airflow/gcp/example\_dags/example\_mlengine.py](https://codecov.io/gh/apache/airflow/pull/6271/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfbWxlbmdpbmUucHk=) | `83.92% <83.92%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6271/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/executors/sequential\_executor.py](https://codecov.io/gh/apache/airflow/pull/6271/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvc2VxdWVudGlhbF9leGVjdXRvci5weQ==) | `47.61% <0%> (-52.39%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6271/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6271/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/kube\_client.py](https://codecov.io/gh/apache/airflow/pull/6271/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL2t1YmVfY2xpZW50LnB5) | `33.33% <0%> (-41.67%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6271/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `70.14% <0%> (-28.36%)` | :arrow_down: | | [airflow/utils/log/colored\_log.py](https://codecov.io/gh/apache/airflow/pull/6271/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvY29sb3JlZF9sb2cucHk=) | `72.72% <0%> (-20.46%)` | :arrow_down: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6271/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `86.44% <0%> (-6.78%)` | :arrow_down: | | ... and [7 more](https://codecov.io/gh/apache/airflow/pull/6271/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6271?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6271?src=pr=footer). Last update [b2dc163...64c00aa](https://codecov.io/gh/apache/airflow/pull/6271?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6232: [AIRFLOW-5582] AutoCommit in jdbc is missing get_autocommit
codecov-io edited a comment on issue #6232: [AIRFLOW-5582] AutoCommit in jdbc is missing get_autocommit URL: https://github.com/apache/airflow/pull/6232#issuecomment-537324813 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6232?src=pr=h1) Report > Merging [#6232](https://codecov.io/gh/apache/airflow/pull/6232?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/76fe45e1d127b657b1aad5c0fd657e011f5a09bc?src=pr=desc) will **decrease** coverage by `0.01%`. > The diff coverage is `50%`. [](https://codecov.io/gh/apache/airflow/pull/6232?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6232 +/- ## == - Coverage 80.05% 80.04% -0.02% == Files 610 610 Lines 3526435288 +24 == + Hits2823228246 +14 - Misses 7032 7042 +10 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6232?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/hooks/jdbc\_hook.py](https://codecov.io/gh/apache/airflow/pull/6232/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9qZGJjX2hvb2sucHk=) | `89.47% <50%> (-4.65%)` | :arrow_down: | | [airflow/executors/dask\_executor.py](https://codecov.io/gh/apache/airflow/pull/6232/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvZGFza19leGVjdXRvci5weQ==) | `2% <0%> (-4%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6232/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `56.37% <0%> (-0.52%)` | :arrow_down: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6232/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.24% <0%> (-0.51%)` | :arrow_down: | | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/6232/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `95.27% <0%> (ø)` | :arrow_up: | | [...rflow/gcp/example\_dags/example\_gcp\_dlp\_operator.py](https://codecov.io/gh/apache/airflow/pull/6232/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfZ2NwX2RscF9vcGVyYXRvci5weQ==) | `0% <0%> (ø)` | :arrow_up: | | [...ample\_dags/example\_emr\_job\_flow\_automatic\_steps.py](https://codecov.io/gh/apache/airflow/pull/6232/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX2Vtcl9qb2JfZmxvd19hdXRvbWF0aWNfc3RlcHMucHk=) | `0% <0%> (ø)` | :arrow_up: | | [airflow/utils/log/wasb\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/6232/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvd2FzYl90YXNrX2hhbmRsZXIucHk=) | `42.46% <0%> (ø)` | :arrow_up: | | [...contrib/example\_dags/example\_papermill\_operator.py](https://codecov.io/gh/apache/airflow/pull/6232/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3BhcGVybWlsbF9vcGVyYXRvci5weQ==) | `0% <0%> (ø)` | :arrow_up: | | [airflow/www/security.py](https://codecov.io/gh/apache/airflow/pull/6232/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvc2VjdXJpdHkucHk=) | `91.41% <0%> (ø)` | :arrow_up: | | ... and [216 more](https://codecov.io/gh/apache/airflow/pull/6232/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6232?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6232?src=pr=footer). Last update [76fe45e...203cfb3](https://codecov.io/gh/apache/airflow/pull/6232?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj merged pull request #6280: [AIRFLOW-XXX] Fix extra-packages tables
mik-laj merged pull request #6280: [AIRFLOW-XXX] Fix extra-packages tables URL: https://github.com/apache/airflow/pull/6280 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6280: [AIRFLOW-XXX] Fix extra-packages tables
codecov-io commented on issue #6280: [AIRFLOW-XXX] Fix extra-packages tables URL: https://github.com/apache/airflow/pull/6280#issuecomment-539295175 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6280?src=pr=h1) Report > Merging [#6280](https://codecov.io/gh/apache/airflow/pull/6280?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/df5fbcb5160f9ad985ce0361941ad15ab91e930f?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6280?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6280 +/- ## == - Coverage 80.28% 80.27% -0.01% == Files 612 612 Lines 3532035320 == - Hits2835728354 -3 - Misses 6963 6966 +3 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6280?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6280/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `56.37% <0%> (-0.52%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6280?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6280?src=pr=footer). Last update [df5fbcb...9be304c](https://codecov.io/gh/apache/airflow/pull/6280?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6279: [AIRFLOW-5609][WIP] Add MLEngine models operators
mik-laj commented on a change in pull request #6279: [AIRFLOW-5609][WIP] Add
MLEngine models operators
URL: https://github.com/apache/airflow/pull/6279#discussion_r33224
##
File path: airflow/gcp/hooks/mlengine.py
##
@@ -275,3 +275,19 @@ def get_model(self, project_id: str, model_name: str) ->
Optional[Dict]:
self.log.error('Model was not found: %s', e)
return None
raise
+
+def delete_model(self, project_id: str, model_name: str) -> None:
Review comment:
Here we need a fallback for project id, but I prefer to do it in a separate
PR for whole integration. I prefer to maintain uniformity instead of
functionality
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-3160) Load latest_dagruns asynchronously
[ https://issues.apache.org/jira/browse/AIRFLOW-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946407#comment-16946407 ] ASF subversion and git services commented on AIRFLOW-3160: -- Commit 61bf073b85b4728e9991409fd409001da8b8a9d3 in airflow's branch refs/heads/v1-10-test from Kaxil Naik [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=61bf073 ] [AIRFLOW-3160] Load latest_dagruns asynchronously (#5339) > Load latest_dagruns asynchronously > --- > > Key: AIRFLOW-3160 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3160 > Project: Apache Airflow > Issue Type: Improvement > Components: webserver >Affects Versions: 1.10.0 >Reporter: Dan Davydov >Assignee: Dan Davydov >Priority: Major > Fix For: 1.10.6 > > > The front page loads very slowly when the DB has latency because one blocking > query is made per DAG against the DB. > > The latest dagruns should be loaded asynchronously and in batch like the > other UI elements that query the database. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-3160) Load latest_dagruns asynchronously
[ https://issues.apache.org/jira/browse/AIRFLOW-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946402#comment-16946402 ] ASF subversion and git services commented on AIRFLOW-3160: -- Commit 9feb56bb7be7a7a05bf7d286ec2b3ccf6d38fd4c in airflow's branch refs/heads/v1-10-test from Kaxil Naik [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=9feb56b ] [AIRFLOW-3160] (Unrevert) Load latest_dagruns asynchronously (#5339) > Load latest_dagruns asynchronously > --- > > Key: AIRFLOW-3160 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3160 > Project: Apache Airflow > Issue Type: Improvement > Components: webserver >Affects Versions: 1.10.0 >Reporter: Dan Davydov >Assignee: Dan Davydov >Priority: Major > Fix For: 2.0.0, 1.10.6 > > > The front page loads very slowly when the DB has latency because one blocking > query is made per DAG against the DB. > > The latest dagruns should be loaded asynchronously and in batch like the > other UI elements that query the database. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] codecov-io edited a comment on issue #6265: [AIRFLOW-5597] Linkify urls in task instance log
codecov-io edited a comment on issue #6265: [AIRFLOW-5597] Linkify urls in task instance log URL: https://github.com/apache/airflow/pull/6265#issuecomment-538605802 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6265?src=pr=h1) Report > Merging [#6265](https://codecov.io/gh/apache/airflow/pull/6265?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/1347b77970b86503adc1f9b54f766c4dce89bc59?src=pr=desc) will **increase** coverage by `0.19%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6265?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6265 +/- ## == + Coverage 80.09% 80.28% +0.19% == Files 612 612 Lines 3532035320 == + Hits2828928357 +68 + Misses 7031 6963 -68 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6265?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/contrib/operators/ssh\_operator.py](https://codecov.io/gh/apache/airflow/pull/6265/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zc2hfb3BlcmF0b3IucHk=) | `82.5% <0%> (-1.25%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6265/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `56.89% <0%> (+0.34%)` | :arrow_up: | | [airflow/gcp/hooks/mlengine.py](https://codecov.io/gh/apache/airflow/pull/6265/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvbWxlbmdpbmUucHk=) | `78.63% <0%> (+57.26%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6265?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6265?src=pr=footer). Last update [1347b77...47a53fc](https://codecov.io/gh/apache/airflow/pull/6265?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-3160) Load latest_dagruns asynchronously
[ https://issues.apache.org/jira/browse/AIRFLOW-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-3160: Fix Version/s: (was: 2.0.0) > Load latest_dagruns asynchronously > --- > > Key: AIRFLOW-3160 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3160 > Project: Apache Airflow > Issue Type: Improvement > Components: webserver >Affects Versions: 1.10.0 >Reporter: Dan Davydov >Assignee: Dan Davydov >Priority: Major > Fix For: 1.10.6 > > > The front page loads very slowly when the DB has latency because one blocking > query is made per DAG against the DB. > > The latest dagruns should be loaded asynchronously and in batch like the > other UI elements that query the database. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-3160) Load latest_dagruns asynchronously
[ https://issues.apache.org/jira/browse/AIRFLOW-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-3160: Fix Version/s: 1.10.6 > Load latest_dagruns asynchronously > --- > > Key: AIRFLOW-3160 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3160 > Project: Apache Airflow > Issue Type: Improvement > Components: webserver >Affects Versions: 1.10.0 >Reporter: Dan Davydov >Assignee: Dan Davydov >Priority: Major > Fix For: 2.0.0, 1.10.6 > > > The front page loads very slowly when the DB has latency because one blocking > query is made per DAG against the DB. > > The latest dagruns should be loaded asynchronously and in batch like the > other UI elements that query the database. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] mik-laj opened a new pull request #6280: [AIRFLOW-XXX] Fix extra-packages tables
mik-laj opened a new pull request #6280: [AIRFLOW-XXX] Fix extra-packages tables
URL: https://github.com/apache/airflow/pull/6280
https://airflow.readthedocs.io/en/latest/installation.html
Something is broken
---
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (AIRFLOW-5613) Suport multiple LDAP Servers
[
https://issues.apache.org/jira/browse/AIRFLOW-5613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan Hansohn updated AIRFLOW-5613:
--
Description:
Since Airflow is already using ldap3 for LDAP connections, I would like to
propose supporting multiple LDAP servers for failover purposes.
This change would be minimal and only be invoked when the {{uri}} param was
populated with a comma-separated string of LDAP servers.
The following would continue to use a ldap3
[Server|[https://ldap3.readthedocs.io/server.html]] object just as it does
today:
{code:java}
uri = ldaps://:
{code}
But when a comma-separated value was included a ldap3
[ServerPool|[https://ldap3.readthedocs.io/server.html#server-pool]] object
would be instantiated instead:
{code:java}
uri = ldaps://:,ldaps://:
{code}
I wanted to poll the community and see if this is something others might find
useful. I have a PR I am ready to submit to implement this change. Thoughts?
was:
Since Airflow is already using ldap3 for LDAP connections, I would like to
propose supporting multiple LDAP servers for failover purposes.
This change would be minimal and only be invoked when the {{uri}} param was
populated with a comma-separated string of LDAP servers.
The following would continue to use a ldap3
[Server|[https://ldap3.readthedocs.io/server.html]] object just as it does
today:
{code:java}
uri = ldaps://:
{code}
But when a comma-separated value was included a ldap3 [ServerPool|#server-pool]
object would be instantiated instead:
{code:java}
uri = ldaps://:,ldaps://:
{code}
I wanted to poll the community and see if this is something others might find
useful. I have a PR I am ready to submit to implement this change. Thoughts?
> Suport multiple LDAP Servers
>
>
> Key: AIRFLOW-5613
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5613
> Project: Apache Airflow
> Issue Type: Improvement
> Components: authentication
>Affects Versions: 2.0.0
>Reporter: Ryan Hansohn
>Assignee: Ryan Hansohn
>Priority: Minor
>
> Since Airflow is already using ldap3 for LDAP connections, I would like to
> propose supporting multiple LDAP servers for failover purposes.
> This change would be minimal and only be invoked when the {{uri}} param was
> populated with a comma-separated string of LDAP servers.
> The following would continue to use a ldap3
> [Server|[https://ldap3.readthedocs.io/server.html]] object just as it does
> today:
> {code:java}
> uri = ldaps://:
> {code}
> But when a comma-separated value was included a ldap3
> [ServerPool|[https://ldap3.readthedocs.io/server.html#server-pool]] object
> would be instantiated instead:
> {code:java}
> uri = ldaps://:,ldaps://:
> {code}
> I wanted to poll the community and see if this is something others might find
> useful. I have a PR I am ready to submit to implement this change. Thoughts?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] mik-laj commented on issue #6265: [AIRFLOW-5597] Linkify urls in task instance log
mik-laj commented on issue #6265: [AIRFLOW-5597] Linkify urls in task instance log URL: https://github.com/apache/airflow/pull/6265#issuecomment-539274639 I restarted this Travis's job. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ZxMYS edited a comment on issue #6265: [AIRFLOW-5597] Linkify urls in task instance log
ZxMYS edited a comment on issue #6265: [AIRFLOW-5597] Linkify urls in task instance log URL: https://github.com/apache/airflow/pull/6265#issuecomment-539274377 Is there a way to re-run the checks? I don't think the error in travis ci is related to my change. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ZxMYS commented on issue #6265: [AIRFLOW-5597] Linkify urls in task instance log
ZxMYS commented on issue #6265: [AIRFLOW-5597] Linkify urls in task instance log URL: https://github.com/apache/airflow/pull/6265#issuecomment-539274377 Is there a way to re-run the checks? I don't think the error above is related to my change. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5605) Docker image requirement anaconda-client 1.7.2 not found
[
https://issues.apache.org/jira/browse/AIRFLOW-5605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946380#comment-16946380
]
Iemand commented on AIRFLOW-5605:
-
Ok, somehow the attachement didn't work,
`airflow/scripts/ci/kubernetes/docker/requirements.txt`:
{code:java}
alabaster==0.7.11
alembic==1.0.11
anaconda-client==1.7.2
anaconda-navigator==1.9.2
anaconda-project==0.8.2
apispec==2.0.2
appdirs==1.4.3
asn1crypto==0.24.0
astroid==2.0.4
astropy==3.0.4
atomicwrites==1.2.1
attrs==18.2.0
Automat==0.7.0
Babel==2.6.0
backcall==0.1.0
backports.shutil-get-terminal-size==1.0.0
beautifulsoup4==4.6.3
bitarray==0.8.3
bkcharts==0.2
blaze==0.11.3
bleach==2.1.4
bokeh==0.13.0
boto==2.49.0
Bottleneck==1.2.1
cached-property==1.5.1
certifi==2019.6.16
cffi==1.12.3
chardet==3.0.4
click==6.7
cloudpickle==0.5.5
clyent==1.2.2
colorama==0.3.9
colorlog==4.0.2
comet-git-pure==0.19.8
comet-ml==1.0.38
conda==4.7.10
conda-build==3.15.1
conda-package-handling==1.3.11
configobj==5.0.6
configparser==3.5.3
constantly==15.1.0
contextlib2==0.5.5
core-data-management==0.0.1
core-empath-boss==0.0.1
core-model-construction==0.0.1
croniter==0.3.30
cryptography==2.7
cycler==0.10.0
Cython==0.28.5
cytoolz==0.9.0.1
dask==0.19.1
datashape==0.5.4
decorator==4.3.0
defusedxml==0.5.0
dill==0.2.9
distributed==1.23.1
docutils==0.14
dumb-init==1.2.2
entrypoints==0.2.3
et-xmlfile==1.0.1
everett==0.9
fastcache==1.0.2
filelock==3.0.8
Flask==1.1.1
Flask-Admin==1.5.3
Flask-AppBuilder==1.13.1
Flask-Babel==0.12.2
Flask-Caching==1.3.3
Flask-Cors==3.0.6
Flask-JWT-Extended==3.21.0
Flask-Login==0.4.1
Flask-OpenID==1.2.5
Flask-SQLAlchemy==2.4.0
flask-swagger==0.2.13
Flask-WTF==0.14.2
funcsigs==1.0.0
future==0.16.0
gevent==1.3.6
glob2==0.6
gmpy2==2.0.8
greenlet==0.4.15
gunicorn==19.9.0
h5py==2.8.0
heapdict==1.0.0
html5lib==1.0.1
hyperlink==18.0.0
idna==2.8
imageio==2.4.1
imagesize==1.1.0
incremental==17.5.0
ipykernel==4.9.0
ipython==6.5.0
ipython-genutils==0.2.0
ipywidgets==7.4.1
iso8601==0.1.12
isort==4.3.4
itsdangerous==0.24
jdcal==1.4
jedi==0.12.1
jeepney==0.3.1
Jinja2==2.10.1
json-merge-patch==0.2
jsonschema==3.0.2
jupyter==1.0.0
jupyter-client==5.2.3
jupyter-console==5.2.0
jupyter-core==4.4.0
jupyterlab==0.34.9
jupyterlab-launcher==0.13.1
keyring==13.2.1
kiwisolver==1.0.1
lazy-object-proxy==1.3.1
libarchive-c==2.8
llvmlite==0.24.0
locket==0.2.0
lockfile==0.12.2
lxml==4.2.5
Mako==1.1.0
Markdown==2.6.11
MarkupSafe==1.0
marshmallow==2.19.5
marshmallow-enum==1.4.1
marshmallow-sqlalchemy==0.17.0
matplotlib==2.2.3
mccabe==0.6.1
mistune==0.8.3
mkl-fft==1.0.4
mkl-random==1.0.1
more-itertools==4.3.0
mpmath==1.0.0
msgpack==0.5.6
multipledispatch==0.6.0
navigator-updater==0.2.1
nbconvert==5.4.0
nbformat==4.4.0
netifaces==0.10.7
networkx==2.1
nltk==3.3
nose==1.3.7
notebook==5.6.0
numba==0.39.0
numexpr==2.6.8
numpy==1.15.1
numpydoc==0.8.0
nvidia-ml-py3==7.352.0
odo==0.5.1
olefile==0.46
openpyxl==2.5.6
ordereddict==1.1
packaging==17.1
pandas==0.23.4
pandocfilters==1.4.2
parso==0.3.1
partd==0.3.8
path.py==11.1.0
pathlib2==2.3.2
patsy==0.5.0
pendulum==1.4.4
pep8==1.7.1
pexpect==4.6.0
pickleshare==0.7.4
Pillow==5.2.0
pkginfo==1.4.2
pluggy==0.7.1
ply==3.11
prison==0.1.0
prometheus-client==0.3.1
prompt-toolkit==1.0.15
psutil==5.4.7
psycopg2==2.7.7
ptyprocess==0.6.0
py==1.6.0
pyasn1==0.4.4
pyasn1-modules==0.2.2
pycodestyle==2.4.0
pycosat==0.6.3
pycparser==2.19
pycrypto==2.6.1
pycurl==7.43.0.2
pyflakes==2.0.0
Pygments==2.2.0
PyJWT==1.7.1
pylint==2.1.1
pyodbc==4.0.24
pyOpenSSL==19.0.0
pyparsing==2.2.0
pyrsistent==0.15.4
PySocks==1.7.0
pytest==3.8.0
pytest-arraydiff==0.2
pytest-astropy==0.4.0
pytest-doctestplus==0.1.3
pytest-openfiles==0.3.0
pytest-remotedata==0.3.0
python-daemon==2.1.2
python-dateutil==2.7.3
python-editor==1.0.4
python3-openid==3.1.0
pytz==2018.5
pytzdata==2019.2
PyWavelets==1.0.0
PyYAML==3.13
pyzmq==17.1.2
QtAwesome==0.4.4
qtconsole==4.4.1
QtPy==1.5.0
requests==2.22.0
rope==0.11.0
ruamel-yaml==0.15.46
scikit-image==0.14.0
scikit-learn==0.19.2
scipy==1.1.0
seaborn==0.9.0
SecretStorage==3.1.0
Send2Trash==1.5.0
service-identity==17.0.0
setproctitle==1.1.10
simplegeneric==0.8.1
singledispatch==3.4.0.3
six==1.12.0
smop==0.41b0
snowballstemmer==1.2.1
sortedcollections==1.0.1
sortedcontainers==2.0.5
Sphinx==1.7.9
sphinxcontrib-websupport==1.1.0
spyder==3.3.1
spyder-kernels==0.2.6
SQLAlchemy==1.3.6
statsmodels==0.9.0
sympy==1.2
tables==3.4.4
tabulate==0.8.3
tblib==1.3.2
tenacity==4.12.0
termcolor==1.1.0
terminado==0.8.1
testpath==0.3.1
text-unidecode==1.2
thrift==0.11.0
toolz==0.9.0
tornado==5.1
tqdm==4.32.1
traitlets==4.3.2
Twisted==18.7.0
tzlocal==1.5.1
unicodecsv==0.14.1
urllib3==1.24.2
wcwidth==0.1.7
webencodings==0.5.1
websocket-client==0.54.0
Werkzeug==0.15.5
widgetsnbextension==3.4.1
wrapt==1.10.11
WTForms==2.2.1
wurlitzer==1.0.2
xgboost==0.90
xlrd==1.1.0
XlsxWriter==1.1.0
xlwt==1.3.0
zict==0.1.3
zope.deprecation==4.4.0
zope.interface==4.5.0{code}
> Docker image requirement anaconda-client 1.7.2
[jira] [Updated] (AIRFLOW-5613) Suport multiple LDAP Servers
[
https://issues.apache.org/jira/browse/AIRFLOW-5613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan Hansohn updated AIRFLOW-5613:
--
Description:
Since Airflow is already using ldap3 for LDAP connections, I would like to
propose supporting multiple LDAP servers for failover purposes.
This change would be minimal and only be invoked when the {{uri}} param was
populated with a comma-separated string of LDAP servers.
The following would continue to use a ldap3
[Server|[https://ldap3.readthedocs.io/server.html]] object just as it does
today:
{code:java}
uri = ldaps://:
{code}
But when a comma-separated value was included a ldap3 [ServerPool|#server-pool]
object would be instantiated instead:
{code:java}
uri = ldaps://:,ldaps://:
{code}
I wanted to poll the community and see if this is something others might find
useful. I have a PR I am ready to submit to implement this change. Thoughts?
was:
Since Airflow is already using ldap3 for LDAP connections, I would like to
propose supporting multiple LDAP servers for failover purposes.
This change would be minimal and only be invoked when the {{uri}} param was
populated with a comma-separated string of LDAP servers.
The following would continue to use a ldap3
[Server|[https://ldap3.readthedocs.io/server.html]] object just as it does
today:
{code:java}
uri = ldaps://:
{code}
But when a comma-separated value was included a ldap3
[ServerPool|#server-pool]] object would be instantiated instead:
{code:java}
uri = ldaps://:,ldaps://:
{code}
I wanted to poll the community and see if this is something others might find
useful. I have a PR I am ready to submit to implement this change. Thoughts?
> Suport multiple LDAP Servers
>
>
> Key: AIRFLOW-5613
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5613
> Project: Apache Airflow
> Issue Type: Improvement
> Components: authentication
>Affects Versions: 2.0.0
>Reporter: Ryan Hansohn
>Assignee: Ryan Hansohn
>Priority: Minor
>
> Since Airflow is already using ldap3 for LDAP connections, I would like to
> propose supporting multiple LDAP servers for failover purposes.
> This change would be minimal and only be invoked when the {{uri}} param was
> populated with a comma-separated string of LDAP servers.
> The following would continue to use a ldap3
> [Server|[https://ldap3.readthedocs.io/server.html]] object just as it does
> today:
> {code:java}
> uri = ldaps://:
> {code}
> But when a comma-separated value was included a ldap3
> [ServerPool|#server-pool] object would be instantiated instead:
> {code:java}
> uri = ldaps://:,ldaps://:
> {code}
> I wanted to poll the community and see if this is something others might find
> useful. I have a PR I am ready to submit to implement this change. Thoughts?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5605) Docker image requirement anaconda-client 1.7.2 not found
[
https://issues.apache.org/jira/browse/AIRFLOW-5605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946379#comment-16946379
]
Iemand commented on AIRFLOW-5605:
-
Attached `requirements.txt`
> Docker image requirement anaconda-client 1.7.2 not found
>
>
> Key: AIRFLOW-5605
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5605
> Project: Apache Airflow
> Issue Type: Bug
> Components: dependencies
>Affects Versions: 1.10.5
> Environment: OS: Ubuntu 18.04
> Docker: `docker version`
> Client:
> Version: 18.09.7
> API version: 1.39
> Go version:go1.10.8
> Git commit:2d0083d
> Built: Thu Jun 27 17:56:23 2019
> OS/Arch: linux/amd64
> Experimental: false
> Server: Docker Engine - Community
> Engine:
> Version: 18.09.7
> API version: 1.39 (minimum version 1.12)
> Go version: go1.10.8
> Git commit: 2d0083d
> Built:Thu Jun 27 17:23:02 2019
> OS/Arch: linux/amd64
> Experimental: false
>Reporter: Iemand
>Priority: Major
> Labels: Docker, dockerfile
> Attachments: airflow-docker_issue_anaconda-client1.7.2.txt
>
>
> Trying to build an Airflow Docker image, but hitting the error:
> {code:java}
> Step 6/12 : RUN pip install -r /tmp/requirements.txt
> ---> Running in dad143582dff
> Collecting alabaster==0.7.11 (from -r /tmp/requirements.txt (line 1))
> Downloading
> https://files.pythonhosted.org/packages/6e/71/c3648cc2f675063dbe2d669004a59e4a5120172713a1de3c3b14144d4b31/alabaster-0.7.11-py2.py3-none-any.whl
> Collecting alembic==1.0.11 (from -r /tmp/requirements.txt (line 2))
> Downloading
> https://files.pythonhosted.org/packages/7b/8b/0c98c378d93165d9809193f274c3c6e2151120d955b752419c7d43e4d857/alembic-1.0.11.tar.gz
> (1.0MB)
> Collecting anaconda-client==1.7.2 (from -r /tmp/requirements.txt (line 3))
> ERROR: Could not find a version that satisfies the requirement
> anaconda-client==1.7.2 (from -r /tmp/requirements.txt (line 3)) (from
> versions: 1.1.1, 1.2.2)
> ERROR: No matching distribution found for anaconda-client==1.7.2 (from -r
> /tmp/requirements.txt (line 3))
> The command '/bin/sh -c pip install -r /tmp/requirements.txt' returned a
> non-zero code: 1{code}
>
> Please see attached .txt for full output.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5613) Suport multiple LDAP Servers
[
https://issues.apache.org/jira/browse/AIRFLOW-5613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan Hansohn updated AIRFLOW-5613:
--
Description:
Since Airflow is already using ldap3 for LDAP connections, I would like to
propose supporting multiple LDAP servers for failover purposes.
This change would be minimal and only be invoked when the {{uri}} param was
populated with a comma-separated string of LDAP servers.
The following would continue to use a ldap3
[Server|[https://ldap3.readthedocs.io/server.html]] object just as it does
today:
{code:java}
uri = ldaps://:
{code}
But when a comma-separated value was included a ldap3
[ServerPool|#server-pool]] object would be instantiated instead:
{code:java}
uri = ldaps://:,ldaps://:
{code}
I wanted to poll the community and see if this is something others might find
useful. I have a PR I am ready to submit to implement this change. Thoughts?
was:
Since Airflow is already using ldap3 for LDAP connections I would like to
propose supporting multiple LDAP servers for failover purposes.
This change would be minimal and only be invoked when the {{uri}} param was
populated with a comma-separated string of LDAP servers.
The following would continue to use a ldap3
[Server|[https://ldap3.readthedocs.io/server.html]] object just as it does
today:
{code:java}
uri = ldaps://:
{code}
But when a comma-separated value was included a ldap3
[ServerPool|[https://ldap3.readthedocs.io/server.html#server-pool]] object
would be instantiated instead:
{code:java}
uri = ldaps://:,ldaps://:
{code}
I wanted to poll the community and see if this is something others might find
useful. I have a PR I am ready to submit to implement this change. Thoughts?
> Suport multiple LDAP Servers
>
>
> Key: AIRFLOW-5613
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5613
> Project: Apache Airflow
> Issue Type: Improvement
> Components: authentication
>Affects Versions: 2.0.0
>Reporter: Ryan Hansohn
>Assignee: Ryan Hansohn
>Priority: Minor
>
> Since Airflow is already using ldap3 for LDAP connections, I would like to
> propose supporting multiple LDAP servers for failover purposes.
> This change would be minimal and only be invoked when the {{uri}} param was
> populated with a comma-separated string of LDAP servers.
> The following would continue to use a ldap3
> [Server|[https://ldap3.readthedocs.io/server.html]] object just as it does
> today:
> {code:java}
> uri = ldaps://:
> {code}
> But when a comma-separated value was included a ldap3
> [ServerPool|#server-pool]] object would be instantiated instead:
> {code:java}
> uri = ldaps://:,ldaps://:
> {code}
> I wanted to poll the community and see if this is something others might find
> useful. I have a PR I am ready to submit to implement this change. Thoughts?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (AIRFLOW-5613) Suport multiple LDAP Servers
Ryan Hansohn created AIRFLOW-5613:
-
Summary: Suport multiple LDAP Servers
Key: AIRFLOW-5613
URL: https://issues.apache.org/jira/browse/AIRFLOW-5613
Project: Apache Airflow
Issue Type: Improvement
Components: authentication
Affects Versions: 2.0.0
Reporter: Ryan Hansohn
Assignee: Ryan Hansohn
Since Airflow is already using ldap3 for LDAP connections I would like to
propose supporting multiple LDAP servers for failover purposes.
This change would be minimal and only be invoked when the {{uri}} param was
populated with a comma-separated string of LDAP servers.
The following would continue to use a ldap3
[Server|[https://ldap3.readthedocs.io/server.html]] object just as it does
today:
{code:java}
uri = ldaps://:
{code}
But when a comma-separated value was included a ldap3
[ServerPool|[https://ldap3.readthedocs.io/server.html#server-pool]] object
would be instantiated instead:
{code:java}
uri = ldaps://:,ldaps://:
{code}
I wanted to poll the community and see if this is something others might find
useful. I have a PR I am ready to submit to implement this change. Thoughts?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] codecov-io edited a comment on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized DAG in DB for webserver scalability
codecov-io edited a comment on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized DAG in DB for webserver scalability URL: https://github.com/apache/airflow/pull/5743#issuecomment-529755241 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5743?src=pr=h1) Report > Merging [#5743](https://codecov.io/gh/apache/airflow/pull/5743?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/b2dc1636855dcd3ad6957745e5426cd4f5f10c70?src=pr=desc) will **increase** coverage by `0.9%`. > The diff coverage is `85.41%`. [](https://codecov.io/gh/apache/airflow/pull/5743?src=pr=tree) ```diff @@Coverage Diff@@ ## master#5743 +/- ## = + Coverage 80.09% 80.99% +0.9% = Files 612 613 +1 Lines 3532035583+263 = + Hits2828928821+532 + Misses 7031 6762-269 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5743?src=pr=tree) | Coverage Δ | | |---|---|---| | [...rflow/dag/serialization/serialized\_baseoperator.py](https://codecov.io/gh/apache/airflow/pull/5743/diff?src=pr=tree#diff-YWlyZmxvdy9kYWcvc2VyaWFsaXphdGlvbi9zZXJpYWxpemVkX2Jhc2VvcGVyYXRvci5weQ==) | `98.38% <100%> (+41.72%)` | :arrow_up: | | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/5743/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `95.57% <100%> (+0.03%)` | :arrow_up: | | [airflow/settings.py](https://codecov.io/gh/apache/airflow/pull/5743/diff?src=pr=tree#diff-YWlyZmxvdy9zZXR0aW5ncy5weQ==) | `88.48% <100%> (+0.16%)` | :arrow_up: | | [airflow/utils/log/logging\_mixin.py](https://codecov.io/gh/apache/airflow/pull/5743/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvbG9nZ2luZ19taXhpbi5weQ==) | `96.15% <100%> (+0.04%)` | :arrow_up: | | [airflow/www/utils.py](https://codecov.io/gh/apache/airflow/pull/5743/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdXRpbHMucHk=) | `79.9% <100%> (+4.65%)` | :arrow_up: | | [airflow/dag/serialization/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/5743/diff?src=pr=tree#diff-YWlyZmxvdy9kYWcvc2VyaWFsaXphdGlvbi9fX2luaXRfXy5weQ==) | `100% <100%> (ø)` | :arrow_up: | | [airflow/dag/serialization/enums.py](https://codecov.io/gh/apache/airflow/pull/5743/diff?src=pr=tree#diff-YWlyZmxvdy9kYWcvc2VyaWFsaXphdGlvbi9lbnVtcy5weQ==) | `100% <100%> (ø)` | | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/5743/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `75.19% <100%> (ø)` | :arrow_up: | | [airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/5743/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=) | `100% <100%> (ø)` | :arrow_up: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/5743/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `56.14% <16.66%> (-0.41%)` | :arrow_down: | | ... and [41 more](https://codecov.io/gh/apache/airflow/pull/5743/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5743?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5743?src=pr=footer). Last update [b2dc163...7bcfb9f](https://codecov.io/gh/apache/airflow/pull/5743?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5605) Docker image requirement anaconda-client 1.7.2 not found
[
https://issues.apache.org/jira/browse/AIRFLOW-5605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946375#comment-16946375
]
Iemand commented on AIRFLOW-5605:
-
[~ash]
I would assume that too, but somehow the `requirements.txt` downloaded through
the `build.sh` line 47 has this included:
[https://github.com/apache/airflow/blob/master/scripts/ci/kubernetes/docker/build.sh#L47]
I just followed the steps given by this README.md:
[https://github.com/apache/airflow/tree/master/scripts/ci/kubernetes]
Doing: `{{./docker/build.sh}}`
> Docker image requirement anaconda-client 1.7.2 not found
>
>
> Key: AIRFLOW-5605
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5605
> Project: Apache Airflow
> Issue Type: Bug
> Components: dependencies
>Affects Versions: 1.10.5
> Environment: OS: Ubuntu 18.04
> Docker: `docker version`
> Client:
> Version: 18.09.7
> API version: 1.39
> Go version:go1.10.8
> Git commit:2d0083d
> Built: Thu Jun 27 17:56:23 2019
> OS/Arch: linux/amd64
> Experimental: false
> Server: Docker Engine - Community
> Engine:
> Version: 18.09.7
> API version: 1.39 (minimum version 1.12)
> Go version: go1.10.8
> Git commit: 2d0083d
> Built:Thu Jun 27 17:23:02 2019
> OS/Arch: linux/amd64
> Experimental: false
>Reporter: Iemand
>Priority: Major
> Labels: Docker, dockerfile
> Attachments: airflow-docker_issue_anaconda-client1.7.2.txt
>
>
> Trying to build an Airflow Docker image, but hitting the error:
> {code:java}
> Step 6/12 : RUN pip install -r /tmp/requirements.txt
> ---> Running in dad143582dff
> Collecting alabaster==0.7.11 (from -r /tmp/requirements.txt (line 1))
> Downloading
> https://files.pythonhosted.org/packages/6e/71/c3648cc2f675063dbe2d669004a59e4a5120172713a1de3c3b14144d4b31/alabaster-0.7.11-py2.py3-none-any.whl
> Collecting alembic==1.0.11 (from -r /tmp/requirements.txt (line 2))
> Downloading
> https://files.pythonhosted.org/packages/7b/8b/0c98c378d93165d9809193f274c3c6e2151120d955b752419c7d43e4d857/alembic-1.0.11.tar.gz
> (1.0MB)
> Collecting anaconda-client==1.7.2 (from -r /tmp/requirements.txt (line 3))
> ERROR: Could not find a version that satisfies the requirement
> anaconda-client==1.7.2 (from -r /tmp/requirements.txt (line 3)) (from
> versions: 1.1.1, 1.2.2)
> ERROR: No matching distribution found for anaconda-client==1.7.2 (from -r
> /tmp/requirements.txt (line 3))
> The command '/bin/sh -c pip install -r /tmp/requirements.txt' returned a
> non-zero code: 1{code}
>
> Please see attached .txt for full output.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] codecov-io edited a comment on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized DAG in DB for webserver scalability
codecov-io edited a comment on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized DAG in DB for webserver scalability URL: https://github.com/apache/airflow/pull/5743#issuecomment-529755241 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5743?src=pr=h1) Report > Merging [#5743](https://codecov.io/gh/apache/airflow/pull/5743?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/b2dc1636855dcd3ad6957745e5426cd4f5f10c70?src=pr=desc) will **increase** coverage by `0.9%`. > The diff coverage is `85.41%`. [](https://codecov.io/gh/apache/airflow/pull/5743?src=pr=tree) ```diff @@Coverage Diff@@ ## master#5743 +/- ## = + Coverage 80.09% 80.99% +0.9% = Files 612 613 +1 Lines 3532035583+263 = + Hits2828928821+532 + Misses 7031 6762-269 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5743?src=pr=tree) | Coverage Δ | | |---|---|---| | [...rflow/dag/serialization/serialized\_baseoperator.py](https://codecov.io/gh/apache/airflow/pull/5743/diff?src=pr=tree#diff-YWlyZmxvdy9kYWcvc2VyaWFsaXphdGlvbi9zZXJpYWxpemVkX2Jhc2VvcGVyYXRvci5weQ==) | `98.38% <100%> (+41.72%)` | :arrow_up: | | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/5743/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `95.57% <100%> (+0.03%)` | :arrow_up: | | [airflow/settings.py](https://codecov.io/gh/apache/airflow/pull/5743/diff?src=pr=tree#diff-YWlyZmxvdy9zZXR0aW5ncy5weQ==) | `88.48% <100%> (+0.16%)` | :arrow_up: | | [airflow/utils/log/logging\_mixin.py](https://codecov.io/gh/apache/airflow/pull/5743/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvbG9nZ2luZ19taXhpbi5weQ==) | `96.15% <100%> (+0.04%)` | :arrow_up: | | [airflow/www/utils.py](https://codecov.io/gh/apache/airflow/pull/5743/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdXRpbHMucHk=) | `79.9% <100%> (+4.65%)` | :arrow_up: | | [airflow/dag/serialization/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/5743/diff?src=pr=tree#diff-YWlyZmxvdy9kYWcvc2VyaWFsaXphdGlvbi9fX2luaXRfXy5weQ==) | `100% <100%> (ø)` | :arrow_up: | | [airflow/dag/serialization/enums.py](https://codecov.io/gh/apache/airflow/pull/5743/diff?src=pr=tree#diff-YWlyZmxvdy9kYWcvc2VyaWFsaXphdGlvbi9lbnVtcy5weQ==) | `100% <100%> (ø)` | | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/5743/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `75.19% <100%> (ø)` | :arrow_up: | | [airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/5743/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=) | `100% <100%> (ø)` | :arrow_up: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/5743/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `56.14% <16.66%> (-0.41%)` | :arrow_down: | | ... and [41 more](https://codecov.io/gh/apache/airflow/pull/5743/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5743?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5743?src=pr=footer). Last update [b2dc163...7bcfb9f](https://codecov.io/gh/apache/airflow/pull/5743?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6169: [AIRFLOW-4970] Add Google Campaign Manager integration
codecov-io edited a comment on issue #6169: [AIRFLOW-4970] Add Google Campaign Manager integration URL: https://github.com/apache/airflow/pull/6169#issuecomment-534125545 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6169?src=pr=h1) Report > Merging [#6169](https://codecov.io/gh/apache/airflow/pull/6169?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/76fe45e1d127b657b1aad5c0fd657e011f5a09bc?src=pr=desc) will **increase** coverage by `0.25%`. > The diff coverage is `85.64%`. [](https://codecov.io/gh/apache/airflow/pull/6169?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6169 +/- ## == + Coverage 80.05% 80.31% +0.25% == Files 610 616 +6 Lines 3526435529 +265 == + Hits2823228535 +303 + Misses 7032 6994 -38 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6169?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/gcp/operators/bigquery.py](https://codecov.io/gh/apache/airflow/pull/6169/diff?src=pr=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2JpZ3F1ZXJ5LnB5) | `86.6% <ø> (ø)` | :arrow_up: | | [...\_platform/example\_dags/example\_campaign\_manager.py](https://codecov.io/gh/apache/airflow/pull/6169/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlci9nb29nbGUvbWFya2V0aW5nX3BsYXRmb3JtL2V4YW1wbGVfZGFncy9leGFtcGxlX2NhbXBhaWduX21hbmFnZXIucHk=) | `0% <0%> (ø)` | | | [...gle/marketing\_platform/sensors/campaign\_manager.py](https://codecov.io/gh/apache/airflow/pull/6169/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlci9nb29nbGUvbWFya2V0aW5nX3BsYXRmb3JtL3NlbnNvcnMvY2FtcGFpZ25fbWFuYWdlci5weQ==) | `100% <100%> (ø)` | | | [...oogle/marketing\_platform/hooks/campaign\_manager.py](https://codecov.io/gh/apache/airflow/pull/6169/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlci9nb29nbGUvbWFya2V0aW5nX3BsYXRmb3JtL2hvb2tzL2NhbXBhaWduX21hbmFnZXIucHk=) | `100% <100%> (ø)` | | | [...e/marketing\_platform/operators/campaign\_manager.py](https://codecov.io/gh/apache/airflow/pull/6169/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlci9nb29nbGUvbWFya2V0aW5nX3BsYXRmb3JtL29wZXJhdG9ycy9jYW1wYWlnbl9tYW5hZ2VyLnB5) | `91.73% <91.73%> (ø)` | | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6169/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `56.72% <0%> (-0.18%)` | :arrow_down: | | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/6169/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `66.17% <0%> (ø)` | :arrow_up: | | [airflow/gcp/utils/mlengine\_operator\_utils.py](https://codecov.io/gh/apache/airflow/pull/6169/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvdXRpbHMvbWxlbmdpbmVfb3BlcmF0b3JfdXRpbHMucHk=) | `95.34% <0%> (ø)` | :arrow_up: | | [airflow/task/task\_runner/cgroup\_task\_runner.py](https://codecov.io/gh/apache/airflow/pull/6169/diff?src=pr=tree#diff-YWlyZmxvdy90YXNrL3Rhc2tfcnVubmVyL2Nncm91cF90YXNrX3J1bm5lci5weQ==) | `34.37% <0%> (ø)` | | | ... and [8 more](https://codecov.io/gh/apache/airflow/pull/6169/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6169?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6169?src=pr=footer). Last update [76fe45e...388d29e](https://codecov.io/gh/apache/airflow/pull/6169?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] jaketf commented on a change in pull request #6210: [AIRFLOW-5567] [Do not Merge] prototype BaseAsyncOperator
jaketf commented on a change in pull request #6210: [AIRFLOW-5567] [Do not Merge] prototype BaseAsyncOperator URL: https://github.com/apache/airflow/pull/6210#discussion_r332293940 ## File path: airflow/models/base_async_operator.py ## @@ -0,0 +1,168 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +""" +Base Asynchronous Operator for kicking off a long running +operations and polling for completion with reschedule mode. +""" + +from abc import abstractmethod +from typing import Dict, List, Optional, Union + +from airflow.models import SkipMixin, TaskReschedule +from airflow.models.xcom import XCOM_EXTERNAL_RESOURCE_ID_KEY +from airflow.sensors.base_sensor_operator import BaseSensorOperator +from airflow.utils.decorators import apply_defaults + +PLACEHOLDER_RESOURCE_ID = 'RESOURCE_ID_NOT_APPLICABLE' + + +class BaseAsyncOperator(BaseSensorOperator, SkipMixin): +""" +AsyncOperators are derived from this class and inherit these attributes. +AsyncOperators should be used for long running operations where the task +can tolerate a longer poke interval. They use the task rescheduling +mechanism similar to sensors to avoid occupying a worker slot between +pokes. + +Developing concrete operators that provide parameterized flexibility +for synchronous or asynchronous poking depending on the invocation is +possible by programing against this `BaseAsyncOperator` interface, +and overriding the execute method as demonstrated below. + +```python3 +class DummyFlexiblePokingOperator(BaseAsyncOperator): + def __init__(self, async=False, *args, **kwargs): +self.async = async +super().__init(*args, **kwargs) + + def execute(self, context: Dict) -> None: +if self.async: + # use the BaseAsyncOperator's execute + super().execute(context) +else: + self.submit_request(context) + while not self.poke(): +time.sleep(self.poke_interval) + self.process_results(context) + + def sumbit_request(self, context: Dict) -> Optional[str]: +return None + + def poke(self, context: Dict) -> bool: +return bool(random.getrandbits(1)) +``` + +AsyncOperators must override the following methods: +:py:meth:`submit_request`: fire a request for a long running operation +:py:meth:`poke`: a method to check if the long running operation is +complete it should return True when a success criteria is met. + +Optionally, AsyncOperators can override: +:py:meth: `process_result` to perform any operations after the success +criteria is met in :py:meth: `poke` + +:py:meth: `poke` is executed at a time interval and succeed when a +criteria is met and fail if and when they time out. + +:param soft_fail: Set to true to mark the task as SKIPPED on failure +:type soft_fail: bool +:param poke_interval: Time in seconds that the job should wait in +between each tries +:type poke_interval: int +:param timeout: Time, in seconds before the task times out and fails. +:type timeout: int + +""" +ui_color = '#9933ff' # type: str + +@apply_defaults +def __init__(self, + *args, + **kwargs) -> None: +super().__init__(mode='reschedule', *args, **kwargs) + +@abstractmethod +def submit_request(self, context: Dict) -> Optional[Union[str, List, Dict]]: +""" +This method should kick off a long running operation. +This method should return the ID for the long running operation if +applicable. +Context is the same dictionary used as when rendering jinja templates. + +Refer to get_template_context for more context. + +:returns: a resource_id for the long running operation. +:rtype: Optional[Union[String, List, Dict]] +""" +raise NotImplementedError + +def process_result(self, context: Dict): +""" +This method can optionally be overriden to process the result of a long running operation. +Context is the same dictionary used as when rendering jinja templates. + +Refer to
[GitHub] [airflow] jaketf commented on a change in pull request #6210: [AIRFLOW-5567] [Do not Merge] prototype BaseAsyncOperator
jaketf commented on a change in pull request #6210: [AIRFLOW-5567] [Do not Merge] prototype BaseAsyncOperator URL: https://github.com/apache/airflow/pull/6210#discussion_r332294008 ## File path: airflow/models/base_async_operator.py ## @@ -0,0 +1,168 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +""" +Base Asynchronous Operator for kicking off a long running +operations and polling for completion with reschedule mode. +""" + +from abc import abstractmethod +from typing import Dict, List, Optional, Union + +from airflow.models import SkipMixin, TaskReschedule +from airflow.models.xcom import XCOM_EXTERNAL_RESOURCE_ID_KEY +from airflow.sensors.base_sensor_operator import BaseSensorOperator +from airflow.utils.decorators import apply_defaults + +PLACEHOLDER_RESOURCE_ID = 'RESOURCE_ID_NOT_APPLICABLE' + + +class BaseAsyncOperator(BaseSensorOperator, SkipMixin): +""" +AsyncOperators are derived from this class and inherit these attributes. +AsyncOperators should be used for long running operations where the task +can tolerate a longer poke interval. They use the task rescheduling +mechanism similar to sensors to avoid occupying a worker slot between +pokes. + +Developing concrete operators that provide parameterized flexibility +for synchronous or asynchronous poking depending on the invocation is +possible by programing against this `BaseAsyncOperator` interface, +and overriding the execute method as demonstrated below. + +```python3 +class DummyFlexiblePokingOperator(BaseAsyncOperator): + def __init__(self, async=False, *args, **kwargs): +self.async = async +super().__init(*args, **kwargs) + + def execute(self, context: Dict) -> None: +if self.async: + # use the BaseAsyncOperator's execute + super().execute(context) +else: + self.submit_request(context) + while not self.poke(): +time.sleep(self.poke_interval) + self.process_results(context) + + def sumbit_request(self, context: Dict) -> Optional[str]: +return None + + def poke(self, context: Dict) -> bool: +return bool(random.getrandbits(1)) +``` + +AsyncOperators must override the following methods: +:py:meth:`submit_request`: fire a request for a long running operation +:py:meth:`poke`: a method to check if the long running operation is +complete it should return True when a success criteria is met. + +Optionally, AsyncOperators can override: +:py:meth: `process_result` to perform any operations after the success +criteria is met in :py:meth: `poke` + +:py:meth: `poke` is executed at a time interval and succeed when a +criteria is met and fail if and when they time out. + +:param soft_fail: Set to true to mark the task as SKIPPED on failure +:type soft_fail: bool +:param poke_interval: Time in seconds that the job should wait in +between each tries +:type poke_interval: int +:param timeout: Time, in seconds before the task times out and fails. +:type timeout: int + +""" +ui_color = '#9933ff' # type: str + +@apply_defaults +def __init__(self, + *args, + **kwargs) -> None: +super().__init__(mode='reschedule', *args, **kwargs) + +@abstractmethod +def submit_request(self, context: Dict) -> Optional[Union[str, List, Dict]]: +""" +This method should kick off a long running operation. +This method should return the ID for the long running operation if +applicable. +Context is the same dictionary used as when rendering jinja templates. + +Refer to get_template_context for more context. + +:returns: a resource_id for the long running operation. +:rtype: Optional[Union[String, List, Dict]] +""" +raise NotImplementedError + +def process_result(self, context: Dict): +""" +This method can optionally be overriden to process the result of a long running operation. +Context is the same dictionary used as when rendering jinja templates. + +Refer to
[GitHub] [airflow] codecov-io commented on issue #6278: [AIRFLOW-XXX] Fix missing backtick in Breeze.rst
codecov-io commented on issue #6278: [AIRFLOW-XXX] Fix missing backtick in Breeze.rst URL: https://github.com/apache/airflow/pull/6278#issuecomment-539250468 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6278?src=pr=h1) Report > Merging [#6278](https://codecov.io/gh/apache/airflow/pull/6278?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/1347b77970b86503adc1f9b54f766c4dce89bc59?src=pr=desc) will **increase** coverage by `0.19%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6278?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6278 +/- ## == + Coverage 80.09% 80.28% +0.19% == Files 612 612 Lines 3532035320 == + Hits2828928357 +68 + Misses 7031 6963 -68 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6278?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/contrib/operators/ssh\_operator.py](https://codecov.io/gh/apache/airflow/pull/6278/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zc2hfb3BlcmF0b3IucHk=) | `82.5% <0%> (-1.25%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6278/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `56.89% <0%> (+0.34%)` | :arrow_up: | | [airflow/gcp/hooks/mlengine.py](https://codecov.io/gh/apache/airflow/pull/6278/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvbWxlbmdpbmUucHk=) | `78.63% <0%> (+57.26%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6278?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6278?src=pr=footer). Last update [1347b77...df89551](https://codecov.io/gh/apache/airflow/pull/6278?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-5612) Add ability to actually do things with created and modified date in GoogleCloudStorageHook
Joel Croteau created AIRFLOW-5612:
-
Summary: Add ability to actually do things with created and
modified date in GoogleCloudStorageHook
Key: AIRFLOW-5612
URL: https://issues.apache.org/jira/browse/AIRFLOW-5612
Project: Apache Airflow
Issue Type: Improvement
Components: gcp
Affects Versions: 1.10.5
Reporter: Joel Croteau
{{GoogleCloudStorageHook}} seems to support only a very small subset of the
actual GCS API. In particular, the only thing it allows you to do with the date
of an object is check if the metadata was updated after a specified time using
{{is_updated_after}}. First of all, this only looks at the metadata update
date, which is probably not what is wanted for most purposes, as
{{timeCreated}} is generally what conveys useful information. Second of all, it
seems rather arbitrary to only allow me to compare if the updated time is
greater than some other time, and not just give me the time and let me make my
own inferences. In particular, for a scheduled workflow with a potential
backfill, I would like to check for a creation date with both a minimum and
maximum value, which this doesn't allow.
Also, tangentially, if you want to get multiple pieces of information on an
object, using {{GoogleCloudStorageHook}} will necessitate a separate call to
{{objects().get()}} for every piece of information, even though everything is
returned by the one call. Would it not make more sense to be able to return an
object structure with all of the needed information in it?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] kaxil merged pull request #6278: [AIRFLOW-XXX] Fix missing backtick in Breeze.rst
kaxil merged pull request #6278: [AIRFLOW-XXX] Fix missing backtick in Breeze.rst URL: https://github.com/apache/airflow/pull/6278 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-5611) Add ability to specify a maximum modified time for objects in GoogleCloudStorageToGoogleCloudStorageOperator
Joel Croteau created AIRFLOW-5611: - Summary: Add ability to specify a maximum modified time for objects in GoogleCloudStorageToGoogleCloudStorageOperator Key: AIRFLOW-5611 URL: https://issues.apache.org/jira/browse/AIRFLOW-5611 Project: Apache Airflow Issue Type: Improvement Components: gcp Affects Versions: 1.10.5 Reporter: Joel Croteau The fact that I can specify a minimum modified time to filter objects on in GoogleCloudStorageToGoogleCloudStorageOperator but not a maximum seems rather arbitrary. Especially considering the typical usage scenario of running a copy on a schedule, I would like to be able to find objects created within a particular schedule interval for my execution, and not just copy all of the latest objects. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] codecov-io edited a comment on issue #6277: [AIRFLOW-2971] Add health check CLI for scheduler
codecov-io edited a comment on issue #6277: [AIRFLOW-2971] Add health check CLI for scheduler URL: https://github.com/apache/airflow/pull/6277#issuecomment-539234117 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6277?src=pr=h1) Report > Merging [#6277](https://codecov.io/gh/apache/airflow/pull/6277?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/1347b77970b86503adc1f9b54f766c4dce89bc59?src=pr=desc) will **increase** coverage by `0.18%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6277?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6277 +/- ## == + Coverage 80.09% 80.28% +0.18% == Files 612 612 Lines 3532035330 +10 == + Hits2828928364 +75 + Misses 7031 6966 -65 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6277?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/6277/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `66.52% <100%> (+0.35%)` | :arrow_up: | | [airflow/contrib/operators/ssh\_operator.py](https://codecov.io/gh/apache/airflow/pull/6277/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zc2hfb3BlcmF0b3IucHk=) | `82.5% <0%> (-1.25%)` | :arrow_down: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6277/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.24% <0%> (-0.51%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6277/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `56.89% <0%> (+0.34%)` | :arrow_up: | | [airflow/gcp/hooks/mlengine.py](https://codecov.io/gh/apache/airflow/pull/6277/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvbWxlbmdpbmUucHk=) | `78.63% <0%> (+57.26%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6277?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6277?src=pr=footer). Last update [1347b77...e62a78e](https://codecov.io/gh/apache/airflow/pull/6277?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-5610) Add ability to specify multiple objects to copy to GoogleCloudStorageToGoogleCloudStorageOperator
Joel Croteau created AIRFLOW-5610: - Summary: Add ability to specify multiple objects to copy to GoogleCloudStorageToGoogleCloudStorageOperator Key: AIRFLOW-5610 URL: https://issues.apache.org/jira/browse/AIRFLOW-5610 Project: Apache Airflow Issue Type: Improvement Components: gcp Affects Versions: 1.10.5 Reporter: Joel Croteau The restriction in GoogleCloudStorageToGoogleCloudStorageOperator that I am only allowed to specify a single object to list is rather arbitrary. If I specify a wildcard, all it does is split at the wildcard and use that to get a prefix and delimiter. Why not just let me do this search myself and return a list of objects? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] codecov-io commented on issue #6277: [AIRFLOW-2971] Add health check CLI for scheduler
codecov-io commented on issue #6277: [AIRFLOW-2971] Add health check CLI for scheduler URL: https://github.com/apache/airflow/pull/6277#issuecomment-539234117 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6277?src=pr=h1) Report > Merging [#6277](https://codecov.io/gh/apache/airflow/pull/6277?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/1347b77970b86503adc1f9b54f766c4dce89bc59?src=pr=desc) will **increase** coverage by `0.18%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6277?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6277 +/- ## == + Coverage 80.09% 80.28% +0.18% == Files 612 612 Lines 3532035330 +10 == + Hits2828928364 +75 + Misses 7031 6966 -65 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6277?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/6277/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `66.52% <100%> (+0.35%)` | :arrow_up: | | [airflow/contrib/operators/ssh\_operator.py](https://codecov.io/gh/apache/airflow/pull/6277/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zc2hfb3BlcmF0b3IucHk=) | `82.5% <0%> (-1.25%)` | :arrow_down: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6277/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.24% <0%> (-0.51%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6277/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `56.89% <0%> (+0.34%)` | :arrow_up: | | [airflow/gcp/hooks/mlengine.py](https://codecov.io/gh/apache/airflow/pull/6277/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvbWxlbmdpbmUucHk=) | `78.63% <0%> (+57.26%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6277?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6277?src=pr=footer). Last update [1347b77...e62a78e](https://codecov.io/gh/apache/airflow/pull/6277?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #6232: [AIRFLOW-5582] AutoCommit in jdbc is missing get_autocommit
feluelle commented on a change in pull request #6232: [AIRFLOW-5582] AutoCommit in jdbc is missing get_autocommit URL: https://github.com/apache/airflow/pull/6232#discussion_r332258183 ## File path: airflow/hooks/jdbc_hook.py ## @@ -57,3 +57,16 @@ def set_autocommit(self, conn, autocommit): :return: """ conn.jconn.setAutoCommit(autocommit) + +def get_autocommit(self, conn): Review comment: Would also be nice if you could do it for the set_autocommit as well (see [here](https://codecov.io/gh/apache/airflow/src/master/airflow/hooks/jdbc_hook.py#L59)) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #6232: [AIRFLOW-5582] AutoCommit in jdbc is missing get_autocommit
feluelle commented on a change in pull request #6232: [AIRFLOW-5582] AutoCommit in jdbc is missing get_autocommit URL: https://github.com/apache/airflow/pull/6232#discussion_r332258340 ## File path: airflow/hooks/jdbc_hook.py ## @@ -57,3 +57,16 @@ def set_autocommit(self, conn, autocommit): :return: """ conn.jconn.setAutoCommit(autocommit) + +def get_autocommit(self, conn): +""" +Get autocommit setting for the provided connection. +Return True if conn.autocommit is set to True. +Return False if conn.autocommit is not set or set to False + +:param conn: The connection Review comment: Could you also add the type, please? Either via the docs `:type` or in the function header directly to the arg `conn:` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #6232: [AIRFLOW-5582] AutoCommit in jdbc is missing get_autocommit
feluelle commented on a change in pull request #6232: [AIRFLOW-5582] AutoCommit in jdbc is missing get_autocommit URL: https://github.com/apache/airflow/pull/6232#discussion_r332257897 ## File path: airflow/hooks/jdbc_hook.py ## @@ -57,3 +57,16 @@ def set_autocommit(self, conn, autocommit): :return: """ conn.jconn.setAutoCommit(autocommit) + +def get_autocommit(self, conn): Review comment: Please add a test so that `getAutoCommit` has been called when calling `get_autocommit`. I know it might seems superflous but if the function gets refactored (or changed) in the future we need to make sure to not break sth. / breaking backwards compatiblity, etc. :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil merged pull request #6257: [AIRFLOW-XXX] Split extra packages table in multiple
kaxil merged pull request #6257: [AIRFLOW-XXX] Split extra packages table in multiple URL: https://github.com/apache/airflow/pull/6257 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on issue #6257: [AIRFLOW-XXX] Split extra packages table in multiple
kaxil commented on issue #6257: [AIRFLOW-XXX] Split extra packages table in multiple URL: https://github.com/apache/airflow/pull/6257#issuecomment-539223298 > As an Apache foundation project, we should promote other projects that are that foundation. This allows you to build a strong community. > > This layout will be uniform with the layout that is in Operators and Hooks Reference: https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html#asf-apache-software-foundation > In my opinion, this layout allows you to find faster, because the most used things are in a separate table - foundament. Very often, when you want to install something, you try to look for a specific type of integration. If you have resources, you are looking for software. If you have limited resources, you try to find a service that can do it for you. This is a very clear division in my opinion. Makes Sense This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle commented on issue #6257: [AIRFLOW-XXX] Split extra packages table in multiple
feluelle commented on issue #6257: [AIRFLOW-XXX] Split extra packages table in multiple URL: https://github.com/apache/airflow/pull/6257#issuecomment-539220217 Agree with @mik-laj I am also in favor of pointing out Apache projects. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle edited a comment on issue #6100: [AIRFLOW-5387] Fix show paused pagination bug
feluelle edited a comment on issue #6100: [AIRFLOW-5387] Fix show paused pagination bug URL: https://github.com/apache/airflow/pull/6100#issuecomment-539216627 @alrolorojas ```diff import unittest -from parameterized import parameterized from datetime import datetime from unittest import mock from urllib.parse import parse_qs from bs4 import BeautifulSoup +from parameterized import parameterized ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle commented on issue #6100: [AIRFLOW-5387] Fix show paused pagination bug
feluelle commented on issue #6100: [AIRFLOW-5387] Fix show paused pagination bug URL: https://github.com/apache/airflow/pull/6100#issuecomment-539216627 @arlopurcell ```diff import unittest -from parameterized import parameterized from datetime import datetime from unittest import mock from urllib.parse import parse_qs from bs4 import BeautifulSoup +from parameterized import parameterized ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6279: [AIRFLOW-5609][WIP] Add MLEngine models operators
mik-laj commented on a change in pull request #6279: [AIRFLOW-5609][WIP] Add
MLEngine models operators
URL: https://github.com/apache/airflow/pull/6279#discussion_r33224
##
File path: airflow/gcp/hooks/mlengine.py
##
@@ -275,3 +275,19 @@ def get_model(self, project_id: str, model_name: str) ->
Optional[Dict]:
self.log.error('Model was not found: %s', e)
return None
raise
+
+def delete_model(self, project_id: str, model_name: str) -> None:
Review comment:
Here we need a fallback for project id, but I prefer to do it in a separate
PR for whole integration. I prefer to maintain uniformity at the expense of
functionality
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6275: [AIRFLOW-XXX] Fix heading levels
codecov-io commented on issue #6275: [AIRFLOW-XXX] Fix heading levels URL: https://github.com/apache/airflow/pull/6275#issuecomment-539201285 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6275?src=pr=h1) Report > Merging [#6275](https://codecov.io/gh/apache/airflow/pull/6275?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/1347b77970b86503adc1f9b54f766c4dce89bc59?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6275?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6275 +/- ## == - Coverage 80.09% 80.08% -0.01% == Files 612 612 Lines 3532035320 == - Hits2828928286 -3 - Misses 7031 7034 +3 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6275?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6275/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `85% <0%> (-5%)` | :arrow_down: | | [airflow/contrib/operators/ssh\_operator.py](https://codecov.io/gh/apache/airflow/pull/6275/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zc2hfb3BlcmF0b3IucHk=) | `82.5% <0%> (-1.25%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6275/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `56.89% <0%> (+0.34%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6275?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6275?src=pr=footer). Last update [1347b77...c325587](https://codecov.io/gh/apache/airflow/pull/6275?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5609) Add MLEngine model operators
[
https://issues.apache.org/jira/browse/AIRFLOW-5609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946211#comment-16946211
]
ASF GitHub Bot commented on AIRFLOW-5609:
-
mik-laj commented on pull request #6279: [AIRFLOW-5609] Add MLEngine models
operators
URL: https://github.com/apache/airflow/pull/6279
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add MLEngine model operators
>
>
> Key: AIRFLOW-5609
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5609
> Project: Apache Airflow
> Issue Type: Improvement
> Components: gcp
>Affects Versions: 1.10.5
>Reporter: Kamil Bregula
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] mik-laj opened a new pull request #6279: [AIRFLOW-5609] Add MLEngine models operators
mik-laj opened a new pull request #6279: [AIRFLOW-5609] Add MLEngine models
operators
URL: https://github.com/apache/airflow/pull/6279
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6230: [AIRFLOW-5413] Allow K8S pod to be configured from JSON/YAML file
codecov-io edited a comment on issue #6230: [AIRFLOW-5413] Allow K8S pod to be configured from JSON/YAML file URL: https://github.com/apache/airflow/pull/6230#issuecomment-537654940 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6230?src=pr=h1) Report > Merging [#6230](https://codecov.io/gh/apache/airflow/pull/6230?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/76fe45e1d127b657b1aad5c0fd657e011f5a09bc?src=pr=desc) will **decrease** coverage by `0.27%`. > The diff coverage is `89.65%`. [](https://codecov.io/gh/apache/airflow/pull/6230?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6230 +/- ## == - Coverage 80.05% 79.78% -0.28% == Files 610 611 +1 Lines 3526435322 +58 == - Hits2823228182 -50 - Misses 7032 7140 +108 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6230?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==) | `57.33% <100%> (-7.9%)` | :arrow_down: | | [airflow/kubernetes/pod\_generator.py](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9nZW5lcmF0b3IucHk=) | `88.5% <76.27%> (-6.2%)` | :arrow_down: | | [airflow/kubernetes/worker\_configuration.py](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3dvcmtlcl9jb25maWd1cmF0aW9uLnB5) | `95.74% <90%> (-0.66%)` | :arrow_down: | | [airflow/kubernetes/kube\_config.py](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL2t1YmVfY29uZmlnLnB5) | `96.59% <96.59%> (ø)` | | | [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/executors/sequential\_executor.py](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvc2VxdWVudGlhbF9leGVjdXRvci5weQ==) | `47.61% <0%> (-52.39%)` | :arrow_down: | | [airflow/utils/log/colored\_log.py](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvY29sb3JlZF9sb2cucHk=) | `81.81% <0%> (-11.37%)` | :arrow_down: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `84.74% <0%> (-8.48%)` | :arrow_down: | | [airflow/executors/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvX19pbml0X18ucHk=) | `63.26% <0%> (-4.09%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `53.44% <0%> (-3.45%)` | :arrow_down: | | ... and [10 more](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6230?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6230?src=pr=footer). Last update [76fe45e...9d0d3e3](https://codecov.io/gh/apache/airflow/pull/6230?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6261: [AIRFLOW-5141] Added optional resource usage logging to KubernetesPod…
codecov-io edited a comment on issue #6261: [AIRFLOW-5141] Added optional resource usage logging to KubernetesPod… URL: https://github.com/apache/airflow/pull/6261#issuecomment-538590785 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6261?src=pr=h1) Report > Merging [#6261](https://codecov.io/gh/apache/airflow/pull/6261?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/76fe45e1d127b657b1aad5c0fd657e011f5a09bc?src=pr=desc) will **increase** coverage by `0.01%`. > The diff coverage is `84.29%`. [](https://codecov.io/gh/apache/airflow/pull/6261?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6261 +/- ## == + Coverage 80.05% 80.07% +0.01% == Files 610 611 +1 Lines 3526435382 +118 == + Hits2823228331 +99 - Misses 7032 7051 +19 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6261?src=pr=tree) | Coverage Δ | | |---|---|---| | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6261/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `98.57% <100%> (+0.06%)` | :arrow_up: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6261/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `90.47% <76.92%> (-1.5%)` | :arrow_down: | | [airflow/kubernetes/pod\_usage\_metrics\_logger.py](https://codecov.io/gh/apache/airflow/pull/6261/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF91c2FnZV9tZXRyaWNzX2xvZ2dlci5weQ==) | `84.76% <84.76%> (ø)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6261?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6261?src=pr=footer). Last update [76fe45e...75c33ab](https://codecov.io/gh/apache/airflow/pull/6261?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on issue #5944: [AIRFLOW-5362] Reorder imports
kaxil commented on issue #5944: [AIRFLOW-5362] Reorder imports URL: https://github.com/apache/airflow/pull/5944#issuecomment-539183624 @houqp Yes, the main reason was the PR before this (https://github.com/apache/airflow/pull/6096) was merged with "Rebase + Merge" which caused many un-wanted commits (instead of a single commit) and hence we had to squash those commits and force-push. What was your internal pipeline doing??? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #5787: [AIRFLOW-5172] Add choice of interval edge scheduling
codecov-io edited a comment on issue #5787: [AIRFLOW-5172] Add choice of interval edge scheduling URL: https://github.com/apache/airflow/pull/5787#issuecomment-539088404 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5787?src=pr=h1) Report > Merging [#5787](https://codecov.io/gh/apache/airflow/pull/5787?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/1347b77970b86503adc1f9b54f766c4dce89bc59?src=pr=desc) will **decrease** coverage by `0.05%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/5787?src=pr=tree) ```diff @@Coverage Diff @@ ## master#5787 +/- ## == - Coverage 80.09% 80.04% -0.06% == Files 612 612 Lines 3532035336 +16 == - Hits2828928284 -5 - Misses 7031 7052 +21 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5787?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/5787/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `74.39% <100%> (ø)` | :arrow_up: | | [airflow/sensors/time\_delta\_sensor.py](https://codecov.io/gh/apache/airflow/pull/5787/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL3RpbWVfZGVsdGFfc2Vuc29yLnB5) | `100% <100%> (ø)` | :arrow_up: | | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/5787/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `91.89% <100%> (+0.19%)` | :arrow_up: | | [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/5787/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/5787/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/5787/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `86.44% <0%> (-6.78%)` | :arrow_down: | | [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/5787/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `85% <0%> (-5%)` | :arrow_down: | | [airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/airflow/pull/5787/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5) | `75.82% <0%> (-1.79%)` | :arrow_down: | | [airflow/contrib/operators/ssh\_operator.py](https://codecov.io/gh/apache/airflow/pull/5787/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zc2hfb3BlcmF0b3IucHk=) | `82.5% <0%> (-1.25%)` | :arrow_down: | | [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/5787/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `63.88% <0%> (-1.12%)` | :arrow_down: | | ... and [3 more](https://codecov.io/gh/apache/airflow/pull/5787/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5787?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5787?src=pr=footer). Last update [1347b77...e143fe8](https://codecov.io/gh/apache/airflow/pull/5787?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] houqp commented on issue #5944: [AIRFLOW-5362] Reorder imports
houqp commented on issue #5944: [AIRFLOW-5362] Reorder imports URL: https://github.com/apache/airflow/pull/5944#issuecomment-539179841 @potiuk @kaxil looks like commit 8898920 got merged into master, then was force updated to d719e1fd6705a93a0dfefef4b46478ade5e006ea later on. This broke our internal pipeline. Do you know what happened? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6271: [AIRFLOW-5603] Add MLEngine version operators
codecov-io commented on issue #6271: [AIRFLOW-5603] Add MLEngine version operators URL: https://github.com/apache/airflow/pull/6271#issuecomment-539172069 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6271?src=pr=h1) Report > Merging [#6271](https://codecov.io/gh/apache/airflow/pull/6271?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/1347b77970b86503adc1f9b54f766c4dce89bc59?src=pr=desc) will **decrease** coverage by `0.43%`. > The diff coverage is `93.12%`. [](https://codecov.io/gh/apache/airflow/pull/6271?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6271 +/- ## == - Coverage 80.09% 79.66% -0.44% == Files 612 612 Lines 3532035399 +79 == - Hits2828928199 -90 - Misses 7031 7200 +169 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6271?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/gcp/operators/mlengine.py](https://codecov.io/gh/apache/airflow/pull/6271/diff?src=pr=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL21sZW5naW5lLnB5) | `85.48% <100%> (+6.29%)` | :arrow_up: | | [airflow/gcp/example\_dags/example\_mlengine.py](https://codecov.io/gh/apache/airflow/pull/6271/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfbWxlbmdpbmUucHk=) | `83.92% <83.92%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6271/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/executors/sequential\_executor.py](https://codecov.io/gh/apache/airflow/pull/6271/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvc2VxdWVudGlhbF9leGVjdXRvci5weQ==) | `47.61% <0%> (-52.39%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6271/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6271/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/kube\_client.py](https://codecov.io/gh/apache/airflow/pull/6271/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL2t1YmVfY2xpZW50LnB5) | `33.33% <0%> (-41.67%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6271/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `70.14% <0%> (-28.36%)` | :arrow_down: | | [airflow/utils/log/colored\_log.py](https://codecov.io/gh/apache/airflow/pull/6271/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvY29sb3JlZF9sb2cucHk=) | `72.72% <0%> (-20.46%)` | :arrow_down: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6271/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `86.44% <0%> (-6.78%)` | :arrow_down: | | ... and [7 more](https://codecov.io/gh/apache/airflow/pull/6271/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6271?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6271?src=pr=footer). Last update [1347b77...478fc30](https://codecov.io/gh/apache/airflow/pull/6271?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-5609) Add MLEngine model operators
Kamil Bregula created AIRFLOW-5609: -- Summary: Add MLEngine model operators Key: AIRFLOW-5609 URL: https://issues.apache.org/jira/browse/AIRFLOW-5609 Project: Apache Airflow Issue Type: Improvement Components: gcp Affects Versions: 1.10.5 Reporter: Kamil Bregula -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5602) Use unittest.mock in MLEngine hook tests
[ https://issues.apache.org/jira/browse/AIRFLOW-5602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula resolved AIRFLOW-5602. Fix Version/s: 1.10.6 Assignee: Kamil Bregula Resolution: Fixed > Use unittest.mock in MLEngine hook tests > > > Key: AIRFLOW-5602 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5602 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Assignee: Kamil Bregula >Priority: Major > Fix For: 1.10.6 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5602) Use unittest.mock in MLEngine hook tests
[ https://issues.apache.org/jira/browse/AIRFLOW-5602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula updated AIRFLOW-5602: --- Fix Version/s: (was: 1.10.6) 2.0.0 > Use unittest.mock in MLEngine hook tests > > > Key: AIRFLOW-5602 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5602 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Assignee: Kamil Bregula >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5602) Use unittest.mock in MLEngine hook tests
[ https://issues.apache.org/jira/browse/AIRFLOW-5602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946151#comment-16946151 ] ASF subversion and git services commented on AIRFLOW-5602: -- Commit b2dc1636855dcd3ad6957745e5426cd4f5f10c70 in airflow's branch refs/heads/master from Kamil Breguła [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=b2dc163 ] [AIRFLOW-5602] Use unittest.mock in MLEngine hook tests (#6268) > Use unittest.mock in MLEngine hook tests > > > Key: AIRFLOW-5602 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5602 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] mik-laj merged pull request #6268: [AIRFLOW-5602] Use unittest.mock in MLEngine hook tests
mik-laj merged pull request #6268: [AIRFLOW-5602] Use unittest.mock in MLEngine hook tests URL: https://github.com/apache/airflow/pull/6268 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5602) Use unittest.mock in MLEngine hook tests
[ https://issues.apache.org/jira/browse/AIRFLOW-5602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946150#comment-16946150 ] ASF GitHub Bot commented on AIRFLOW-5602: - mik-laj commented on pull request #6268: [AIRFLOW-5602] Use unittest.mock in MLEngine hook tests URL: https://github.com/apache/airflow/pull/6268 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Use unittest.mock in MLEngine hook tests > > > Key: AIRFLOW-5602 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5602 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] kaxil opened a new pull request #6278: [AIRFLOW-XXX] Fix missing backtick in Breeze.rst
kaxil opened a new pull request #6278: [AIRFLOW-XXX] Fix missing backtick in
Breeze.rst
URL: https://github.com/apache/airflow/pull/6278
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
N/a. Doc change
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-2971) Health check command for scheduler
[
https://issues.apache.org/jira/browse/AIRFLOW-2971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946138#comment-16946138
]
ASF GitHub Bot commented on AIRFLOW-2971:
-
colinjc commented on pull request #6277: [AIRFLOW-2971] Add health check CLI
for scheduler
URL: https://github.com/apache/airflow/pull/6277
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
This PR adds the CLI command `airflow health scheduler`, which exits with
status code 1 if the scheduler heartbeat is past the health check threshold.
This duplicates some functionality contained within the webserver health check
endpoint, but can be used in environments where health checks should only have
access to the machine running scheduler itself e.g. Kubernetes health/readiness
checks.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Health check command for scheduler
> --
>
> Key: AIRFLOW-2971
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2971
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
>Reporter: Jon Davies
>Priority: Major
>
> As part of a Kubernetes deployment of Airflow, I would like to define an exec
> command based health check for the Airflow scheduler:
> -
> https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/
> ...the webserver is simple as all that needs is checking that the HTTP port
> is available. For the scheduler, it would be neat to have a command such as:
> airflow scheduler health
> That returned OK and exit 0/NOT OK and a non-zero value when it cannot reach
> the database for instance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] colinjc opened a new pull request #6277: [AIRFLOW-2971] Add health check CLI for scheduler
colinjc opened a new pull request #6277: [AIRFLOW-2971] Add health check CLI
for scheduler
URL: https://github.com/apache/airflow/pull/6277
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
This PR adds the CLI command `airflow health scheduler`, which exits with
status code 1 if the scheduler heartbeat is past the health check threshold.
This duplicates some functionality contained within the webserver health check
endpoint, but can be used in environments where health checks should only have
access to the machine running scheduler itself e.g. Kubernetes health/readiness
checks.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ZxMYS commented on a change in pull request #6265: [AIRFLOW-5597] Linkify urls in task instance log
ZxMYS commented on a change in pull request #6265: [AIRFLOW-5597] Linkify urls
in task instance log
URL: https://github.com/apache/airflow/pull/6265#discussion_r332154727
##
File path: airflow/www/templates/airflow/ti_log.html
##
@@ -121,10 +121,18 @@ {{ title }}
if (res.message) {
// Auto scroll window to the end if current window location is
near the end.
if(auto_tailing && checkAutoTailingCondition()) {
- var should_scroll = true
+ var should_scroll = true;
}
// The message may contain HTML, so either have to escape it or
write it as text.
-document.getElementById(`try-${try_number}`).textContent +=
res.message + "\n";
+var escaped_message = $("").text(res.message).html();
+
+// Detect urls
+var url_regex = /(https?:\/\/[^\s<>"]+)/g;
Review comment:
This regex doesn't work when there's a anchor tag (#). I switched to another
more restrict regex - https://www.regexpal.com/?fam=111808
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lindsable commented on a change in pull request #6075: [AIRFLOW-5266] Allow aws_athena_hook to get all query results
lindsable commented on a change in pull request #6075: [AIRFLOW-5266] Allow aws_athena_hook to get all query results URL: https://github.com/apache/airflow/pull/6075#discussion_r332166708 ## File path: airflow/contrib/hooks/aws_athena_hook.py ## @@ -157,6 +178,7 @@ def poll_query_status(self, query_execution_id, max_tries=None): break if max_tries and try_number >= max_tries: # Break loop if max_tries reached final_query_state = query_state +self.stop_query(query_execution_id) Review comment: Yup! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] lindsable commented on a change in pull request #6075: [AIRFLOW-5266] Allow aws_athena_hook to get all query results
lindsable commented on a change in pull request #6075: [AIRFLOW-5266] Allow aws_athena_hook to get all query results URL: https://github.com/apache/airflow/pull/6075#discussion_r332169164 ## File path: tests/contrib/hooks/test_aws_athena_hook.py ## @@ -0,0 +1,88 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# 'License'); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# 'AS IS' BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +import unittest +from unittest import mock + +try: +from airflow.contrib.hooks.aws_athena_hook import AWSAthenaHook +except ImportError: +AWSAthenaHook = None # type: ignore + + +class MockAthenaClient: Review comment: I had an issue mocking this function because of the way boto3 implements the api. It's not a defined function but an api call. There are a few suggestions that I found [here:](https://stackoverflow.com/questions/37143597/mocking-boto3-s3-client-method-python) but we would still have to maintain it. The other aws hooks that use boto3 are able to use the[moto](https://github.com/spulec/moto) package. but athena isn't mocked yet This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] lindsable commented on a change in pull request #6075: [AIRFLOW-5266] Allow aws_athena_hook to get all query results
lindsable commented on a change in pull request #6075: [AIRFLOW-5266] Allow aws_athena_hook to get all query results URL: https://github.com/apache/airflow/pull/6075#discussion_r332169164 ## File path: tests/contrib/hooks/test_aws_athena_hook.py ## @@ -0,0 +1,88 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# 'License'); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# 'AS IS' BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +import unittest +from unittest import mock + +try: +from airflow.contrib.hooks.aws_athena_hook import AWSAthenaHook +except ImportError: +AWSAthenaHook = None # type: ignore + + +class MockAthenaClient: Review comment: I had an issue mocking this function because of the way boto3 implements the api. It's not a defined function but an api call. There are a few suggestions that I found [here:](https://stackoverflow.com/questions/37143597/mocking-boto3-s3-client-method-python) but we would still have to maintain it. The other aws hooks that use boto3 are able to use the [moto](https://github.com/spulec/moto) package. but athena isn't mocked yet This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] lindsable commented on a change in pull request #6075: [AIRFLOW-5266] Allow aws_athena_hook to get all query results
lindsable commented on a change in pull request #6075: [AIRFLOW-5266] Allow aws_athena_hook to get all query results URL: https://github.com/apache/airflow/pull/6075#discussion_r332169164 ## File path: tests/contrib/hooks/test_aws_athena_hook.py ## @@ -0,0 +1,88 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# 'License'); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# 'AS IS' BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +import unittest +from unittest import mock + +try: +from airflow.contrib.hooks.aws_athena_hook import AWSAthenaHook +except ImportError: +AWSAthenaHook = None # type: ignore + + +class MockAthenaClient: Review comment: I had an issue mocking this function because of the way boto3 implements the api. It's not a defined function but an api call. There are a few suggestions that I found [here:](https://stackoverflow.com/questions/37143597/mocking-boto3-s3-client-method-python)but we would still have to maintain it. The other aws hooks that use boto3 are able to use the[moto](https://github.com/spulec/moto) package. but athena isn't mocked yet This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] lindsable commented on a change in pull request #6075: [AIRFLOW-5266] Allow aws_athena_hook to get all query results
lindsable commented on a change in pull request #6075: [AIRFLOW-5266] Allow aws_athena_hook to get all query results URL: https://github.com/apache/airflow/pull/6075#discussion_r332166708 ## File path: airflow/contrib/hooks/aws_athena_hook.py ## @@ -157,6 +178,7 @@ def poll_query_status(self, query_execution_id, max_tries=None): break if max_tries and try_number >= max_tries: # Break loop if max_tries reached final_query_state = query_state +self.stop_query(query_execution_id) Review comment: Yup! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mgasner closed pull request #6276: [AIRFLOW-3872 Repr fix when attributes are not present on task instance
mgasner closed pull request #6276: [AIRFLOW-3872 Repr fix when attributes are not present on task instance URL: https://github.com/apache/airflow/pull/6276 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3872) repr() is broken for operator instances when task_id is not set
[ https://issues.apache.org/jira/browse/AIRFLOW-3872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946098#comment-16946098 ] ASF GitHub Bot commented on AIRFLOW-3872: - mgasner commented on pull request #6276: [AIRFLOW-3872 Repr fix when attributes are not present on task instance URL: https://github.com/apache/airflow/pull/6276 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > repr() is broken for operator instances when task_id is not set > --- > > Key: AIRFLOW-3872 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3872 > Project: Apache Airflow > Issue Type: Bug > Components: operators, plugins >Affects Versions: 1.10.2 >Reporter: Max Gasner >Assignee: Max Gasner >Priority: Trivial > > If you define a new operator and inspect it during __init__, or at any other > time before task_id is set, repr() raises an AttributeError: "'MyOperator' > object has no attribute 'task_id'. repr() should do something sensible > instead of raising here. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] mgasner opened a new pull request #6276: [AIRFLOW-3872Repr fix
mgasner opened a new pull request #6276: [AIRFLOW-3872Repr fix
URL: https://github.com/apache/airflow/pull/6276
Make sure you have checked _all_ steps below.
### Jira
- [X] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request #6169: [AIRFLOW-4970] Add Google Campaign Manager integration
nuclearpinguin commented on a change in pull request #6169: [AIRFLOW-4970] Add Google Campaign Manager integration URL: https://github.com/apache/airflow/pull/6169#discussion_r332162377 ## File path: tests/provider/google/marketing_platform/hooks/test_campaign_manager.py ## @@ -0,0 +1,280 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +from unittest import TestCase, mock + +from airflow.provider.google.marketing_platform.hooks.campaign_manager import GoogleCampaignManagerHook +from tests.gcp.utils.base_gcp_mock import mock_base_gcp_hook_default_project_id + +API_VERSION = "v3.3" +GCP_CONN_ID = "google_cloud_default" + + +class TestGoogleCampaignManagerHook(TestCase): +def setUp(self): +with mock.patch( +"airflow.gcp.hooks.base.GoogleCloudBaseHook.__init__", +new=mock_base_gcp_hook_default_project_id, +): +self.hook = GoogleCampaignManagerHook( +gcp_conn_id=GCP_CONN_ID, api_version=API_VERSION +) + +@mock.patch( +"airflow.provider.google.marketing_platform.hooks." +"campaign_manager.GoogleCampaignManagerHook._authorize" +) +@mock.patch( +"airflow.provider.google.marketing_platform.hooks." +"campaign_manager.build" +) +def test_gen_conn(self, mock_build, mock_authorize): +result = self.hook.get_conn() +mock_build.assert_called_once_with( +"dfareporting", +API_VERSION, +http=mock_authorize.return_value, +cache_discovery=False, +) +self.assertEqual(mock_build.return_value, result) + +@mock.patch( +"airflow.provider.google.marketing_platform.hooks." +"campaign_manager.GoogleCampaignManagerHook.get_conn" +) +@mock.patch( +"airflow.provider.google.marketing_platform.hooks." +"campaign_manager.GoogleCloudBaseHook.__init__" +) +def test_delete_report(self, mock_base_hook, get_conn_mock): +profile_id = "PROFILE_ID" +report_id = "REPORT_ID" + +return_value = "TEST" + get_conn_mock.return_value.reports.return_value.delete.return_value.execute.return_value = ( +return_value +) + +result = self.hook.delete_report(profile_id=profile_id, report_id=report_id) + + get_conn_mock.return_value.reports.return_value.delete.assert_called_once_with( +profileId=profile_id, reportId=report_id +) + +self.assertEqual(return_value, result) + +@mock.patch( +"airflow.provider.google.marketing_platform.hooks." +"campaign_manager.GoogleCampaignManagerHook.get_conn" +) +@mock.patch( +"airflow.provider.google.marketing_platform.hooks." +"campaign_manager.GoogleCloudBaseHook.__init__" +) +def test_get_report(self, mock_base_hook, get_conn_mock): +profile_id = "PROFILE_ID" +report_id = "REPORT_ID" +file_id = "FILE_ID" + +return_value = "TEST" + get_conn_mock.return_value.reports.return_value.files.return_value.get.\ +return_value.execute.return_value = return_value + +result = self.hook.get_report( +profile_id=profile_id, report_id=report_id, file_id=file_id +) + + get_conn_mock.return_value.reports.return_value.files.return_value.get.assert_called_once_with( +profileId=profile_id, reportId=report_id, fileId=file_id +) + +self.assertEqual(return_value, result) + +@mock.patch( +"airflow.provider.google.marketing_platform.hooks." +"campaign_manager.GoogleCampaignManagerHook.get_conn" +) +@mock.patch( +"airflow.provider.google.marketing_platform.hooks." +"campaign_manager.GoogleCloudBaseHook.__init__" +) +def test_get_report_file(self, mock_base_hook, get_conn_mock): +profile_id = "PROFILE_ID" +report_id = "REPORT_ID" +file_id = "FILE_ID" + +return_value = "TEST" + get_conn_mock.return_value.reports.return_value.files.return_value.get_media.return_value = ( +return_value +) + +result =
[GitHub] [airflow] mik-laj commented on a change in pull request #6169: [AIRFLOW-4970] Add Google Campaign Manager integration
mik-laj commented on a change in pull request #6169: [AIRFLOW-4970] Add Google Campaign Manager integration URL: https://github.com/apache/airflow/pull/6169#discussion_r332155287 ## File path: tests/provider/google/marketing_platform/hooks/test_campaign_manager.py ## @@ -0,0 +1,280 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +from unittest import TestCase, mock + +from airflow.provider.google.marketing_platform.hooks.campaign_manager import GoogleCampaignManagerHook +from tests.gcp.utils.base_gcp_mock import mock_base_gcp_hook_default_project_id + +API_VERSION = "v3.3" +GCP_CONN_ID = "google_cloud_default" + + +class TestGoogleCampaignManagerHook(TestCase): +def setUp(self): +with mock.patch( +"airflow.gcp.hooks.base.GoogleCloudBaseHook.__init__", +new=mock_base_gcp_hook_default_project_id, +): +self.hook = GoogleCampaignManagerHook( +gcp_conn_id=GCP_CONN_ID, api_version=API_VERSION +) + +@mock.patch( +"airflow.provider.google.marketing_platform.hooks." +"campaign_manager.GoogleCampaignManagerHook._authorize" +) +@mock.patch( +"airflow.provider.google.marketing_platform.hooks." +"campaign_manager.build" +) +def test_gen_conn(self, mock_build, mock_authorize): +result = self.hook.get_conn() +mock_build.assert_called_once_with( +"dfareporting", +API_VERSION, +http=mock_authorize.return_value, +cache_discovery=False, +) +self.assertEqual(mock_build.return_value, result) + +@mock.patch( +"airflow.provider.google.marketing_platform.hooks." +"campaign_manager.GoogleCampaignManagerHook.get_conn" +) +@mock.patch( +"airflow.provider.google.marketing_platform.hooks." +"campaign_manager.GoogleCloudBaseHook.__init__" +) +def test_delete_report(self, mock_base_hook, get_conn_mock): +profile_id = "PROFILE_ID" +report_id = "REPORT_ID" + +return_value = "TEST" + get_conn_mock.return_value.reports.return_value.delete.return_value.execute.return_value = ( +return_value +) + +result = self.hook.delete_report(profile_id=profile_id, report_id=report_id) + + get_conn_mock.return_value.reports.return_value.delete.assert_called_once_with( +profileId=profile_id, reportId=report_id +) + +self.assertEqual(return_value, result) + +@mock.patch( +"airflow.provider.google.marketing_platform.hooks." +"campaign_manager.GoogleCampaignManagerHook.get_conn" +) +@mock.patch( +"airflow.provider.google.marketing_platform.hooks." +"campaign_manager.GoogleCloudBaseHook.__init__" +) +def test_get_report(self, mock_base_hook, get_conn_mock): +profile_id = "PROFILE_ID" +report_id = "REPORT_ID" +file_id = "FILE_ID" + +return_value = "TEST" + get_conn_mock.return_value.reports.return_value.files.return_value.get.\ +return_value.execute.return_value = return_value + +result = self.hook.get_report( +profile_id=profile_id, report_id=report_id, file_id=file_id +) + + get_conn_mock.return_value.reports.return_value.files.return_value.get.assert_called_once_with( +profileId=profile_id, reportId=report_id, fileId=file_id +) + +self.assertEqual(return_value, result) + +@mock.patch( +"airflow.provider.google.marketing_platform.hooks." +"campaign_manager.GoogleCampaignManagerHook.get_conn" +) +@mock.patch( +"airflow.provider.google.marketing_platform.hooks." +"campaign_manager.GoogleCloudBaseHook.__init__" +) +def test_get_report_file(self, mock_base_hook, get_conn_mock): +profile_id = "PROFILE_ID" +report_id = "REPORT_ID" +file_id = "FILE_ID" + +return_value = "TEST" + get_conn_mock.return_value.reports.return_value.files.return_value.get_media.return_value = ( +return_value +) + +result = self.hook.get_report_file(
[GitHub] [airflow] ZxMYS commented on a change in pull request #6265: [AIRFLOW-5597] Linkify urls in task instance log
ZxMYS commented on a change in pull request #6265: [AIRFLOW-5597] Linkify urls
in task instance log
URL: https://github.com/apache/airflow/pull/6265#discussion_r332154727
##
File path: airflow/www/templates/airflow/ti_log.html
##
@@ -121,10 +121,18 @@ {{ title }}
if (res.message) {
// Auto scroll window to the end if current window location is
near the end.
if(auto_tailing && checkAutoTailingCondition()) {
- var should_scroll = true
+ var should_scroll = true;
}
// The message may contain HTML, so either have to escape it or
write it as text.
-document.getElementById(`try-${try_number}`).textContent +=
res.message + "\n";
+var escaped_message = $("").text(res.message).html();
+
+// Detect urls
+var url_regex = /(https?:\/\/[^\s<>"]+)/g;
Review comment:
This regex doesn't work when there's a anchor tag (#). I switched to another
more restrict regex - https://www.regexpal.com/?fam=111806
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6169: [AIRFLOW-4970] Add Google Campaign Manager integration
mik-laj commented on a change in pull request #6169: [AIRFLOW-4970] Add Google
Campaign Manager integration
URL: https://github.com/apache/airflow/pull/6169#discussion_r332154256
##
File path: airflow/provider/google/marketing_platform/hooks/campaign_manager.py
##
@@ -0,0 +1,232 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Google Campaign Manager hook.
+"""
+from typing import Any, Dict, List, Optional
+
+from googleapiclient import http
+from googleapiclient.discovery import Resource, build
+
+from airflow.gcp.hooks.base import GoogleCloudBaseHook
+
+
+class GoogleCampaignManagerHook(GoogleCloudBaseHook):
+"""
+Hook for Google Campaign Manager.
+"""
+
+_conn = None # type: Optional[Resource]
+
+def __init__(
+self,
+api_version: str = "v3.3",
+gcp_conn_id: str = "google_cloud_default",
+delegate_to: Optional[str] = None,
+) -> None:
+super().__init__(gcp_conn_id, delegate_to)
+self.api_version = api_version
+
+def get_conn(self) -> Resource:
+"""
+Retrieves connection to Campaign Manager.
+"""
+if not self._conn:
+http_authorized = self._authorize()
+self._conn = build(
+"dfareporting",
+self.api_version,
+http=http_authorized,
+cache_discovery=False,
+)
+return self._conn
+
+def delete_report(self, profile_id: str, report_id: str) -> Any:
+"""
+Deletes a report by its ID.
+
+:param profile_id: The DFA user profile ID.
+:type profile_id: str
+:param report_id: The ID of the report.
+:type report_id: str
+"""
+response = (
+self.get_conn() # pylint: disable=no-member
+.reports()
+.delete(profileId=profile_id, reportId=report_id)
+.execute(num_retries=self.num_retries)
+)
+return response
+
+def insert_report(self, profile_id: str, report: Dict[str, Any]) -> Any:
+"""
+Creates a report.
+
+:param profile_id: The DFA user profile ID.
+:type profile_id: str
+:param report: The report resource to be inserted.
+:type report: Dict[str, Any]
+"""
+response = (
+self.get_conn() # pylint: disable=no-member
+.reports()
+.insert(profileId=profile_id, body=report)
+.execute(num_retries=self.num_retries)
+)
+return response
+
+def list_reports(
+self,
+profile_id: str,
+max_results: Optional[int] = None,
+scope: Optional[str] = None,
+sort_field: Optional[str] = None,
+sort_order: Optional[str] = None,
+) -> List[Dict]:
+"""
+Retrieves list of reports.
+
+:param profile_id: The DFA user profile ID.
+:type profile_id: str
+:param max_results: Maximum number of results to return.
+:type max_results: Optional[int]
+:param scope: The scope that defines which results are returned.
+:type scope: Optional[str]
+:param sort_field: The field by which to sort the list.
+:type sort_field: Optional[str]
+:param sort_order: Order of sorted results.
+:type sort_order: Optional[str]
+"""
+reports = [] # type: List[Dict]
+conn = self.get_conn()
+request = conn.reports().list( # pylint: disable=no-member
+profileId=profile_id,
+maxResults=max_results,
+scope=scope,
+sortField=sort_field,
+sortOrder=sort_order,
+)
+while request is not None:
+response = request.execute(num_retries=self.num_retries)
+reports = response.get("items", [])
Review comment:
```suggestion
reports.extends(response.get("items", []))
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
[GitHub] [airflow] mik-laj opened a new pull request #6275: [AIRFLOW-XXX] Fix heading levels
mik-laj opened a new pull request #6275: [AIRFLOW-XXX] Fix heading levels
URL: https://github.com/apache/airflow/pull/6275
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] alrolorojas commented on issue #6100: [AIRFLOW-5387] Fix show paused pagination bug
alrolorojas commented on issue #6100: [AIRFLOW-5387] Fix show paused pagination bug URL: https://github.com/apache/airflow/pull/6100#issuecomment-539120223 @ashb @feluelle Done! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] iroddis commented on issue #5787: [AIRFLOW-5172] Add choice of interval edge scheduling
iroddis commented on issue #5787: [AIRFLOW-5172] Add choice of interval edge scheduling URL: https://github.com/apache/airflow/pull/5787#issuecomment-539093733 The assumption that the run time for an interval is the dag.following_schedule(dttm) is baked in all over the place. I've found some more cases, and rebased off of current master. Making the change in following_schedule() was not the right route; that method is frequently used to iterate through run times. I instead added a dag.period_end(dttm) method to use that incorporates the new `schedule_at_interval_end` parameter. As of this last checkin, all tests seem to be passing except some that look like race conditions, as they'll pass and not pass from run to run. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #5787: [AIRFLOW-5172] Add choice of interval edge scheduling
codecov-io commented on issue #5787: [AIRFLOW-5172] Add choice of interval edge scheduling URL: https://github.com/apache/airflow/pull/5787#issuecomment-539088404 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5787?src=pr=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@1347b77`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `0%`. [](https://codecov.io/gh/apache/airflow/pull/5787?src=pr=tree) ```diff @@ Coverage Diff@@ ## master #5787 +/- ## Coverage ? 9.49% Files ? 606 Lines ? 35032 Branches ? 0 Hits ?3325 Misses? 31707 Partials ? 0 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5787?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/5787/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `25% <0%> (ø)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5787?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5787?src=pr=footer). Last update [1347b77...e143fe8](https://codecov.io/gh/apache/airflow/pull/5787?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #5787: [AIRFLOW-5172] Add choice of interval edge scheduling
codecov-io edited a comment on issue #5787: [AIRFLOW-5172] Add choice of interval edge scheduling URL: https://github.com/apache/airflow/pull/5787#issuecomment-539088404 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5787?src=pr=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@1347b77`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `0%`. [](https://codecov.io/gh/apache/airflow/pull/5787?src=pr=tree) ```diff @@ Coverage Diff@@ ## master #5787 +/- ## Coverage ? 9.49% Files ? 606 Lines ? 35032 Branches ? 0 Hits ?3325 Misses? 31707 Partials ? 0 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5787?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/5787/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `25% <0%> (ø)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5787?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5787?src=pr=footer). Last update [1347b77...e143fe8](https://codecov.io/gh/apache/airflow/pull/5787?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #6252: [AIRFLOW-4676] Make airflow/macros Pylint compatible
feluelle commented on a change in pull request #6252: [AIRFLOW-4676] Make
airflow/macros Pylint compatible
URL: https://github.com/apache/airflow/pull/6252#discussion_r332104342
##
File path: airflow/macros/__init__.py
##
@@ -91,4 +92,4 @@ def _integrate_plugins():
from airflow.plugins_manager import macros_modules
for macros_module in macros_modules:
sys.modules[macros_module.__name__] = macros_module
-globals()[macros_module._name] = macros_module
+globals()[macros_module.__name__.split('.')[-1]] = macros_module
Review comment:
@BasPH What do you think since you introduced pylint?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #6252: [AIRFLOW-4676] Make airflow/macros Pylint compatible
feluelle commented on a change in pull request #6252: [AIRFLOW-4676] Make
airflow/macros Pylint compatible
URL: https://github.com/apache/airflow/pull/6252#discussion_r332103097
##
File path: airflow/macros/__init__.py
##
@@ -91,4 +92,4 @@ def _integrate_plugins():
from airflow.plugins_manager import macros_modules
for macros_module in macros_modules:
sys.modules[macros_module.__name__] = macros_module
-globals()[macros_module._name] = macros_module
+globals()[macros_module.__name__.split('.')[-1]] = macros_module
Review comment:
Yes, we could do that, but I think `_` does not ever mean it is "private". I
have seen it in some places like in the gcp classes that are accessing those
`_` variables. Because some are using it to mark a variable as "internal" which
then would be totally okay to access from outside of its class.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow-site] mik-laj commented on a change in pull request #64: Add Hugo project
mik-laj commented on a change in pull request #64: Add Hugo project URL: https://github.com/apache/airflow-site/pull/64#discussion_r332098668 ## File path: landing-pages/site/config.toml ## @@ -0,0 +1,150 @@ +baseURL = "/" +title = "Goldydocs" + +enableRobotsTXT = true + +# Hugo allows theme composition (and inheritance). The precedence is from left to right. +theme = ["docsy"] + +# Will give values to .Lastmod etc. +enableGitInfo = true + +# Language settings +contentDir = "content/en" +defaultContentLanguage = "en" +defaultContentLanguageInSubdir = false +# Useful when translating. +enableMissingTranslationPlaceholders = true + +disableKinds = ["taxonomy", "taxonomyTerm"] + +# Highlighting config +pygmentsCodeFences = true +pygmentsUseClasses = false +# Use the new Chroma Go highlighter in Hugo. +pygmentsUseClassic = false +#pygmentsOptions = "linenos=table" +# See https://help.farbox.com/pygments.html +pygmentsStyle = "tango" + +# Configure how URLs look like per section. +[permalinks] +blog = "/:section/:year/:month/:day/:slug/" + +## Configuration for BlackFriday markdown parser: https://github.com/russross/blackfriday +[blackfriday] +plainIDAnchors = true +hrefTargetBlank = true +angledQuotes = false +latexDashes = true + +# Image processing configuration. +[imaging] +resampleFilter = "CatmullRom" +quality = 75 +anchor = "smart" + +[services] +[services.googleAnalytics] +# Comment out the next line to disable GA tracking. Also disables the feature described in [params.ui.feedback]. +id = "UA--0" + +# Language configuration + +[languages] +[languages.en] +title = "Goldydocs" +description = "A Docsy example site" +languageName ="English" +# Weight used for sorting. +weight = 1 +[languages.no] +title = "Goldydocs" +description = "Docsy er operativsystem for skyen" +languageName ="Norsk" +contentDir = "content/no" +time_format_default = "02.01.2006" +time_format_blog = "02.01.2006" + + +# Everything below this are Site Params + +[params] +copyright = "The Docsy Authors" +privacy_policy = "https://policies.google.com/privacy; + +# First one is picked as the Twitter card image if not set on page. +# images = ["images/project-illustration.png"] + +# Menu title if your navbar has a versions selector to access old versions of your site. +# This menu appears only if you have at least one [params.versions] set. +version_menu = "Releases" + +# Repository configuration (URLs for in-page links to opening issues and suggesting changes) +github_repo = "https://github.com/google/docsy-example; +# An optional link to a related project repo. For example, the sibling repository where your product code lives. +github_project_repo = "https://github.com/google/docsy; + +# Specify a value here if your content directory is not in your repo's root directory +# github_subdir = "" + +# Google Custom Search Engine ID. Remove or comment out to disable search. +gcs_engine_id = "011737558837375720776:fsdu1nryfng" Review comment: Can you comment out this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow-site] mik-laj commented on issue #64: Add Hugo project
mik-laj commented on issue #64: Add Hugo project URL: https://github.com/apache/airflow-site/pull/64#issuecomment-539079555 Can you add Apache licence header in each file? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #5992: [AIRFLOW-5088][AIP-24][BackPort] Persisting serialized DAG in DB for webserver scalability
codecov-io edited a comment on issue #5992: [AIRFLOW-5088][AIP-24][BackPort] Persisting serialized DAG in DB for webserver scalability URL: https://github.com/apache/airflow/pull/5992#issuecomment-537908140 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5992?src=pr=h1) Report > Merging [#5992](https://codecov.io/gh/apache/airflow/pull/5992?src=pr=desc) into [v1-10-test](https://codecov.io/gh/apache/airflow/commit/89a88a692f911f39c0ce5d1c7fc54759ed78073c?src=pr=desc) will **increase** coverage by `2.48%`. > The diff coverage is `80.24%`. [](https://codecov.io/gh/apache/airflow/pull/5992?src=pr=tree) ```diff @@ Coverage Diff @@ ## v1-10-test#5992 +/- ## == + Coverage 76.64% 79.13% +2.48% == Files 509 516 +7 Lines 3435534811 +456 == + Hits2633027546+1216 + Misses 8025 7265 -760 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5992?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/5992/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `94.67% <100%> (+0.29%)` | :arrow_up: | | [airflow/settings.py](https://codecov.io/gh/apache/airflow/pull/5992/diff?src=pr=tree#diff-YWlyZmxvdy9zZXR0aW5ncy5weQ==) | `85.78% <100%> (+0.15%)` | :arrow_up: | | [airflow/models/dagrun.py](https://codecov.io/gh/apache/airflow/pull/5992/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFncnVuLnB5) | `96.46% <100%> (ø)` | :arrow_up: | | [airflow/dag/serialization/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/5992/diff?src=pr=tree#diff-YWlyZmxvdy9kYWcvc2VyaWFsaXphdGlvbi9fX2luaXRfXy5weQ==) | `100% <100%> (ø)` | | | [airflow/dag/serialization/enums.py](https://codecov.io/gh/apache/airflow/pull/5992/diff?src=pr=tree#diff-YWlyZmxvdy9kYWcvc2VyaWFsaXphdGlvbi9lbnVtcy5weQ==) | `100% <100%> (ø)` | | | [airflow/www\_rbac/views.py](https://codecov.io/gh/apache/airflow/pull/5992/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==) | `75.29% <100%> (+0.03%)` | :arrow_up: | | [airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/5992/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=) | `100% <100%> (ø)` | :arrow_up: | | [...example\_dags/example\_kubernetes\_executor\_config.py](https://codecov.io/gh/apache/airflow/pull/5992/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX2t1YmVybmV0ZXNfZXhlY3V0b3JfY29uZmlnLnB5) | `77.77% <100%> (+77.77%)` | :arrow_up: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/5992/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `56.56% <16.66%> (-0.75%)` | :arrow_down: | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/5992/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `42.73% <30%> (-2.06%)` | :arrow_down: | | ... and [63 more](https://codecov.io/gh/apache/airflow/pull/5992/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5992?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5992?src=pr=footer). Last update [89a88a6...656deaa](https://codecov.io/gh/apache/airflow/pull/5992?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6265: [AIRFLOW-5597] Linkify urls in task instance log
mik-laj commented on a change in pull request #6265: [AIRFLOW-5597] Linkify
urls in task instance log
URL: https://github.com/apache/airflow/pull/6265#discussion_r332093938
##
File path: airflow/www/templates/airflow/ti_log.html
##
@@ -121,10 +121,18 @@ {{ title }}
if (res.message) {
// Auto scroll window to the end if current window location is
near the end.
if(auto_tailing && checkAutoTailingCondition()) {
- var should_scroll = true
+ var should_scroll = true;
}
// The message may contain HTML, so either have to escape it or
write it as text.
-document.getElementById(`try-${try_number}`).textContent +=
res.message + "\n";
+var escaped_message = $("").text(res.message).html();
Review comment:
Can you use a non-jquery function? We have
[escpeHtml](https://github.com/ZxMYS/incubator-airflow/blob/xzhu/linkify/airflow/www/static/js/base.js#L34-L37)
function
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucacavazzana commented on issue #6090: [AIRFLOW-5470] Add Apache Livy REST operator
lucacavazzana commented on issue #6090: [AIRFLOW-5470] Add Apache Livy REST operator URL: https://github.com/apache/airflow/pull/6090#issuecomment-539074128 thanks for your review. I will look into the suggestions ASAP. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-5600) Add MLEngine system tests
[ https://issues.apache.org/jira/browse/AIRFLOW-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula resolved AIRFLOW-5600. Fix Version/s: 2.0.0 Resolution: Fixed > Add MLEngine system tests > - > > Key: AIRFLOW-5600 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5600 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5600) Add MLEngine system tests
[ https://issues.apache.org/jira/browse/AIRFLOW-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945964#comment-16945964 ] ASF subversion and git services commented on AIRFLOW-5600: -- Commit 1347b77970b86503adc1f9b54f766c4dce89bc59 in airflow's branch refs/heads/master from Kamil Breguła [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=1347b77 ] [AIRFLOW-5600] Add MLEngine system tests (#6264) > Add MLEngine system tests > - > > Key: AIRFLOW-5600 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5600 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5600) Add MLEngine system tests
[ https://issues.apache.org/jira/browse/AIRFLOW-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945963#comment-16945963 ] ASF GitHub Bot commented on AIRFLOW-5600: - mik-laj commented on pull request #6264: [AIRFLOW-5600] Add MLEngine system tests URL: https://github.com/apache/airflow/pull/6264 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add MLEngine system tests > - > > Key: AIRFLOW-5600 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5600 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] mik-laj merged pull request #6264: [AIRFLOW-5600] Add MLEngine system tests
mik-laj merged pull request #6264: [AIRFLOW-5600] Add MLEngine system tests URL: https://github.com/apache/airflow/pull/6264 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #6265: [AIRFLOW-5597] Linkify urls in task instance log
kaxil commented on a change in pull request #6265: [AIRFLOW-5597] Linkify urls
in task instance log
URL: https://github.com/apache/airflow/pull/6265#discussion_r332084839
##
File path: airflow/www/templates/airflow/ti_log.html
##
@@ -121,10 +121,18 @@ {{ title }}
if (res.message) {
// Auto scroll window to the end if current window location is
near the end.
if(auto_tailing && checkAutoTailingCondition()) {
- var should_scroll = true
+ var should_scroll = true;
}
// The message may contain HTML, so either have to escape it or
write it as text.
-document.getElementById(`try-${try_number}`).textContent +=
res.message + "\n";
+var escaped_message = $("").text(res.message).html();
+
+// Detect urls
+var url_regex = /(https?:\/\/[^\s<>"]+)/g;
Review comment:
Will this cover all cases?
How about the following from http://www.noah.org/wiki/RegEx_Python?
```
http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-2971) Health check command for scheduler
[ https://issues.apache.org/jira/browse/AIRFLOW-2971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945954#comment-16945954 ] Colin Campbell commented on AIRFLOW-2971: - I don't think the server health check really resolves this, since it can't be used as a health check for scheduler pods in a k8s environment. This feature would be useful for command based health/liveness checks. https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-liveness-command > Health check command for scheduler > -- > > Key: AIRFLOW-2971 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2971 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Reporter: Jon Davies >Priority: Major > > As part of a Kubernetes deployment of Airflow, I would like to define an exec > command based health check for the Airflow scheduler: > - > https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/ > ...the webserver is simple as all that needs is checking that the HTTP port > is available. For the scheduler, it would be neat to have a command such as: > airflow scheduler health > That returned OK and exit 0/NOT OK and a non-zero value when it cannot reach > the database for instance. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow-site] kgabryje opened a new pull request #64: Add Hugo project
kgabryje opened a new pull request #64: Add Hugo project URL: https://github.com/apache/airflow-site/pull/64 https://github.com/apache/airflow-site/issues/7 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6264: [AIRFLOW-5600] Add MLEngine system tests
codecov-io commented on issue #6264: [AIRFLOW-5600] Add MLEngine system tests URL: https://github.com/apache/airflow/pull/6264#issuecomment-539057970 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6264?src=pr=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@41d6ca5`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `0%`. [](https://codecov.io/gh/apache/airflow/pull/6264?src=pr=tree) ```diff @@ Coverage Diff@@ ## master #6264 +/- ## Coverage ? 9.54% Files ? 611 Lines ? 35320 Branches ? 0 Hits ?3370 Misses? 31950 Partials ? 0 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6264?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/gcp/operators/mlengine.py](https://codecov.io/gh/apache/airflow/pull/6264/diff?src=pr=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL21sZW5naW5lLnB5) | `0% <ø> (ø)` | | | [airflow/gcp/utils/mlengine\_prediction\_summary.py](https://codecov.io/gh/apache/airflow/pull/6264/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvdXRpbHMvbWxlbmdpbmVfcHJlZGljdGlvbl9zdW1tYXJ5LnB5) | `0% <0%> (ø)` | | | [airflow/gcp/example\_dags/example\_mlengine.py](https://codecov.io/gh/apache/airflow/pull/6264/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfbWxlbmdpbmUucHk=) | `0% <0%> (ø)` | | | [airflow/gcp/utils/mlengine\_operator\_utils.py](https://codecov.io/gh/apache/airflow/pull/6264/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvdXRpbHMvbWxlbmdpbmVfb3BlcmF0b3JfdXRpbHMucHk=) | `0% <0%> (ø)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6264?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6264?src=pr=footer). Last update [41d6ca5...6f4f84e](https://codecov.io/gh/apache/airflow/pull/6264?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] Tagar commented on a change in pull request #6090: [AIRFLOW-5470] Add Apache Livy REST operator
Tagar commented on a change in pull request #6090: [AIRFLOW-5470] Add Apache
Livy REST operator
URL: https://github.com/apache/airflow/pull/6090#discussion_r332069650
##
File path: airflow/contrib/hooks/livy_hook.py
##
@@ -0,0 +1,297 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+This module contains the Apache Livy hook.
+"""
+
+import re
+from enum import Enum
+import json
+import requests
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class BatchState(Enum):
+"""
+Batch session states
+"""
+NOT_STARTED = 'not_started'
+STARTING = 'starting'
+RUNNING = 'running'
+IDLE = 'idle'
+BUSY = 'busy'
+SHUTTING_DOWN = 'shutting_down'
+ERROR = 'error'
+DEAD = 'dead'
+KILLED = 'killed'
+SUCCESS = 'success'
+
+
+TERMINAL_STATES = {
+BatchState.SUCCESS,
+BatchState.DEAD,
+BatchState.KILLED,
+BatchState.ERROR,
+}
+
+
+class LivyHook(BaseHook, LoggingMixin):
+"""
+Hook for Apache Livy through the REST API.
+
+For more information about the API refer to
+https://livy.apache.org/docs/latest/rest-api.html
+
+:param livy_conn_id: reference to a pre-defined Livy Connection.
+:type livy_conn_id: str
+"""
+def __init__(self, livy_conn_id='livy_default'):
+super(LivyHook, self).__init__(livy_conn_id)
+self._livy_conn_id = livy_conn_id
+self._build_base_url()
+
+def _build_base_url(self):
+"""
+Build connection URL
+"""
+params = self.get_connection(self._livy_conn_id)
+
+base_url = params.host
+
+if not base_url:
+raise AirflowException("Missing Livy endpoint hostname")
+
+if '://' not in base_url:
+base_url = '{}://{}'.format('http', base_url)
+if not re.search(r':\d+$', base_url):
+base_url = '{}:{}'.format(base_url, str(params.port or 8998))
+
+self._base_url = base_url
+
+def get_conn(self):
+pass
+
+def post_batch(self, *args, **kwargs):
+"""
+Perform request to submit batch
+"""
+
+batch_submit_body = json.dumps(LivyHook.build_post_batch_body(*args,
**kwargs))
+headers = {'Content-Type': 'application/json'}
+
+self.log.info("Submitting job {} to {}".format(batch_submit_body,
self._base_url))
+response = requests.post(self._base_url + '/batches',
data=batch_submit_body, headers=headers)
Review comment:
@ashb I was looking for Kerberos authentication so useername/password
wouldn't do, and overriding `get_conn` in the subclass is the way to do it -
thank you for pointing this out
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6237: AIRFLOW-5581: Join KubernetesJobWatcher in terminate call and unblock queues from blocking forever
codecov-io edited a comment on issue #6237: AIRFLOW-5581: Join KubernetesJobWatcher in terminate call and unblock queues from blocking forever URL: https://github.com/apache/airflow/pull/6237#issuecomment-538409875 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6237?src=pr=h1) Report > Merging [#6237](https://codecov.io/gh/apache/airflow/pull/6237?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/18c62da290ba34e6d09b003e2f52b1a1da3a336e?src=pr=desc) will **decrease** coverage by `0.15%`. > The diff coverage is `6.66%`. [](https://codecov.io/gh/apache/airflow/pull/6237?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6237 +/- ## == - Coverage 80.04% 79.89% -0.16% == Files 610 610 Lines 3526435306 +42 == - Hits2822728207 -20 - Misses 7037 7099 +62 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6237?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/6237/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==) | `58.89% <6.66%> (-6.34%)` | :arrow_down: | | [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6237/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/hooks/postgres\_hook.py](https://codecov.io/gh/apache/airflow/pull/6237/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9wb3N0Z3Jlc19ob29rLnB5) | `94.73% <0%> (-1.76%)` | :arrow_down: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6237/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `91.52% <0%> (-1.7%)` | :arrow_down: | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/6237/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `86.44% <0%> (-1.7%)` | :arrow_down: | | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/6237/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `95.27% <0%> (ø)` | :arrow_up: | | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/6237/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `66.38% <0%> (+0.2%)` | :arrow_up: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6237/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.75% <0%> (+0.5%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6237?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6237?src=pr=footer). Last update [18c62da...55ba865](https://codecov.io/gh/apache/airflow/pull/6237?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-5601) Use built-in pagination mechanism in MLEngine hook
[ https://issues.apache.org/jira/browse/AIRFLOW-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula resolved AIRFLOW-5601. Fix Version/s: 2.0.0 Assignee: Kamil Bregula Resolution: Fixed > Use built-in pagination mechanism in MLEngine hook > -- > > Key: AIRFLOW-5601 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5601 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Assignee: Kamil Bregula >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
