[GitHub] [airflow] nuclearpinguin commented on a change in pull request #7517: [AIRFLOW-6817] Lazy-load `airflow.DAG` to keep user-facing API untouched
nuclearpinguin commented on a change in pull request #7517: [AIRFLOW-6817] Lazy-load `airflow.DAG` to keep user-facing API untouched URL: https://github.com/apache/airflow/pull/7517#discussion_r383119737 ## File path: airflow/__init__.py ## @@ -42,3 +42,22 @@ login: Optional[Callable] = None integrate_plugins() + +__all__ = ['__version__', 'login', 'DAG'] + + +def __getattr__(name): +# PEP-562: Lazy loaded attributes on python modules +if name == "DAG": +from airflow.models.dag import DAG # pylint: disable=redefined-outer-name Review comment: Worth to try but I have a feeling that this has to be disabled in all places where the cycle occurs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request #7520: [AIRFLOW-6896] AzureCosmosDBHook: Move DB call out of __init__
nuclearpinguin commented on a change in pull request #7520: [AIRFLOW-6896]
AzureCosmosDBHook: Move DB call out of __init__
URL: https://github.com/apache/airflow/pull/7520#discussion_r383119015
##
File path: tests/providers/microsoft/azure/hooks/test_azure_cosmos.py
##
@@ -53,6 +53,14 @@ def setUp(self):
)
)
+@mock.patch('azure.cosmos.cosmos_client.CosmosClient')
+def test_client(self):
+import azure.cosmos.cosmos_client as cosmos_client
Review comment:
Why import here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Closed] (AIRFLOW-6825) when build from source and use rbac=True UI is stuck
[ https://issues.apache.org/jira/browse/AIRFLOW-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall closed AIRFLOW-6825. - Resolution: Fixed I am closing this for now. Let me know if you experience further issues. Thanks, Felix > when build from source and use rbac=True UI is stuck > > > Key: AIRFLOW-6825 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6825 > Project: Apache Airflow > Issue Type: Bug > Components: authentication >Affects Versions: 1.10.7 >Reporter: Rajender Reddy Kompally >Priority: Blocker > Attachments: airflow_rbac_issue.png > > > When we build airflow 1.10.7 from source and run using rbac = True option UI > is stuck and an't go to any view tabs. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6825) when build from source and use rbac=True UI is stuck
[ https://issues.apache.org/jira/browse/AIRFLOW-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17043192#comment-17043192 ] Felix Uellendall commented on AIRFLOW-6825: --- Hey [~rreddy3], It looks like you did not build the bundled js / css. If you are building from source you als need to [generate the bundles files with yarn|https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#generate-bundled-files-with-yarn] Start from here: https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#nodejs-environment-setup Best Regards, Felix > when build from source and use rbac=True UI is stuck > > > Key: AIRFLOW-6825 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6825 > Project: Apache Airflow > Issue Type: Bug > Components: authentication >Affects Versions: 1.10.7 >Reporter: Rajender Reddy Kompally >Priority: Blocker > Attachments: airflow_rbac_issue.png > > > When we build airflow 1.10.7 from source and run using rbac = True option UI > is stuck and an't go to any view tabs. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6616) Add Tableau Hook and Operators
[ https://issues.apache.org/jira/browse/AIRFLOW-6616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall resolved AIRFLOW-6616. --- Resolution: Duplicate [~rokm] What do you think? I think this issue is fixed now since AIRFLOW-6790 is merged into master and will be released in 2.0. If you think there is some functionality missing, please open up a new ticket specifying the exact Operator you are missing. Thanks :) Best Regards, Felix > Add Tableau Hook and Operators > -- > > Key: AIRFLOW-6616 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6616 > Project: Apache Airflow > Issue Type: New Feature > Components: hooks, operators >Affects Versions: 2.0.0 >Reporter: Rok Mihevc >Priority: Minor > Fix For: 2.0.0 > > > Tableau (tableau.com) provides a python client > (github.com/tableau/server-client-python) that enables automation of certain > tasks (e.g. report sending). > Addition of Tableau hook and operators will enable using Tableau to generate > and send as steps in a wider ETL process. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk commented on a change in pull request #7517: [AIRFLOW-6817] Lazy-load `airflow.DAG` to keep user-facing API untouched
potiuk commented on a change in pull request #7517: [AIRFLOW-6817] Lazy-load `airflow.DAG` to keep user-facing API untouched URL: https://github.com/apache/airflow/pull/7517#discussion_r383111273 ## File path: airflow/__init__.py ## @@ -42,3 +42,22 @@ login: Optional[Callable] = None integrate_plugins() + +__all__ = ['__version__', 'login', 'DAG'] + + +def __getattr__(name): +# PEP-562: Lazy loaded attributes on python modules +if name == "DAG": +from airflow.models.dag import DAG # pylint: disable=redefined-outer-name Review comment: I think we also need to disable cyclic dependency pylint check as well (cyclic-import). I think disabling it in the first line of the file (just above the licence) will do the trick. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7517: [AIRFLOW-6817] Lazy-load `airflow.DAG` to keep user-facing API untouched
potiuk commented on a change in pull request #7517: [AIRFLOW-6817] Lazy-load `airflow.DAG` to keep user-facing API untouched URL: https://github.com/apache/airflow/pull/7517#discussion_r383111273 ## File path: airflow/__init__.py ## @@ -42,3 +42,22 @@ login: Optional[Callable] = None integrate_plugins() + +__all__ = ['__version__', 'login', 'DAG'] + + +def __getattr__(name): +# PEP-562: Lazy loaded attributes on python modules +if name == "DAG": +from airflow.models.dag import DAG # pylint: disable=redefined-outer-name Review comment: I think we also need to disable cyclic dependency pylint check as well This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7517: [AIRFLOW-6817] Lazy-load `airflow.DAG` to keep user-facing API untouched
potiuk commented on issue #7517: [AIRFLOW-6817] Lazy-load `airflow.DAG` to keep user-facing API untouched URL: https://github.com/apache/airflow/pull/7517#issuecomment-590194706 Fantastic! Works like a charm in both IntelliJ and VSCode! I think we need to just make pytest understand it as well. I will also add later some more protection against importing airflow.DAG in other places than example_dags. Also I have already idea how we can significantly speed up Pylint for local pre-commits (bringing back parallelism without risking cyclic dependencies). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7484: [AIRFLOW-6864][depends on 6863] Make airflow/jobs pylint compatible
codecov-io edited a comment on issue #7484: [AIRFLOW-6864][depends on 6863] Make airflow/jobs pylint compatible URL: https://github.com/apache/airflow/pull/7484#issuecomment-590051665 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7484?src=pr=h1) Report > Merging [#7484](https://codecov.io/gh/apache/airflow/pull/7484?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/20c507f0928efb406a7677d68a6382c100bdcfd3?src=pr=desc) will **decrease** coverage by `0.01%`. > The diff coverage is `90.99%`. [](https://codecov.io/gh/apache/airflow/pull/7484?src=pr=tree) ```diff @@Coverage Diff@@ ## master #7484 +/- ## = - Coverage 86.82% 86.8% -0.02% = Files 893 893 Lines 42188 42325 +137 = + Hits36628 36742 +114 - Misses 55605583 +23 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7484?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/task/task\_runner/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/7484/diff?src=pr=tree#diff-YWlyZmxvdy90YXNrL3Rhc2tfcnVubmVyL19faW5pdF9fLnB5) | `63.63% <ø> (ø)` | :arrow_up: | | [airflow/task/task\_runner/base\_task\_runner.py](https://codecov.io/gh/apache/airflow/pull/7484/diff?src=pr=tree#diff-YWlyZmxvdy90YXNrL3Rhc2tfcnVubmVyL2Jhc2VfdGFza19ydW5uZXIucHk=) | `88.88% <ø> (ø)` | :arrow_up: | | [airflow/cli/commands/scheduler\_command.py](https://codecov.io/gh/apache/airflow/pull/7484/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvc2NoZWR1bGVyX2NvbW1hbmQucHk=) | `0% <0%> (ø)` | :arrow_up: | | [airflow/utils/types.py](https://codecov.io/gh/apache/airflow/pull/7484/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy90eXBlcy5weQ==) | `100% <100%> (ø)` | | | [airflow/cli/commands/task\_command.py](https://codecov.io/gh/apache/airflow/pull/7484/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvdGFza19jb21tYW5kLnB5) | `72.94% <100%> (+0.16%)` | :arrow_up: | | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/7484/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `91.29% <100%> (ø)` | :arrow_up: | | [airflow/ti\_deps/deps/dagrun\_id\_dep.py](https://codecov.io/gh/apache/airflow/pull/7484/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvZGFncnVuX2lkX2RlcC5weQ==) | `100% <100%> (ø)` | :arrow_up: | | [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/7484/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `90.47% <100%> (+0.35%)` | :arrow_up: | | [airflow/models/dagrun.py](https://codecov.io/gh/apache/airflow/pull/7484/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFncnVuLnB5) | `96.55% <100%> (-0.02%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7484/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `92.15% <100%> (+0.04%)` | :arrow_up: | | ... and [17 more](https://codecov.io/gh/apache/airflow/pull/7484/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7484?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7484?src=pr=footer). Last update [20c507f...9360697](https://codecov.io/gh/apache/airflow/pull/7484?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6441) AIP-25 The Knative Executor
[ https://issues.apache.org/jira/browse/AIRFLOW-6441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17043161#comment-17043161 ] ASF GitHub Bot commented on AIRFLOW-6441: - dimberman commented on pull request #7027: [WIP][AIRFLOW-6441] KnativeExecutor URL: https://github.com/apache/airflow/pull/7027 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > AIP-25 The Knative Executor > --- > > Key: AIRFLOW-6441 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6441 > Project: Apache Airflow > Issue Type: New Feature > Components: executors >Affects Versions: 2.0.0 >Reporter: Daniel Imberman >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] dimberman closed pull request #7027: [WIP][AIRFLOW-6441] KnativeExecutor
dimberman closed pull request #7027: [WIP][AIRFLOW-6441] KnativeExecutor URL: https://github.com/apache/airflow/pull/7027 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7521: [AIRFLOW-6897] Simplify DagFileProcessorManager
codecov-io commented on issue #7521: [AIRFLOW-6897] Simplify DagFileProcessorManager URL: https://github.com/apache/airflow/pull/7521#issuecomment-590169752 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7521?src=pr=h1) Report > Merging [#7521](https://codecov.io/gh/apache/airflow/pull/7521?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/0ec2774120d43fa667a371b384e6006e1d1c7821?src=pr=desc) will **decrease** coverage by `0.45%`. > The diff coverage is `96.42%`. [](https://codecov.io/gh/apache/airflow/pull/7521?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7521 +/- ## == - Coverage 86.81% 86.36% -0.46% == Files 893 893 Lines 4219342194 +1 == - Hits3662936439 -190 - Misses 5564 5755 +191 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7521?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7521/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.95% <96.42%> (+0.02%)` | :arrow_up: | | [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7521/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0%> (-64.16%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7521/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7521/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/7521/diff?src=pr=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==) | `30.43% <0%> (-45.66%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7521/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0%> (-45.08%)` | :arrow_down: | | [airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/7521/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5) | `55% <0%> (-45%)` | :arrow_down: | | [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7521/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `69.69% <0%> (-25.26%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7521/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [airflow/providers/apache/hive/hooks/hive.py](https://codecov.io/gh/apache/airflow/pull/7521/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvaG9va3MvaGl2ZS5weQ==) | `76.02% <0%> (-1.54%)` | :arrow_down: | | ... and [2 more](https://codecov.io/gh/apache/airflow/pull/7521/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7521?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7521?src=pr=footer). Last update [0ec2774...d1a1372](https://codecov.io/gh/apache/airflow/pull/7521?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6897) Simplify DagFileProcessorManager
[ https://issues.apache.org/jira/browse/AIRFLOW-6897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17043149#comment-17043149 ] ASF GitHub Bot commented on AIRFLOW-6897: - mik-laj commented on pull request #7521: [AIRFLOW-6897] Simplify DagFileProcessorManager URL: https://github.com/apache/airflow/pull/7521 **Extract start_new_processes and prepare_file_path_queue** These functions are independent of each other, but as separate methods, they are easier to understand. A good methods name is the best documentation. I also changed to `_heartbeat_count` to `_no_run` because it better describes the role. This variable can increase without affecting heartbeat when DagFileProcessorManager is running in asynchronous mode.q **Move emit_metrics call** It is not related to generating file_path_queue, so it should be called by the caller. **Inline heartbeat method** This function does not make sense, because every time a full loop is performed. It only hides the logic and makes it difficult to understand the code. **Move _kill_timed_out_processors to loop** It is not related to generating the collect_results method, so it should be called by the caller. Now I have the impression that the sequence of operations is much more easy to understand because it is only found in the start method. In this PR I do not want to change the sequence of operations. It is only refactoring without functional changes. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [X] Description above provides context of the change - [X] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [X] Unit tests coverage for changes (not needed for documentation changes) - [X] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [X] Relevant documentation is updated including usage instructions. - [X] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Simplify DagFileProcessorManager > > > Key: AIRFLOW-6897 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6897 > Project: Apache Airflow > Issue Type: New Feature > Components: scheduler >Affects Versions: 1.10.9 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] mik-laj opened a new pull request #7521: [AIRFLOW-6897] Simplify DagFileProcessorManager
mik-laj opened a new pull request #7521: [AIRFLOW-6897] Simplify DagFileProcessorManager URL: https://github.com/apache/airflow/pull/7521 **Extract start_new_processes and prepare_file_path_queue** These functions are independent of each other, but as separate methods, they are easier to understand. A good methods name is the best documentation. I also changed to `_heartbeat_count` to `_no_run` because it better describes the role. This variable can increase without affecting heartbeat when DagFileProcessorManager is running in asynchronous mode.q **Move emit_metrics call** It is not related to generating file_path_queue, so it should be called by the caller. **Inline heartbeat method** This function does not make sense, because every time a full loop is performed. It only hides the logic and makes it difficult to understand the code. **Move _kill_timed_out_processors to loop** It is not related to generating the collect_results method, so it should be called by the caller. Now I have the impression that the sequence of operations is much more easy to understand because it is only found in the start method. In this PR I do not want to change the sequence of operations. It is only refactoring without functional changes. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [X] Description above provides context of the change - [X] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [X] Unit tests coverage for changes (not needed for documentation changes) - [X] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [X] Relevant documentation is updated including usage instructions. - [X] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-6897) Simplify DagFileProcessorManager
Kamil Bregula created AIRFLOW-6897: -- Summary: Simplify DagFileProcessorManager Key: AIRFLOW-6897 URL: https://issues.apache.org/jira/browse/AIRFLOW-6897 Project: Apache Airflow Issue Type: New Feature Components: scheduler Affects Versions: 1.10.9 Reporter: Kamil Bregula -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] codecov-io commented on issue #7495: [AIRFLOW-6873] fix mypyd errors and add new type annotations
codecov-io commented on issue #7495: [AIRFLOW-6873] fix mypyd errors and add new type annotations URL: https://github.com/apache/airflow/pull/7495#issuecomment-590156007 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7495?src=pr=h1) Report > Merging [#7495](https://codecov.io/gh/apache/airflow/pull/7495?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/0ec2774120d43fa667a371b384e6006e1d1c7821?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `86.95%`. [](https://codecov.io/gh/apache/airflow/pull/7495?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7495 +/- ## == - Coverage 86.81% 86.52% -0.29% == Files 893 893 Lines 4219342208 +15 == - Hits3662936521 -108 - Misses 5564 5687 +123 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7495?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/settings.py](https://codecov.io/gh/apache/airflow/pull/7495/diff?src=pr=tree#diff-YWlyZmxvdy9zZXR0aW5ncy5weQ==) | `93.66% <100%> (ø)` | :arrow_up: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7495/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `90.24% <100%> (ø)` | :arrow_up: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/7495/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `95.17% <100%> (ø)` | :arrow_up: | | [airflow/utils/log/logging\_mixin.py](https://codecov.io/gh/apache/airflow/pull/7495/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvbG9nZ2luZ19taXhpbi5weQ==) | `95.38% <100%> (ø)` | :arrow_up: | | [airflow/models/dagbag.py](https://codecov.io/gh/apache/airflow/pull/7495/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnYmFnLnB5) | `89.56% <100%> (ø)` | :arrow_up: | | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/7495/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `91.37% <100%> (+0.07%)` | :arrow_up: | | [airflow/utils/helpers.py](https://codecov.io/gh/apache/airflow/pull/7495/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9oZWxwZXJzLnB5) | `82.6% <100%> (+0.1%)` | :arrow_up: | | [airflow/stats.py](https://codecov.io/gh/apache/airflow/pull/7495/diff?src=pr=tree#diff-YWlyZmxvdy9zdGF0cy5weQ==) | `83.56% <66.66%> (-1.74%)` | :arrow_down: | | [airflow/models/dagrun.py](https://codecov.io/gh/apache/airflow/pull/7495/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFncnVuLnB5) | `96.52% <83.33%> (-0.05%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7495/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | ... and [4 more](https://codecov.io/gh/apache/airflow/pull/7495/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7495?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7495?src=pr=footer). Last update [0ec2774...39413fc](https://codecov.io/gh/apache/airflow/pull/7495?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7492: [AIRFLOW-6871] optimize tree view for large DAGs
codecov-io edited a comment on issue #7492: [AIRFLOW-6871] optimize tree view for large DAGs URL: https://github.com/apache/airflow/pull/7492#issuecomment-589921201 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7492?src=pr=h1) Report > Merging [#7492](https://codecov.io/gh/apache/airflow/pull/7492?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/0ec2774120d43fa667a371b384e6006e1d1c7821?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `83.78%`. [](https://codecov.io/gh/apache/airflow/pull/7492?src=pr=tree) ```diff @@Coverage Diff@@ ## master #7492 +/- ## = - Coverage 86.81% 86.8% -0.01% = Files 893 893 Lines 42193 42206 +13 = + Hits36629 36639 +10 - Misses 55645567 +3 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7492?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/7492/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.22% <83.78%> (-0.07%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7492/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `88.12% <0%> (+0.19%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7492?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7492?src=pr=footer). Last update [0ec2774...2697fda](https://codecov.io/gh/apache/airflow/pull/7492?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6896) AzureCosmosDBHook: Move DB call out of __init__
[ https://issues.apache.org/jira/browse/AIRFLOW-6896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17043118#comment-17043118 ] ASF GitHub Bot commented on AIRFLOW-6896: - kaxil commented on pull request #7520: [AIRFLOW-6896] AzureCosmosDBHook: Move DB call out of __init__ URL: https://github.com/apache/airflow/pull/7520 Currently, the Azure client is created in AzureCosmosDBHook.__init__ . This gets connection data from DB. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > AzureCosmosDBHook: Move DB call out of __init__ > --- > > Key: AIRFLOW-6896 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6896 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib >Affects Versions: 2.0.0 >Reporter: Kaxil Naik >Assignee: Kaxil Naik >Priority: Major > Fix For: 2.0.0 > > > Currently, the Azure client is created in AzureCosmosDBHook.__init__ . This > gets connection data from DB. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] kaxil opened a new pull request #7520: [AIRFLOW-6896] AzureCosmosDBHook: Move DB call out of __init__
kaxil opened a new pull request #7520: [AIRFLOW-6896] AzureCosmosDBHook: Move DB call out of __init__ URL: https://github.com/apache/airflow/pull/7520 Currently, the Azure client is created in AzureCosmosDBHook.__init__ . This gets connection data from DB. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-6896) AzureCosmosDBHook: Move DB call out of __init__
Kaxil Naik created AIRFLOW-6896: --- Summary: AzureCosmosDBHook: Move DB call out of __init__ Key: AIRFLOW-6896 URL: https://issues.apache.org/jira/browse/AIRFLOW-6896 Project: Apache Airflow Issue Type: Improvement Components: contrib Affects Versions: 2.0.0 Reporter: Kaxil Naik Assignee: Kaxil Naik Fix For: 2.0.0 Currently, the Azure client is created in AzureCosmosDBHook.__init__ . This gets connection data from DB. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6895) AzureFileShareHook: Move DB call out of __init__
Kaxil Naik created AIRFLOW-6895: --- Summary: AzureFileShareHook: Move DB call out of __init__ Key: AIRFLOW-6895 URL: https://issues.apache.org/jira/browse/AIRFLOW-6895 Project: Apache Airflow Issue Type: Improvement Components: contrib Affects Versions: 2.0.0 Reporter: Kaxil Naik Assignee: Kaxil Naik Fix For: 2.0.0 Currently, the Azure client is created in AzureFileShareHook.__init__ . This gets connection data from DB. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6895) AzureFileShareHook: Move DB call out of __init__
[ https://issues.apache.org/jira/browse/AIRFLOW-6895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17043114#comment-17043114 ] ASF GitHub Bot commented on AIRFLOW-6895: - kaxil commented on pull request #7519: [AIRFLOW-6895] AzureFileShareHook: Move DB call out of __init__ URL: https://github.com/apache/airflow/pull/7519 Currently, the Azure client is created in AzureFileShareHook.__init__ . This gets connection data from DB. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > AzureFileShareHook: Move DB call out of __init__ > > > Key: AIRFLOW-6895 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6895 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib >Affects Versions: 2.0.0 >Reporter: Kaxil Naik >Assignee: Kaxil Naik >Priority: Major > Fix For: 2.0.0 > > > Currently, the Azure client is created in AzureFileShareHook.__init__ . This > gets connection data from DB. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] kaxil opened a new pull request #7519: [AIRFLOW-6895] AzureFileShareHook: Move DB call out of __init__
kaxil opened a new pull request #7519: [AIRFLOW-6895] AzureFileShareHook: Move DB call out of __init__ URL: https://github.com/apache/airflow/pull/7519 Currently, the Azure client is created in AzureFileShareHook.__init__ . This gets connection data from DB. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on issue #7395: [AIRFLOW-5629] Implement K8s priorityClassName in KubernetesPo…
kaxil commented on issue #7395: [AIRFLOW-5629] Implement K8s priorityClassName in KubernetesPo… URL: https://github.com/apache/airflow/pull/7395#issuecomment-590137575 @simis2626 good work @dimberman I think this was safe to merge but if you think otherwise let me know. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-5629) Support specify priority and priorityClassName spec in KubernetesPodOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-5629. - Fix Version/s: 1.10.10 Resolution: Fixed > Support specify priority and priorityClassName spec in KubernetesPodOperator > > > Key: AIRFLOW-5629 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5629 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Affects Versions: 2.0.0 >Reporter: Yaliang Wang >Assignee: Simon Levett >Priority: Major > Labels: features > Fix For: 1.10.10 > > > Since Kubernetes 1.14, Pod can have priority and priority class to indicate > the importance of the > pod([https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/]). > We should support to config the priority and priority class in the > KubernetesPodOperator -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5629) Support specify priority and priorityClassName spec in KubernetesPodOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17043111#comment-17043111 ] ASF subversion and git services commented on AIRFLOW-5629: -- Commit 0ec2774120d43fa667a371b384e6006e1d1c7821 in airflow's branch refs/heads/master from Simon [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=0ec2774 ] [AIRFLOW-5629] Implement Kubernetes priorityClassName in KubernetesPodOperator (#7395) > Support specify priority and priorityClassName spec in KubernetesPodOperator > > > Key: AIRFLOW-5629 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5629 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Affects Versions: 2.0.0 >Reporter: Yaliang Wang >Assignee: Simon Levett >Priority: Major > Labels: features > Fix For: 1.10.10 > > > Since Kubernetes 1.14, Pod can have priority and priority class to indicate > the importance of the > pod([https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/]). > We should support to config the priority and priority class in the > KubernetesPodOperator -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] kaxil merged pull request #7395: [AIRFLOW-5629] Implement K8s priorityClassName in KubernetesPo…
kaxil merged pull request #7395: [AIRFLOW-5629] Implement K8s priorityClassName in KubernetesPo… URL: https://github.com/apache/airflow/pull/7395 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5629) Support specify priority and priorityClassName spec in KubernetesPodOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17043109#comment-17043109 ] ASF GitHub Bot commented on AIRFLOW-5629: - kaxil commented on pull request #7395: [AIRFLOW-5629] Implement K8s priorityClassName in KubernetesPo… URL: https://github.com/apache/airflow/pull/7395 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Support specify priority and priorityClassName spec in KubernetesPodOperator > > > Key: AIRFLOW-5629 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5629 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Affects Versions: 2.0.0 >Reporter: Yaliang Wang >Assignee: Simon Levett >Priority: Major > Labels: features > > Since Kubernetes 1.14, Pod can have priority and priority class to indicate > the importance of the > pod([https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/]). > We should support to config the priority and priority class in the > KubernetesPodOperator -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] boring-cyborg[bot] commented on issue #7395: [AIRFLOW-5629] Implement K8s priorityClassName in KubernetesPo…
boring-cyborg[bot] commented on issue #7395: [AIRFLOW-5629] Implement K8s priorityClassName in KubernetesPo… URL: https://github.com/apache/airflow/pull/7395#issuecomment-590137320 Awesome work, congrats on your first merged pull request! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6516) Airflow Local Setting ConfigMap overrides airflow.cfg VolumeMount
[ https://issues.apache.org/jira/browse/AIRFLOW-6516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17043108#comment-17043108 ] ASF GitHub Bot commented on AIRFLOW-6516: - kaxil commented on pull request #7518: [AIRFLOW-6516] Allow different configmaps for airflow.cfg & airflow_l… URL: https://github.com/apache/airflow/pull/7518 …ocal_settings.py (#7109) https://github.com/apache/airflow/pull/7109 fixed a bug to allow airflow.cfg and airflow_local_settings.py to be specified in a configmap. However if someone wanted to use a different configmap for them, it would have resulted in an error **(Note I had already fixed this in the backport - https://github.com/apache/airflow/commit/adb409fd713d5c80c595d681d6a16f3ed892d5c2 and is already fixed in 1.10.8 and 1.10.9)**. This fix is for Airflow 2.0 and master --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Airflow Local Setting ConfigMap overrides airflow.cfg VolumeMount > - > > Key: AIRFLOW-6516 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6516 > Project: Apache Airflow > Issue Type: Bug > Components: core >Affects Versions: 1.10.8 >Reporter: Kaxil Naik >Assignee: Kaxil Naik >Priority: Major > Fix For: 1.10.8 > > > https://github.com/apache/airflow/commit/4748a518ced4ebd30caec2ac290c782ef023b0d7 > introduced a bug because we were re-assigning the "airflow-config" key in > volume_mounts/ > Because of this airflow.cfg was not mounted and only airflow_local_settings > was mounted if the `airflow_local_settings_configmap` is also set. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] kaxil opened a new pull request #7518: [AIRFLOW-6516] Allow different configmaps for airflow.cfg & airflow_l…
kaxil opened a new pull request #7518: [AIRFLOW-6516] Allow different configmaps for airflow.cfg & airflow_l… URL: https://github.com/apache/airflow/pull/7518 …ocal_settings.py (#7109) https://github.com/apache/airflow/pull/7109 fixed a bug to allow airflow.cfg and airflow_local_settings.py to be specified in a configmap. However if someone wanted to use a different configmap for them, it would have resulted in an error **(Note I had already fixed this in the backport - https://github.com/apache/airflow/commit/adb409fd713d5c80c595d681d6a16f3ed892d5c2 and is already fixed in 1.10.8 and 1.10.9)**. This fix is for Airflow 2.0 and master --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-6894) Prevent db query in example_dags
[ https://issues.apache.org/jira/browse/AIRFLOW-6894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula resolved AIRFLOW-6894. Fix Version/s: 2.0.0 Resolution: Fixed > Prevent db query in example_dags > > > Key: AIRFLOW-6894 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6894 > Project: Apache Airflow > Issue Type: New Feature > Components: examples >Affects Versions: 1.10.9 >Reporter: Kamil Bregula >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6894) Prevent db query in example_dags
[ https://issues.apache.org/jira/browse/AIRFLOW-6894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17043096#comment-17043096 ] ASF subversion and git services commented on AIRFLOW-6894: -- Commit dcf87435219307d4e916a8abc2b819ad75e2b1cf in airflow's branch refs/heads/master from Kamil Breguła [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=dcf8743 ] [AIRFLOW-6894] Prevent db query in example_dags (#7516) > Prevent db query in example_dags > > > Key: AIRFLOW-6894 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6894 > Project: Apache Airflow > Issue Type: New Feature > Components: examples >Affects Versions: 1.10.9 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6894) Prevent db query in example_dags
[ https://issues.apache.org/jira/browse/AIRFLOW-6894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17043095#comment-17043095 ] ASF GitHub Bot commented on AIRFLOW-6894: - mik-laj commented on pull request #7516: [AIRFLOW-6894] Prevent DB query in example_dags URL: https://github.com/apache/airflow/pull/7516 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Prevent db query in example_dags > > > Key: AIRFLOW-6894 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6894 > Project: Apache Airflow > Issue Type: New Feature > Components: examples >Affects Versions: 1.10.9 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] mik-laj merged pull request #7516: [AIRFLOW-6894] Prevent DB query in example_dags
mik-laj merged pull request #7516: [AIRFLOW-6894] Prevent DB query in example_dags URL: https://github.com/apache/airflow/pull/7516 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil edited a comment on issue #7517: [AIRFLOW-6817] Lazy-load `airflow.DAG` to keep user-facing API untouched
kaxil edited a comment on issue #7517: [AIRFLOW-6817] Lazy-load `airflow.DAG`
to keep user-facing API untouched
URL: https://github.com/apache/airflow/pull/7517#issuecomment-590132339
Travis failure:
```
ERROR collecting tests/api/client/test_local_client.py
ImportError while importing test module
'/opt/airflow/tests/api/client/test_local_client.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/api/client/test_local_client.py:26: in
from airflow.example_dags import example_bash_operator
airflow/example_dags/example_bash_operator.py:23: in
from airflow import DAG
E ImportError: cannot import name 'DAG'
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on issue #7517: [AIRFLOW-6817] Lazy-load `airflow.DAG` to keep user-facing API untouched
kaxil commented on issue #7517: [AIRFLOW-6817] Lazy-load `airflow.DAG` to keep
user-facing API untouched
URL: https://github.com/apache/airflow/pull/7517#issuecomment-590132339
```
ERROR collecting tests/api/client/test_local_client.py
ImportError while importing test module
'/opt/airflow/tests/api/client/test_local_client.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/api/client/test_local_client.py:26: in
from airflow.example_dags import example_bash_operator
airflow/example_dags/example_bash_operator.py:23: in
from airflow import DAG
E ImportError: cannot import name 'DAG'
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[airflow-site] branch asf-site updated: Update Meetup details (#253)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/airflow-site.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 70fba4b Update Meetup details (#253)
70fba4b is described below
commit 70fba4b89c505658f460c6afc4a6c2f490c12bfc
Author: Kaxil Naik
AuthorDate: Sun Feb 23 23:51:19 2020 +
Update Meetup details (#253)
---
.idea/.gitignore | 2 ++
meetups.json | 25 -
2 files changed, 14 insertions(+), 13 deletions(-)
diff --git a/.idea/.gitignore b/.idea/.gitignore
new file mode 100644
index 000..e7e9d11
--- /dev/null
+++ b/.idea/.gitignore
@@ -0,0 +1,2 @@
+# Default ignored files
+/workspace.xml
diff --git a/meetups.json b/meetups.json
index 4711ef0..0e8b8b2 100644
--- a/meetups.json
+++ b/meetups.json
@@ -1,80 +1,79 @@
[
{
"city": "Amsterdam",
-"continent": "Europe",
"country": "Netherlands",
"date": "",
-"members": 337,
+"members": 346,
"url": "https://www.meetup.com/Amsterdam-Airflow-meetup/;
},
{
"city": "San Francisco",
"country": "USA",
"date": "",
-"members": 1663,
+"members": 1760,
"url": "https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/;
},
{
"city": "London",
"country": "United Kingdom",
"date": "",
-"members": 648,
+"members": 663,
"url": "https://www.meetup.com/London-Apache-Airflow-Meetup/;
},
{
-"city": "New York City",
+"city": "New York",
"country": "USA",
"date": "",
-"members": 625,
+"members": 717,
"url": "https://www.meetup.com/NYC-Apache-Airflow-Meetup/;
},
{
"city": "Melbourne",
"country": "Australia",
"date": "",
-"members": 125,
+"members": 136,
"url": "https://www.meetup.com/Melbourne-Apache-Airflow-Meetup/;
},
{
"city": "Paris",
"country": "France",
"date": "",
-"members": 550,
+"members": 560,
"url": "https://www.meetup.com/Paris-Apache-Airflow-Meetup/;
},
{
"city": "Portland",
"country": "USA",
"date": "",
-"members": 69,
+"members": 71,
"url": "https://www.meetup.com/Portland-Apache-Airflow-Meetup/;
},
{
"city": "Tokyo",
"country": "Japan",
"date": "",
-"members": 109,
+"members": 115,
"url": "https://www.meetup.com/Tokyo-Apache-Airflow-incubating-Meetup/;
},
{
"city": "Bangalore",
"country": "India",
"date": "",
-"members": 138,
+"members": 538,
"url": "https://www.meetup.com/Bangalore-Apache-Airflow-Meetup/;
},
{
"city": "Seattle",
"country": "USA",
"date": "",
-"members": 164,
+"members": 179,
"url": "https://www.meetup.com/Seattle-Apache-Airflow-Users-Group/;
},
{
"city": "Warsaw",
"country": "Poland",
"date": "",
-"members": 206,
+"members": 265,
"url": "https://www.meetup.com/Warsaw-Airflow-Meetup/;
}
]
[airflow-site] branch master updated: Update Meetup details (#253)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow-site.git
The following commit(s) were added to refs/heads/master by this push:
new 7697217 Update Meetup details (#253)
7697217 is described below
commit 76972175891f3bb71bb88526521df18e2aa04978
Author: Kaxil Naik
AuthorDate: Sun Feb 23 23:48:45 2020 +
Update Meetup details (#253)
---
landing-pages/site/static/meetups.json | 25 -
1 file changed, 12 insertions(+), 13 deletions(-)
diff --git a/landing-pages/site/static/meetups.json
b/landing-pages/site/static/meetups.json
index 4711ef0..0e8b8b2 100644
--- a/landing-pages/site/static/meetups.json
+++ b/landing-pages/site/static/meetups.json
@@ -1,80 +1,79 @@
[
{
"city": "Amsterdam",
-"continent": "Europe",
"country": "Netherlands",
"date": "",
-"members": 337,
+"members": 346,
"url": "https://www.meetup.com/Amsterdam-Airflow-meetup/;
},
{
"city": "San Francisco",
"country": "USA",
"date": "",
-"members": 1663,
+"members": 1760,
"url": "https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/;
},
{
"city": "London",
"country": "United Kingdom",
"date": "",
-"members": 648,
+"members": 663,
"url": "https://www.meetup.com/London-Apache-Airflow-Meetup/;
},
{
-"city": "New York City",
+"city": "New York",
"country": "USA",
"date": "",
-"members": 625,
+"members": 717,
"url": "https://www.meetup.com/NYC-Apache-Airflow-Meetup/;
},
{
"city": "Melbourne",
"country": "Australia",
"date": "",
-"members": 125,
+"members": 136,
"url": "https://www.meetup.com/Melbourne-Apache-Airflow-Meetup/;
},
{
"city": "Paris",
"country": "France",
"date": "",
-"members": 550,
+"members": 560,
"url": "https://www.meetup.com/Paris-Apache-Airflow-Meetup/;
},
{
"city": "Portland",
"country": "USA",
"date": "",
-"members": 69,
+"members": 71,
"url": "https://www.meetup.com/Portland-Apache-Airflow-Meetup/;
},
{
"city": "Tokyo",
"country": "Japan",
"date": "",
-"members": 109,
+"members": 115,
"url": "https://www.meetup.com/Tokyo-Apache-Airflow-incubating-Meetup/;
},
{
"city": "Bangalore",
"country": "India",
"date": "",
-"members": 138,
+"members": 538,
"url": "https://www.meetup.com/Bangalore-Apache-Airflow-Meetup/;
},
{
"city": "Seattle",
"country": "USA",
"date": "",
-"members": 164,
+"members": 179,
"url": "https://www.meetup.com/Seattle-Apache-Airflow-Users-Group/;
},
{
"city": "Warsaw",
"country": "Poland",
"date": "",
-"members": 206,
+"members": 265,
"url": "https://www.meetup.com/Warsaw-Airflow-Meetup/;
}
]
[GitHub] [airflow] codecov-io commented on issue #7516: [AIRFLOW-6894] Prevent DB query in example_dags
codecov-io commented on issue #7516: [AIRFLOW-6894] Prevent DB query in example_dags URL: https://github.com/apache/airflow/pull/7516#issuecomment-590131157 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7516?src=pr=h1) Report > Merging [#7516](https://codecov.io/gh/apache/airflow/pull/7516?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/4e0e2f0da39626c1bc8026a33c2080a6b11f2c41?src=pr=desc) will **decrease** coverage by `0.26%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7516?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7516 +/- ## == - Coverage 86.79% 86.53% -0.27% == Files 887 893 +6 Lines 4197642191 +215 == + Hits3643236508 +76 - Misses 5544 5683 +139 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7516?src=pr=tree) | Coverage Δ | | |---|---|---| | [...flow/providers/google/cloud/operators/cloud\_sql.py](https://codecov.io/gh/apache/airflow/pull/7516/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9jbG91ZF9zcWwucHk=) | `84.68% <100%> (+0.06%)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7516/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7516/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7516/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0%> (-45.08%)` | :arrow_down: | | [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7516/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `69.38% <0%> (-25.52%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7516/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [airflow/utils/file.py](https://codecov.io/gh/apache/airflow/pull/7516/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9maWxlLnB5) | `87.83% <0%> (-1.36%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7516/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.93% <0%> (-0.2%)` | :arrow_down: | | [airflow/api/common/experimental/trigger\_dag.py](https://codecov.io/gh/apache/airflow/pull/7516/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC90cmlnZ2VyX2RhZy5weQ==) | `98.03% <0%> (-0.04%)` | :arrow_down: | | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/7516/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `96.52% <0%> (ø)` | :arrow_up: | | ... and [33 more](https://codecov.io/gh/apache/airflow/pull/7516/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7516?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7516?src=pr=footer). Last update [4e0e2f0...7a3707f](https://codecov.io/gh/apache/airflow/pull/7516?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow-site] kaxil merged pull request #253: Update Meetup details
kaxil merged pull request #253: Update Meetup details URL: https://github.com/apache/airflow-site/pull/253 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[airflow-site] branch update-meetup-details created (now b049e21)
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a change to branch update-meetup-details in repository https://gitbox.apache.org/repos/asf/airflow-site.git. at b049e21 Update Meetup details This branch includes the following new commits: new b049e21 Update Meetup details The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[GitHub] [airflow-site] kaxil opened a new pull request #253: Update Meetup details
kaxil opened a new pull request #253: Update Meetup details URL: https://github.com/apache/airflow-site/pull/253 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[airflow-site] 01/01: Update Meetup details
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch update-meetup-details
in repository https://gitbox.apache.org/repos/asf/airflow-site.git
commit b049e2165c0e8f1c04878fc866e16dd525a4f322
Author: Kaxil Naik
AuthorDate: Sun Feb 23 23:42:47 2020 +
Update Meetup details
---
landing-pages/site/static/meetups.json | 25 -
1 file changed, 12 insertions(+), 13 deletions(-)
diff --git a/landing-pages/site/static/meetups.json
b/landing-pages/site/static/meetups.json
index 4711ef0..0e8b8b2 100644
--- a/landing-pages/site/static/meetups.json
+++ b/landing-pages/site/static/meetups.json
@@ -1,80 +1,79 @@
[
{
"city": "Amsterdam",
-"continent": "Europe",
"country": "Netherlands",
"date": "",
-"members": 337,
+"members": 346,
"url": "https://www.meetup.com/Amsterdam-Airflow-meetup/;
},
{
"city": "San Francisco",
"country": "USA",
"date": "",
-"members": 1663,
+"members": 1760,
"url": "https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/;
},
{
"city": "London",
"country": "United Kingdom",
"date": "",
-"members": 648,
+"members": 663,
"url": "https://www.meetup.com/London-Apache-Airflow-Meetup/;
},
{
-"city": "New York City",
+"city": "New York",
"country": "USA",
"date": "",
-"members": 625,
+"members": 717,
"url": "https://www.meetup.com/NYC-Apache-Airflow-Meetup/;
},
{
"city": "Melbourne",
"country": "Australia",
"date": "",
-"members": 125,
+"members": 136,
"url": "https://www.meetup.com/Melbourne-Apache-Airflow-Meetup/;
},
{
"city": "Paris",
"country": "France",
"date": "",
-"members": 550,
+"members": 560,
"url": "https://www.meetup.com/Paris-Apache-Airflow-Meetup/;
},
{
"city": "Portland",
"country": "USA",
"date": "",
-"members": 69,
+"members": 71,
"url": "https://www.meetup.com/Portland-Apache-Airflow-Meetup/;
},
{
"city": "Tokyo",
"country": "Japan",
"date": "",
-"members": 109,
+"members": 115,
"url": "https://www.meetup.com/Tokyo-Apache-Airflow-incubating-Meetup/;
},
{
"city": "Bangalore",
"country": "India",
"date": "",
-"members": 138,
+"members": 538,
"url": "https://www.meetup.com/Bangalore-Apache-Airflow-Meetup/;
},
{
"city": "Seattle",
"country": "USA",
"date": "",
-"members": 164,
+"members": 179,
"url": "https://www.meetup.com/Seattle-Apache-Airflow-Users-Group/;
},
{
"city": "Warsaw",
"country": "Poland",
"date": "",
-"members": 206,
+"members": 265,
"url": "https://www.meetup.com/Warsaw-Airflow-Meetup/;
}
]
[GitHub] [airflow] houqp edited a comment on issue #7495: [AIRFLOW-6873] fix mypyd errors and add new type annotations
houqp edited a comment on issue #7495: [AIRFLOW-6873] fix mypyd errors and add new type annotations URL: https://github.com/apache/airflow/pull/7495#issuecomment-590129641 @potiuk @nuclearpinguin all feedbacks addressed, ready for another round of review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] houqp commented on issue #7495: [AIRFLOW-6873] fix mypyd errors and add new type annotations
houqp commented on issue #7495: [AIRFLOW-6873] fix mypyd errors and add new type annotations URL: https://github.com/apache/airflow/pull/7495#issuecomment-590129641 @potiuk @nuclearpinguin all feedback addressed, ready for another round of review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] stale[bot] closed pull request #5788: [POC] multi-threading using asyncio
stale[bot] closed pull request #5788: [POC] multi-threading using asyncio URL: https://github.com/apache/airflow/pull/5788 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] houqp commented on issue #7492: [AIRFLOW-6871] optimize tree view for large DAGs
houqp commented on issue #7492: [AIRFLOW-6871] optimize tree view for large DAGs URL: https://github.com/apache/airflow/pull/7492#issuecomment-590128295 @ashb a quick analysis can be found at: https://developpaper.com/json-parse-is-faster-than-object-literal-syntax/. This optimization only makes sense for very large JSON payload, which is the case for tree.html. For the large DAGs that we have, it reduces page load time by couple hundred milliseconds. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb edited a comment on issue #7517: [AIRFLOW-6817] Lazy-load `airflow.DAG` to keep user-facing API untouched
ashb edited a comment on issue #7517: [AIRFLOW-6817] Lazy-load `airflow.DAG` to keep user-facing API untouched URL: https://github.com/apache/airflow/pull/7517#issuecomment-590126692 ~Oh, can remove a line from updating.md now~ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on issue #7517: [AIRFLOW-6817] Lazy-load `airflow.DAG` to keep user-facing API untouched
ashb commented on issue #7517: [AIRFLOW-6817] Lazy-load `airflow.DAG` to keep user-facing API untouched URL: https://github.com/apache/airflow/pull/7517#issuecomment-590126692 Oh, can remove a line from updating.md now This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on issue #7517: [AIRFLOW-6817] Lazy-load `airflow.DAG` to keep user-facing API untouched
ashb commented on issue #7517: [AIRFLOW-6817] Lazy-load `airflow.DAG` to keep user-facing API untouched URL: https://github.com/apache/airflow/pull/7517#issuecomment-590126623 Probably worth some people double checking this in their IDE of choice -- I did install VSCode and PyCharm Community Edition for this, but I am not an IDE user. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6883) Improve import time of airflow module
[ https://issues.apache.org/jira/browse/AIRFLOW-6883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17043082#comment-17043082 ] Ash Berlin-Taylor commented on AIRFLOW-6883: I left this Jira open even though I closed the PR as this is a more general purpose Jira and we can create sub-tasks under it until we are "happy" with this. > Improve import time of airflow module > - > > Key: AIRFLOW-6883 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6883 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 2.0.0 >Reporter: Ash Berlin-Taylor >Assignee: Ash Berlin-Taylor >Priority: Major > > Importing the file_task_handler is importing TaksInstance (which currently > pulls in much of the rest of airflow) -- but it is only used for type > checking, so can be put behind a guard -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6817) Move airflow __init__.py imports to sub-packages
[ https://issues.apache.org/jira/browse/AIRFLOW-6817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17043081#comment-17043081 ] ASF GitHub Bot commented on AIRFLOW-6817: - ashb commented on pull request #7517: [AIRFLOW-6817] Lazy-load `airflow.DAG` to keep user-facing API untouched URL: https://github.com/apache/airflow/pull/7517 `from airflow import DAG` is _such_ a common pattern that it's worth making sure this stays working. Since we are now on Python 3 we can use PEP-562 to have officially supported lazy-loading of attributes on a module. Since the example_dags will be referred to/copied by users I have updated them all back to import from the airflow module, but have left any internal uses untouched. This approach taken from Celery: https://github.com/celery/celery/blob/f2ddd894c32f642a20f03b805b97e460f4fb3b4f/celery/__init__.py#L63-L78 --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Move airflow __init__.py imports to sub-packages > > > Key: AIRFLOW-6817 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6817 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6883) Improve import time of airflow module
[ https://issues.apache.org/jira/browse/AIRFLOW-6883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula updated AIRFLOW-6883: --- Summary: Improve import time of airflow module (was: Improve import time of airflow moudle) > Improve import time of airflow module > - > > Key: AIRFLOW-6883 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6883 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 2.0.0 >Reporter: Ash Berlin-Taylor >Assignee: Ash Berlin-Taylor >Priority: Major > > Importing the file_task_handler is importing TaksInstance (which currently > pulls in much of the rest of airflow) -- but it is only used for type > checking, so can be put behind a guard -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] ashb opened a new pull request #7517: [AIRFLOW-6817] Lazy-load `airflow.DAG` to keep user-facing API untouched
ashb opened a new pull request #7517: [AIRFLOW-6817] Lazy-load `airflow.DAG` to keep user-facing API untouched URL: https://github.com/apache/airflow/pull/7517 `from airflow import DAG` is _such_ a common pattern that it's worth making sure this stays working. Since we are now on Python 3 we can use PEP-562 to have officially supported lazy-loading of attributes on a module. Since the example_dags will be referred to/copied by users I have updated them all back to import from the airflow module, but have left any internal uses untouched. This approach taken from Celery: https://github.com/celery/celery/blob/f2ddd894c32f642a20f03b805b97e460f4fb3b4f/celery/__init__.py#L63-L78 --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #7217: [AIRFLOW-5946] Store source code in db
mik-laj commented on a change in pull request #7217: [AIRFLOW-5946] Store
source code in db
URL: https://github.com/apache/airflow/pull/7217#discussion_r383046137
##
File path: airflow/models/dag.py
##
@@ -1753,6 +1756,44 @@ def get_last_dagrun(self, session=None,
include_externally_triggered=False):
def safe_dag_id(self):
return self.dag_id.replace('.', '__dot__')
+@property
+def code(self):
Review comment:
The property that invokes a database query is very dangerous. This can
result in a mistake that has a big impact on performance.
Here is an example:
https://github.com/apache/airflow/pull/7476
The property that calls the database query caused the scheduler to run 33%
slower. While reviewing the code, we never could see this problem.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-6883) Improve import time of airflow moudle
[ https://issues.apache.org/jira/browse/AIRFLOW-6883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17043079#comment-17043079 ] ASF GitHub Bot commented on AIRFLOW-6883: - ashb commented on pull request #7505: [AIRFLOW-6883] Speed up import of airflow.utils.log.file_task_hanlder URL: https://github.com/apache/airflow/pull/7505 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Improve import time of airflow moudle > - > > Key: AIRFLOW-6883 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6883 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 2.0.0 >Reporter: Ash Berlin-Taylor >Assignee: Ash Berlin-Taylor >Priority: Major > > Importing the file_task_handler is importing TaksInstance (which currently > pulls in much of the rest of airflow) -- but it is only used for type > checking, so can be put behind a guard -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] ashb commented on issue #7505: [AIRFLOW-6883] Speed up import of airflow.utils.log.file_task_hanlder
ashb commented on issue #7505: [AIRFLOW-6883] Speed up import of airflow.utils.log.file_task_hanlder URL: https://github.com/apache/airflow/pull/7505#issuecomment-590125393 Fixed in other ways This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb closed pull request #7505: [AIRFLOW-6883] Speed up import of airflow.utils.log.file_task_hanlder
ashb closed pull request #7505: [AIRFLOW-6883] Speed up import of airflow.utils.log.file_task_hanlder URL: https://github.com/apache/airflow/pull/7505 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] anitakar commented on a change in pull request #7217: [AIRFLOW-5946] Store source code in db
anitakar commented on a change in pull request #7217: [AIRFLOW-5946] Store source code in db URL: https://github.com/apache/airflow/pull/7217#discussion_r383045579 ## File path: airflow/models/__init__.py ## @@ -43,3 +43,19 @@ # Must be loaded after loading DAG model. # noinspection PyUnresolvedReferences import airflow.jobs # noqa: F401 isort # isort:skip + + +def dag_fileloc_hash(full_filepath: str) -> int: Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] anitakar commented on a change in pull request #7217: [AIRFLOW-5946] Store source code in db
anitakar commented on a change in pull request #7217: [AIRFLOW-5946] Store
source code in db
URL: https://github.com/apache/airflow/pull/7217#discussion_r383045556
##
File path: airflow/www/views.py
##
@@ -520,16 +521,60 @@ def last_dagruns(self, session=None):
@has_access
@provide_session
def code(self, session=None):
-dm = models.DagModel
-dag_id = request.args.get('dag_id')
-dag = session.query(dm).filter(dm.dag_id == dag_id).first()
-try:
+
+def get_serialized_dag(dag_id):
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[airflow-site] branch master updated: Add blogpost for 1.10.8 and 1.10.9 (#252)
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/airflow-site.git The following commit(s) were added to refs/heads/master by this push: new 6c989f5 Add blogpost for 1.10.8 and 1.10.9 (#252) 6c989f5 is described below commit 6c989f5c31920f7da1394b64d1189bf2fba1c6f6 Author: Kaxil Naik AuthorDate: Sun Feb 23 22:34:43 2020 + Add blogpost for 1.10.8 and 1.10.9 (#252) --- .../airflow-1.10.8-1.10.9/airflow-dag-tags.png | Bin 0 -> 246407 bytes .../content/en/blog/airflow-1.10.8-1.10.9/index.md | 76 + 2 files changed, 76 insertions(+) diff --git a/landing-pages/site/content/en/blog/airflow-1.10.8-1.10.9/airflow-dag-tags.png b/landing-pages/site/content/en/blog/airflow-1.10.8-1.10.9/airflow-dag-tags.png new file mode 100644 index 000..b838b92 Binary files /dev/null and b/landing-pages/site/content/en/blog/airflow-1.10.8-1.10.9/airflow-dag-tags.png differ diff --git a/landing-pages/site/content/en/blog/airflow-1.10.8-1.10.9/index.md b/landing-pages/site/content/en/blog/airflow-1.10.8-1.10.9/index.md new file mode 100644 index 000..36e12e6 --- /dev/null +++ b/landing-pages/site/content/en/blog/airflow-1.10.8-1.10.9/index.md @@ -0,0 +1,76 @@ +--- +title: "Apache Airflow 1.10.8 & 1.10.9" +linkTitle: "Apache Airflow 1.10.8 & 1.10.9" +author: "Kaxil Naik" +twitter: "kaxil" +github: "kaxil" +linkedin: "kaxil" +description: "We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow." +tags: ["release"] +date: 2020-02-23 +--- + +Airflow 1.10.8 contains 160 commits since 1.10.7 and includes 4 new features, 42 improvements, 36 bug fixes, and several doc changes. + +We released 1.10.9 on the same day as one of the Flask dependencies (Werkzeug) released 1.0 which broke Airflow 1.10.8. + +**Details**: + +* **PyPI**: [https://pypi.org/project/apache-airflow/1.10.9/](https://pypi.org/project/apache-airflow/1.10.9/) +* **Docs**: [https://airflow.apache.org/docs/1.10.9/](https://airflow.apache.org/docs/1.10.9/) +* **Changelog (1.10.8)**: [http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07](http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07) +* **Changelog (1.10.9)**: [http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10](http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10) + +Some of the noteworthy new features (user-facing) are: + +- [Add tags to DAGs and use it for filtering in the UI (RBAC only)](https://github.com/apache/airflow/pull/6489) +- [New Executor: DebugExecutor for Local debugging from your IDE](http://airflow.apache.org/docs/1.10.9/executor/debug.html) +- [Allow passing conf in "Add DAG Run" (Triggered Dags) view](https://github.com/apache/airflow/pull/7281) +- [Allow dags to run for future execution dates for manually triggered DAGs (only if `schedule_interval=None`)](https://github.com/apache/airflow/pull/7038) +- [Dedicated page in documentation for all configs in airflow.cfg](https://airflow.apache.org/docs/1.10.9/configurations-ref.html) + +### Add tags to DAGs and use it for filtering in the UI + +In order to filter DAGs (e.g by team), you can add tags in each dag. The filter is saved in a cookie and can be reset by the reset button. + +For example: + +In your Dag file, pass a list of tags you want to add to DAG object: + +```python +dag = DAG( +dag_id='example_dag_tag', +schedule_interval='0 0 * * *', +tags=['example'] +) +``` + +**Screenshot**: + + +**Note**: This feature is only available for the RBAC UI (enabled using `rbac=True` in `[webserver]` section in your `airflow.cfg`). + + +## Special Note / Deprecations + +### Python 2 +Python 2 has reached end of its life on Jan 2020. Airflow Master no longer supports Python 2. +Airflow 1.10.* would be the last series to support Python 2. + +We strongly recommend users to use Python >= 3.6 + +### Use Airflow RBAC UI +Airflow 1.10.9 ships with 2 UIs, the default is non-RBAC Flask-admin based UI and Flask-appbuilder based UI. + +The Flask-AppBuilder (FAB) based UI is allows Role-based Access Control and has more advanced features compared to +the legacy Flask-admin based UI. This UI can be enabled by setting `rbac=True` in `[webserver]` section in your `airflow.cfg`. + +Flask-admin based UI is deprecated and new features won't be ported to it. This UI will still be the default +for 1.10.* series but would no longer be available from Airflow 2.0 + + +## List of Contributors + +According to git shortlog, the following people contributed to the 1.10.8 and 1.10.9 release. Thank you to all contributors! + +Anita Fronczak, Ash Berlin-Taylor, BasPH, Bharat Kashyap, Bharath Palaksha, Bhavika Tekwani, Bjorn Olsen, Brian Phillips, Cooper Gillan, Daniel Cohen, Daniel Imberman, Daniel Standish, Gabriel
[GitHub] [airflow-site] kaxil merged pull request #252: Add blogpost for 1.10.8 and 1.10.9
kaxil merged pull request #252: Add blogpost for 1.10.8 and 1.10.9 URL: https://github.com/apache/airflow-site/pull/252 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[airflow-site] branch add-release-notes updated (c9101be -> f462ea0)
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a change to branch add-release-notes in repository https://gitbox.apache.org/repos/asf/airflow-site.git. from c9101be Apply review suggestions add f462ea0 Update index.md No new revisions were added by this update. Summary of changes: landing-pages/site/content/en/blog/airflow-1.10.8-1.10.9/index.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
[GitHub] [airflow] kaxil commented on a change in pull request #7489: [AIRFLOW-6869][WIP] Bulk fetch DAGRuns for _process_task_instances
kaxil commented on a change in pull request #7489: [AIRFLOW-6869][WIP] Bulk
fetch DAGRuns for _process_task_instances
URL: https://github.com/apache/airflow/pull/7489#discussion_r383042987
##
File path: airflow/jobs/scheduler_job.py
##
@@ -699,23 +727,30 @@ def _process_dags(self, dagbag, dags, tis_out):
self.log.info("Not processing DAG %s since it's paused",
dag.dag_id)
continue
-self.log.info("Processing %s", dag.dag_id)
+dag_id = dag.dag_id
+self.log.info("Processing %s", dag_id)
+dag_runs_for_dag = dag_runs_by_dag_id[dag_id] if dag_id in
dag_runs_by_dag_id else []
# Only creates DagRun for DAGs that are not subdag since
# DagRun of subdags are created when SubDagOperator executes.
if not dag.is_subdag:
dag_run = self.create_dag_run(dag)
if dag_run:
+dag_runs_for_dag.append(dag_run)
expected_start_date =
dag.following_schedule(dag_run.execution_date)
if expected_start_date:
schedule_delay = dag_run.start_date -
expected_start_date
Stats.timing(
'dagrun.schedule_delay.{dag_id}'.format(dag_id=dag.dag_id),
schedule_delay)
-self.log.info("Created %s", dag_run)
-self._process_task_instances(dag, tis_out)
-if check_slas:
-self.manage_slas(dag)
+self.log.info("Created %s", dag_run)
+
+if dag_runs_for_dag:
+tis_out.extend(self._process_task_instances(dag,
dag_runs_for_dag))
+if check_slas:
+self.manage_slas(dag)
Review comment:
Yes, you are right
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #7484: [AIRFLOW-6864][depends on 6863] Make airflow/jobs pylint compatible
mik-laj commented on a change in pull request #7484: [AIRFLOW-6864][depends on 6863] Make airflow/jobs pylint compatible URL: https://github.com/apache/airflow/pull/7484#discussion_r383042738 ## File path: airflow/utils/types.py ## @@ -20,3 +20,4 @@ class DagRunType: """Class with DagRun types""" BACKFILL_JOB = "backfill_" SCHEDULED = "scheduled__" +MANUAL = "manual__" Review comment: Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7516: [AIRFLOW-6894] Prevent DB query in example_dags
kaxil commented on a change in pull request #7516: [AIRFLOW-6894] Prevent DB query in example_dags URL: https://github.com/apache/airflow/pull/7516#discussion_r383042446 ## File path: tests/test_utils/asserts.py ## @@ -16,9 +16,53 @@ # under the License. import re +from contextlib import contextmanager + +from sqlalchemy import event + +from airflow.settings import engine def assert_equal_ignore_multiple_spaces(case, first, second, msg=None): def _trim(s): return re.sub(r"\s+", " ", s.strip()) return case.assertEqual(_trim(first), _trim(second), msg) + + +class CountQueriesResult: +def __init__(self): +self.count = 0 + + +class CountQueries(object): +""" +Counts the number of queries sent to Airflow Database in a given context. + +Does not support multiple processes. When a new process is started in context, its queries will +not be included. +""" +def __init__(self): +self.result = CountQueriesResult() + +def __enter__(self): +event.listen(engine, "after_cursor_execute", self.after_cursor_execute) +return self.result + +def __exit__(self, type, value, traceback): +event.remove(engine, "after_cursor_execute", self.after_cursor_execute) + +def after_cursor_execute(self, *args, **kwargs): +self.result.count += 1 + + +count_queries = CountQueries + + +@contextmanager +def assert_queries_count(expected_count, message_fmt=None): Review comment: awesome This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #7516: [AIRFLOW-6894] Prevent DB query in example_dags
mik-laj commented on a change in pull request #7516: [AIRFLOW-6894] Prevent DB query in example_dags URL: https://github.com/apache/airflow/pull/7516#discussion_r383040768 ## File path: tests/test_utils/asserts.py ## @@ -16,9 +16,53 @@ # under the License. import re +from contextlib import contextmanager + +from sqlalchemy import event + +from airflow.settings import engine def assert_equal_ignore_multiple_spaces(case, first, second, msg=None): def _trim(s): return re.sub(r"\s+", " ", s.strip()) return case.assertEqual(_trim(first), _trim(second), msg) + + +class CountQueriesResult: +def __init__(self): +self.count = 0 + + +class CountQueries(object): +""" +Counts the number of queries sent to Airflow Database in a given context. + +Does not support multiple processes. When a new process is started in context, its queries will +not be included. +""" +def __init__(self): +self.result = CountQueriesResult() + +def __enter__(self): +event.listen(engine, "after_cursor_execute", self.after_cursor_execute) +return self.result + +def __exit__(self, type, value, traceback): +event.remove(engine, "after_cursor_execute", self.after_cursor_execute) + +def after_cursor_execute(self, *args, **kwargs): +self.result.count += 1 + + +count_queries = CountQueries + + +@contextmanager +def assert_queries_count(expected_count, message_fmt=None): Review comment: I want to use this decorator also to prevent regression in scheduler performance. Some methods are critical and I have optimized it to use very few queries, but it can be easily broken. This context manager will allow us to detect a regression regarding it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow-site] kaxil commented on a change in pull request #252: Add blogpost for 1.10.8 and 1.10.9
kaxil commented on a change in pull request #252: Add blogpost for 1.10.8 and 1.10.9 URL: https://github.com/apache/airflow-site/pull/252#discussion_r383042298 ## File path: landing-pages/site/content/en/blog/airflow-1.10.8-1.10.9.md ## @@ -0,0 +1,77 @@ +--- +title: "Apache Airflow 1.10.8 & 1.10.9" +linkTitle: "Apache Airflow 1.10.8 & 1.10.9" +author: "Kaxil Naik" +twitter: "kaxil" +github: "kaxil" +linkedin: "kaxil" +description: "We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow." +tags: ["release"] +date: 2020-02-23 +--- + +We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow. + +Airflow 1.10.8 contains 160 commits since 1.10.7 and includes 4 new features, 42 improvements, 36 bug fixes, and several doc changes. + +We released 1.10.9 on the same day as one of the Flask dependencies (Werkzeug) released 1.0 which broke Airflow 1.10.8. + +**Details**: +* **PyPI**: [https://pypi.org/project/apache-airflow/1.10.9/](https://pypi.org/project/apache-airflow/1.10.9/) +* **Docs**: [https://airflow.apache.org/docs/1.10.9/](https://airflow.apache.org/docs/1.10.9/) +* **Changelog (1.10.8)**: [http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07](http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07) +* **Changelog (1.10.9)**: [http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10](http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10) + +Some of the noteworthy new features (user-facing) are: + +- [Add tags to DAGs and use it for filtering in the UI](https://github.com/apache/airflow/pull/6489) +- [New Executor: DebugExecutor for Local debugging from your IDE](http://airflow.apache.org/docs/1.10.9/executor/debug.html) +- [Allow passing conf in "Add DAG Run" view](https://github.com/apache/airflow/pull/7281) +- [Allow dags to run for future execution dates](https://github.com/apache/airflow/pull/7038) Review comment: Updated This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[airflow-site] branch add-release-notes updated (2822caa -> c9101be)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a change to branch add-release-notes
in repository https://gitbox.apache.org/repos/asf/airflow-site.git.
from 2822caa Add blogpost for 1.10.8 and 1.10.9
add c9101be Apply review suggestions
No new revisions were added by this update.
Summary of changes:
.../en/blog/airflow-1.10.8-1.10.9/airflow-dag-tags.png | Bin 0 -> 246407 bytes
.../index.md} | 8
2 files changed, 4 insertions(+), 4 deletions(-)
create mode 100644
landing-pages/site/content/en/blog/airflow-1.10.8-1.10.9/airflow-dag-tags.png
rename landing-pages/site/content/en/blog/{airflow-1.10.8-1.10.9.md =>
airflow-1.10.8-1.10.9/index.md} (89%)
[GitHub] [airflow] nuclearpinguin commented on a change in pull request #7484: [AIRFLOW-6864][depends on 6863] Make airflow/jobs pylint compatible
nuclearpinguin commented on a change in pull request #7484:
[AIRFLOW-6864][depends on 6863] Make airflow/jobs pylint compatible
URL: https://github.com/apache/airflow/pull/7484#discussion_r383041742
##
File path: airflow/jobs/backfill_job.py
##
@@ -49,7 +51,7 @@ class BackfillJob(BaseJob):
triggers a set of task instance runs, in the right order and lasts for
as long as it takes for the set of task instance to be completed.
"""
-ID_PREFIX = 'backfill_'
+ID_PREFIX = DagRunType.BACKFILL_JOB
ID_FORMAT_PREFIX = ID_PREFIX + '{0}'
Review comment:
I will leave it for other PR. It's not strictly related to this PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow-site] kaxil commented on a change in pull request #252: Add blogpost for 1.10.8 and 1.10.9
kaxil commented on a change in pull request #252: Add blogpost for 1.10.8 and 1.10.9 URL: https://github.com/apache/airflow-site/pull/252#discussion_r383041721 ## File path: landing-pages/site/content/en/blog/airflow-1.10.8-1.10.9.md ## @@ -0,0 +1,77 @@ +--- +title: "Apache Airflow 1.10.8 & 1.10.9" +linkTitle: "Apache Airflow 1.10.8 & 1.10.9" +author: "Kaxil Naik" +twitter: "kaxil" +github: "kaxil" +linkedin: "kaxil" +description: "We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow." +tags: ["release"] +date: 2020-02-23 +--- + +We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow. + +Airflow 1.10.8 contains 160 commits since 1.10.7 and includes 4 new features, 42 improvements, 36 bug fixes, and several doc changes. + +We released 1.10.9 on the same day as one of the Flask dependencies (Werkzeug) released 1.0 which broke Airflow 1.10.8. + +**Details**: +* **PyPI**: [https://pypi.org/project/apache-airflow/1.10.9/](https://pypi.org/project/apache-airflow/1.10.9/) +* **Docs**: [https://airflow.apache.org/docs/1.10.9/](https://airflow.apache.org/docs/1.10.9/) +* **Changelog (1.10.8)**: [http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07](http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07) +* **Changelog (1.10.9)**: [http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10](http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10) + +Some of the noteworthy new features (user-facing) are: + +- [Add tags to DAGs and use it for filtering in the UI](https://github.com/apache/airflow/pull/6489) +- [New Executor: DebugExecutor for Local debugging from your IDE](http://airflow.apache.org/docs/1.10.9/executor/debug.html) +- [Allow passing conf in "Add DAG Run" view](https://github.com/apache/airflow/pull/7281) Review comment: "Browse" -> "DAG Runs" -> "Add a new Record" (+ plus sign) brings this screen This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow-site] kaxil commented on a change in pull request #252: Add blogpost for 1.10.8 and 1.10.9
kaxil commented on a change in pull request #252: Add blogpost for 1.10.8 and 1.10.9 URL: https://github.com/apache/airflow-site/pull/252#discussion_r383041640 ## File path: landing-pages/site/content/en/blog/airflow-1.10.8-1.10.9.md ## @@ -0,0 +1,77 @@ +--- +title: "Apache Airflow 1.10.8 & 1.10.9" +linkTitle: "Apache Airflow 1.10.8 & 1.10.9" +author: "Kaxil Naik" +twitter: "kaxil" +github: "kaxil" +linkedin: "kaxil" +description: "We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow." +tags: ["release"] +date: 2020-02-23 +--- + +We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow. + +Airflow 1.10.8 contains 160 commits since 1.10.7 and includes 4 new features, 42 improvements, 36 bug fixes, and several doc changes. + +We released 1.10.9 on the same day as one of the Flask dependencies (Werkzeug) released 1.0 which broke Airflow 1.10.8. + +**Details**: +* **PyPI**: [https://pypi.org/project/apache-airflow/1.10.9/](https://pypi.org/project/apache-airflow/1.10.9/) +* **Docs**: [https://airflow.apache.org/docs/1.10.9/](https://airflow.apache.org/docs/1.10.9/) +* **Changelog (1.10.8)**: [http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07](http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07) +* **Changelog (1.10.9)**: [http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10](http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10) + +Some of the noteworthy new features (user-facing) are: + +- [Add tags to DAGs and use it for filtering in the UI](https://github.com/apache/airflow/pull/6489) +- [New Executor: DebugExecutor for Local debugging from your IDE](http://airflow.apache.org/docs/1.10.9/executor/debug.html) +- [Allow passing conf in "Add DAG Run" view](https://github.com/apache/airflow/pull/7281) Review comment:   This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow-site] ashb commented on a change in pull request #252: Add blogpost for 1.10.8 and 1.10.9
ashb commented on a change in pull request #252: Add blogpost for 1.10.8 and 1.10.9 URL: https://github.com/apache/airflow-site/pull/252#discussion_r383041237 ## File path: landing-pages/site/content/en/blog/airflow-1.10.8-1.10.9.md ## @@ -0,0 +1,77 @@ +--- +title: "Apache Airflow 1.10.8 & 1.10.9" +linkTitle: "Apache Airflow 1.10.8 & 1.10.9" +author: "Kaxil Naik" +twitter: "kaxil" +github: "kaxil" +linkedin: "kaxil" +description: "We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow." +tags: ["release"] +date: 2020-02-23 +--- + +We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow. + +Airflow 1.10.8 contains 160 commits since 1.10.7 and includes 4 new features, 42 improvements, 36 bug fixes, and several doc changes. + +We released 1.10.9 on the same day as one of the Flask dependencies (Werkzeug) released 1.0 which broke Airflow 1.10.8. + +**Details**: +* **PyPI**: [https://pypi.org/project/apache-airflow/1.10.9/](https://pypi.org/project/apache-airflow/1.10.9/) +* **Docs**: [https://airflow.apache.org/docs/1.10.9/](https://airflow.apache.org/docs/1.10.9/) +* **Changelog (1.10.8)**: [http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07](http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07) +* **Changelog (1.10.9)**: [http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10](http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10) + +Some of the noteworthy new features (user-facing) are: + +- [Add tags to DAGs and use it for filtering in the UI](https://github.com/apache/airflow/pull/6489) +- [New Executor: DebugExecutor for Local debugging from your IDE](http://airflow.apache.org/docs/1.10.9/executor/debug.html) +- [Allow passing conf in "Add DAG Run" view](https://github.com/apache/airflow/pull/7281) +- [Allow dags to run for future execution dates](https://github.com/apache/airflow/pull/7038) Review comment: We should say that here then This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow-site] ashb commented on a change in pull request #252: Add blogpost for 1.10.8 and 1.10.9
ashb commented on a change in pull request #252: Add blogpost for 1.10.8 and 1.10.9 URL: https://github.com/apache/airflow-site/pull/252#discussion_r383041211 ## File path: landing-pages/site/content/en/blog/airflow-1.10.8-1.10.9.md ## @@ -0,0 +1,77 @@ +--- +title: "Apache Airflow 1.10.8 & 1.10.9" +linkTitle: "Apache Airflow 1.10.8 & 1.10.9" +author: "Kaxil Naik" +twitter: "kaxil" +github: "kaxil" +linkedin: "kaxil" +description: "We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow." +tags: ["release"] +date: 2020-02-23 +--- + +We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow. + +Airflow 1.10.8 contains 160 commits since 1.10.7 and includes 4 new features, 42 improvements, 36 bug fixes, and several doc changes. + +We released 1.10.9 on the same day as one of the Flask dependencies (Werkzeug) released 1.0 which broke Airflow 1.10.8. + +**Details**: +* **PyPI**: [https://pypi.org/project/apache-airflow/1.10.9/](https://pypi.org/project/apache-airflow/1.10.9/) +* **Docs**: [https://airflow.apache.org/docs/1.10.9/](https://airflow.apache.org/docs/1.10.9/) +* **Changelog (1.10.8)**: [http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07](http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07) +* **Changelog (1.10.9)**: [http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10](http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10) + +Some of the noteworthy new features (user-facing) are: + +- [Add tags to DAGs and use it for filtering in the UI](https://github.com/apache/airflow/pull/6489) +- [New Executor: DebugExecutor for Local debugging from your IDE](http://airflow.apache.org/docs/1.10.9/executor/debug.html) +- [Allow passing conf in "Add DAG Run" view](https://github.com/apache/airflow/pull/7281) Review comment: What's the tooltip for the action it's accessible from? Cos that whole screen is new. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request #7484: [AIRFLOW-6864][depends on 6863] Make airflow/jobs pylint compatible
nuclearpinguin commented on a change in pull request #7484:
[AIRFLOW-6864][depends on 6863] Make airflow/jobs pylint compatible
URL: https://github.com/apache/airflow/pull/7484#discussion_r383041194
##
File path: airflow/jobs/backfill_job.py
##
@@ -49,7 +51,7 @@ class BackfillJob(BaseJob):
triggers a set of task instance runs, in the right order and lasts for
as long as it takes for the set of task instance to be completed.
"""
-ID_PREFIX = 'backfill_'
+ID_PREFIX = DagRunType.BACKFILL_JOB
ID_FORMAT_PREFIX = ID_PREFIX + '{0}'
Review comment:
I am happy to do this in other PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #7516: [AIRFLOW-6894] Prevent DB query in example_dags
mik-laj commented on a change in pull request #7516: [AIRFLOW-6894] Prevent DB query in example_dags URL: https://github.com/apache/airflow/pull/7516#discussion_r383040768 ## File path: tests/test_utils/asserts.py ## @@ -16,9 +16,53 @@ # under the License. import re +from contextlib import contextmanager + +from sqlalchemy import event + +from airflow.settings import engine def assert_equal_ignore_multiple_spaces(case, first, second, msg=None): def _trim(s): return re.sub(r"\s+", " ", s.strip()) return case.assertEqual(_trim(first), _trim(second), msg) + + +class CountQueriesResult: +def __init__(self): +self.count = 0 + + +class CountQueries(object): +""" +Counts the number of queries sent to Airflow Database in a given context. + +Does not support multiple processes. When a new process is started in context, its queries will +not be included. +""" +def __init__(self): +self.result = CountQueriesResult() + +def __enter__(self): +event.listen(engine, "after_cursor_execute", self.after_cursor_execute) +return self.result + +def __exit__(self, type, value, traceback): +event.remove(engine, "after_cursor_execute", self.after_cursor_execute) + +def after_cursor_execute(self, *args, **kwargs): +self.result.count += 1 + + +count_queries = CountQueries + + +@contextmanager +def assert_queries_count(expected_count, message_fmt=None): Review comment: This decorator also wants to use it to prevent regression in scheduler performance. Some methods are critical and I have optimized it to use very few queries, but it can be easily broken. This context manager will allow us to detect a regression regarding it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6894) Prevent db query in example_dags
[ https://issues.apache.org/jira/browse/AIRFLOW-6894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17043070#comment-17043070 ] ASF GitHub Bot commented on AIRFLOW-6894: - mik-laj commented on pull request #7516: [AIRFLOW-6894] Prevent db query in example_dags URL: https://github.com/apache/airflow/pull/7516 CC: @kaxil @nuclearpinguin @ANiteckiP Dear SQLAlchemy expert. Do you want to add something? --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [X] Description above provides context of the change - [X] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [X] Unit tests coverage for changes (not needed for documentation changes) - [X] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [X] Relevant documentation is updated including usage instructions. - [X] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Prevent db query in example_dags > > > Key: AIRFLOW-6894 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6894 > Project: Apache Airflow > Issue Type: New Feature > Components: examples >Affects Versions: 1.10.9 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] mik-laj opened a new pull request #7516: [AIRFLOW-6894] Prevent db query in example_dags
mik-laj opened a new pull request #7516: [AIRFLOW-6894] Prevent db query in example_dags URL: https://github.com/apache/airflow/pull/7516 CC: @kaxil @nuclearpinguin @ANiteckiP Dear SQLAlchemy expert. Do you want to add something? --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [X] Description above provides context of the change - [X] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [X] Unit tests coverage for changes (not needed for documentation changes) - [X] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [X] Relevant documentation is updated including usage instructions. - [X] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow-site] potiuk commented on issue #252: Add blogpost for 1.10.8 and 1.10.9
potiuk commented on issue #252: Add blogpost for 1.10.8 and 1.10.9 URL: https://github.com/apache/airflow-site/pull/252#issuecomment-590118672 Love it :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-6894) Prevent db query in example_dags
Kamil Bregula created AIRFLOW-6894: -- Summary: Prevent db query in example_dags Key: AIRFLOW-6894 URL: https://issues.apache.org/jira/browse/AIRFLOW-6894 Project: Apache Airflow Issue Type: New Feature Components: examples Affects Versions: 1.10.9 Reporter: Kamil Bregula -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow-site] kaxil commented on a change in pull request #252: Add blogpost for 1.10.8 and 1.10.9
kaxil commented on a change in pull request #252: Add blogpost for 1.10.8 and 1.10.9 URL: https://github.com/apache/airflow-site/pull/252#discussion_r383039536 ## File path: landing-pages/site/content/en/blog/airflow-1.10.8-1.10.9.md ## @@ -0,0 +1,77 @@ +--- +title: "Apache Airflow 1.10.8 & 1.10.9" +linkTitle: "Apache Airflow 1.10.8 & 1.10.9" +author: "Kaxil Naik" +twitter: "kaxil" +github: "kaxil" +linkedin: "kaxil" +description: "We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow." +tags: ["release"] +date: 2020-02-23 +--- + +We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow. + +Airflow 1.10.8 contains 160 commits since 1.10.7 and includes 4 new features, 42 improvements, 36 bug fixes, and several doc changes. + +We released 1.10.9 on the same day as one of the Flask dependencies (Werkzeug) released 1.0 which broke Airflow 1.10.8. + +**Details**: +* **PyPI**: [https://pypi.org/project/apache-airflow/1.10.9/](https://pypi.org/project/apache-airflow/1.10.9/) +* **Docs**: [https://airflow.apache.org/docs/1.10.9/](https://airflow.apache.org/docs/1.10.9/) +* **Changelog (1.10.8)**: [http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07](http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07) +* **Changelog (1.10.9)**: [http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10](http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10) + +Some of the noteworthy new features (user-facing) are: + +- [Add tags to DAGs and use it for filtering in the UI](https://github.com/apache/airflow/pull/6489) +- [New Executor: DebugExecutor for Local debugging from your IDE](http://airflow.apache.org/docs/1.10.9/executor/debug.html) +- [Allow passing conf in "Add DAG Run" view](https://github.com/apache/airflow/pull/7281) Review comment: On Trigger Dag Run yes but the view is called "Add Dag Run":  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow-site] kaxil commented on a change in pull request #252: Add blogpost for 1.10.8 and 1.10.9
kaxil commented on a change in pull request #252: Add blogpost for 1.10.8 and 1.10.9 URL: https://github.com/apache/airflow-site/pull/252#discussion_r383039358 ## File path: landing-pages/site/content/en/blog/airflow-1.10.8-1.10.9.md ## @@ -0,0 +1,77 @@ +--- +title: "Apache Airflow 1.10.8 & 1.10.9" +linkTitle: "Apache Airflow 1.10.8 & 1.10.9" +author: "Kaxil Naik" +twitter: "kaxil" +github: "kaxil" +linkedin: "kaxil" +description: "We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow." +tags: ["release"] +date: 2020-02-23 +--- + +We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow. + +Airflow 1.10.8 contains 160 commits since 1.10.7 and includes 4 new features, 42 improvements, 36 bug fixes, and several doc changes. + +We released 1.10.9 on the same day as one of the Flask dependencies (Werkzeug) released 1.0 which broke Airflow 1.10.8. + +**Details**: +* **PyPI**: [https://pypi.org/project/apache-airflow/1.10.9/](https://pypi.org/project/apache-airflow/1.10.9/) +* **Docs**: [https://airflow.apache.org/docs/1.10.9/](https://airflow.apache.org/docs/1.10.9/) +* **Changelog (1.10.8)**: [http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07](http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07) +* **Changelog (1.10.9)**: [http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10](http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10) + +Some of the noteworthy new features (user-facing) are: + +- [Add tags to DAGs and use it for filtering in the UI](https://github.com/apache/airflow/pull/6489) +- [New Executor: DebugExecutor for Local debugging from your IDE](http://airflow.apache.org/docs/1.10.9/executor/debug.html) +- [Allow passing conf in "Add DAG Run" view](https://github.com/apache/airflow/pull/7281) +- [Allow dags to run for future execution dates](https://github.com/apache/airflow/pull/7038) +- [Dedicated page in documentation for all configs in airflow.cfg](https://airflow.apache.org/docs/1.10.9/configurations-ref.html) + +### Add tags to DAGs and use it for filtering in the UI + +In order to filter DAGs (e.g by team), you can add tags in each dag. The filter is saved in a cookie and can be reset by the reset button. + +For example: + +In your Dag file, pass a list of tags you want to add to DAG object: + +```python +dag = DAG( +dag_id='example_dag_tag', +schedule_interval='0 0 * * *', +tags=['example'] +) +``` + +**Screenshot**: + Review comment: Agree, will do This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow-site] kaxil commented on a change in pull request #252: Add blogpost for 1.10.8 and 1.10.9
kaxil commented on a change in pull request #252: Add blogpost for 1.10.8 and 1.10.9 URL: https://github.com/apache/airflow-site/pull/252#discussion_r383039358 ## File path: landing-pages/site/content/en/blog/airflow-1.10.8-1.10.9.md ## @@ -0,0 +1,77 @@ +--- +title: "Apache Airflow 1.10.8 & 1.10.9" +linkTitle: "Apache Airflow 1.10.8 & 1.10.9" +author: "Kaxil Naik" +twitter: "kaxil" +github: "kaxil" +linkedin: "kaxil" +description: "We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow." +tags: ["release"] +date: 2020-02-23 +--- + +We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow. + +Airflow 1.10.8 contains 160 commits since 1.10.7 and includes 4 new features, 42 improvements, 36 bug fixes, and several doc changes. + +We released 1.10.9 on the same day as one of the Flask dependencies (Werkzeug) released 1.0 which broke Airflow 1.10.8. + +**Details**: +* **PyPI**: [https://pypi.org/project/apache-airflow/1.10.9/](https://pypi.org/project/apache-airflow/1.10.9/) +* **Docs**: [https://airflow.apache.org/docs/1.10.9/](https://airflow.apache.org/docs/1.10.9/) +* **Changelog (1.10.8)**: [http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07](http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07) +* **Changelog (1.10.9)**: [http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10](http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10) + +Some of the noteworthy new features (user-facing) are: + +- [Add tags to DAGs and use it for filtering in the UI](https://github.com/apache/airflow/pull/6489) +- [New Executor: DebugExecutor for Local debugging from your IDE](http://airflow.apache.org/docs/1.10.9/executor/debug.html) +- [Allow passing conf in "Add DAG Run" view](https://github.com/apache/airflow/pull/7281) +- [Allow dags to run for future execution dates](https://github.com/apache/airflow/pull/7038) +- [Dedicated page in documentation for all configs in airflow.cfg](https://airflow.apache.org/docs/1.10.9/configurations-ref.html) + +### Add tags to DAGs and use it for filtering in the UI + +In order to filter DAGs (e.g by team), you can add tags in each dag. The filter is saved in a cookie and can be reset by the reset button. + +For example: + +In your Dag file, pass a list of tags you want to add to DAG object: + +```python +dag = DAG( +dag_id='example_dag_tag', +schedule_interval='0 0 * * *', +tags=['example'] +) +``` + +**Screenshot**: + Review comment: Will do This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow-site] kaxil commented on a change in pull request #252: Add blogpost for 1.10.8 and 1.10.9
kaxil commented on a change in pull request #252: Add blogpost for 1.10.8 and 1.10.9 URL: https://github.com/apache/airflow-site/pull/252#discussion_r383039351 ## File path: landing-pages/site/content/en/blog/airflow-1.10.8-1.10.9.md ## @@ -0,0 +1,77 @@ +--- +title: "Apache Airflow 1.10.8 & 1.10.9" +linkTitle: "Apache Airflow 1.10.8 & 1.10.9" +author: "Kaxil Naik" +twitter: "kaxil" +github: "kaxil" +linkedin: "kaxil" +description: "We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow." +tags: ["release"] +date: 2020-02-23 +--- + +We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow. + +Airflow 1.10.8 contains 160 commits since 1.10.7 and includes 4 new features, 42 improvements, 36 bug fixes, and several doc changes. + +We released 1.10.9 on the same day as one of the Flask dependencies (Werkzeug) released 1.0 which broke Airflow 1.10.8. + +**Details**: +* **PyPI**: [https://pypi.org/project/apache-airflow/1.10.9/](https://pypi.org/project/apache-airflow/1.10.9/) +* **Docs**: [https://airflow.apache.org/docs/1.10.9/](https://airflow.apache.org/docs/1.10.9/) +* **Changelog (1.10.8)**: [http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07](http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07) +* **Changelog (1.10.9)**: [http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10](http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10) + +Some of the noteworthy new features (user-facing) are: + +- [Add tags to DAGs and use it for filtering in the UI](https://github.com/apache/airflow/pull/6489) +- [New Executor: DebugExecutor for Local debugging from your IDE](http://airflow.apache.org/docs/1.10.9/executor/debug.html) +- [Allow passing conf in "Add DAG Run" view](https://github.com/apache/airflow/pull/7281) +- [Allow dags to run for future execution dates](https://github.com/apache/airflow/pull/7038) Review comment: Yes This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow-site] ashb commented on a change in pull request #252: Add blogpost for 1.10.8 and 1.10.9
ashb commented on a change in pull request #252: Add blogpost for 1.10.8 and 1.10.9 URL: https://github.com/apache/airflow-site/pull/252#discussion_r383039214 ## File path: landing-pages/site/content/en/blog/airflow-1.10.8-1.10.9.md ## @@ -0,0 +1,77 @@ +--- +title: "Apache Airflow 1.10.8 & 1.10.9" +linkTitle: "Apache Airflow 1.10.8 & 1.10.9" +author: "Kaxil Naik" +twitter: "kaxil" +github: "kaxil" +linkedin: "kaxil" +description: "We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow." +tags: ["release"] +date: 2020-02-23 +--- + +We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow. + +Airflow 1.10.8 contains 160 commits since 1.10.7 and includes 4 new features, 42 improvements, 36 bug fixes, and several doc changes. + +We released 1.10.9 on the same day as one of the Flask dependencies (Werkzeug) released 1.0 which broke Airflow 1.10.8. + +**Details**: +* **PyPI**: [https://pypi.org/project/apache-airflow/1.10.9/](https://pypi.org/project/apache-airflow/1.10.9/) +* **Docs**: [https://airflow.apache.org/docs/1.10.9/](https://airflow.apache.org/docs/1.10.9/) +* **Changelog (1.10.8)**: [http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07](http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07) +* **Changelog (1.10.9)**: [http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10](http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10) + +Some of the noteworthy new features (user-facing) are: + +- [Add tags to DAGs and use it for filtering in the UI](https://github.com/apache/airflow/pull/6489) +- [New Executor: DebugExecutor for Local debugging from your IDE](http://airflow.apache.org/docs/1.10.9/executor/debug.html) +- [Allow passing conf in "Add DAG Run" view](https://github.com/apache/airflow/pull/7281) +- [Allow dags to run for future execution dates](https://github.com/apache/airflow/pull/7038) +- [Dedicated page in documentation for all configs in airflow.cfg](https://airflow.apache.org/docs/1.10.9/configurations-ref.html) + +### Add tags to DAGs and use it for filtering in the UI + +In order to filter DAGs (e.g by team), you can add tags in each dag. The filter is saved in a cookie and can be reset by the reset button. + +For example: + +In your Dag file, pass a list of tags you want to add to DAG object: + +```python +dag = DAG( +dag_id='example_dag_tag', +schedule_interval='0 0 * * *', +tags=['example'] +) +``` + +**Screenshot**: + Review comment: We should probably pull this screen shot in to the site, and not rely on that image staying around on Github. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow-site] ashb commented on a change in pull request #252: Add blogpost for 1.10.8 and 1.10.9
ashb commented on a change in pull request #252: Add blogpost for 1.10.8 and 1.10.9 URL: https://github.com/apache/airflow-site/pull/252#discussion_r383039156 ## File path: landing-pages/site/content/en/blog/airflow-1.10.8-1.10.9.md ## @@ -0,0 +1,77 @@ +--- +title: "Apache Airflow 1.10.8 & 1.10.9" +linkTitle: "Apache Airflow 1.10.8 & 1.10.9" +author: "Kaxil Naik" +twitter: "kaxil" +github: "kaxil" +linkedin: "kaxil" +description: "We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow." +tags: ["release"] +date: 2020-02-23 +--- + +We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow. + +Airflow 1.10.8 contains 160 commits since 1.10.7 and includes 4 new features, 42 improvements, 36 bug fixes, and several doc changes. + +We released 1.10.9 on the same day as one of the Flask dependencies (Werkzeug) released 1.0 which broke Airflow 1.10.8. + +**Details**: +* **PyPI**: [https://pypi.org/project/apache-airflow/1.10.9/](https://pypi.org/project/apache-airflow/1.10.9/) +* **Docs**: [https://airflow.apache.org/docs/1.10.9/](https://airflow.apache.org/docs/1.10.9/) +* **Changelog (1.10.8)**: [http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07](http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07) +* **Changelog (1.10.9)**: [http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10](http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10) + +Some of the noteworthy new features (user-facing) are: + +- [Add tags to DAGs and use it for filtering in the UI](https://github.com/apache/airflow/pull/6489) +- [New Executor: DebugExecutor for Local debugging from your IDE](http://airflow.apache.org/docs/1.10.9/executor/debug.html) +- [Allow passing conf in "Add DAG Run" view](https://github.com/apache/airflow/pull/7281) +- [Allow dags to run for future execution dates](https://github.com/apache/airflow/pull/7038) Review comment: This is specifically only when schedule_interval=None isn't it? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow-site] ashb commented on a change in pull request #252: Add blogpost for 1.10.8 and 1.10.9
ashb commented on a change in pull request #252: Add blogpost for 1.10.8 and 1.10.9 URL: https://github.com/apache/airflow-site/pull/252#discussion_r383039126 ## File path: landing-pages/site/content/en/blog/airflow-1.10.8-1.10.9.md ## @@ -0,0 +1,77 @@ +--- +title: "Apache Airflow 1.10.8 & 1.10.9" +linkTitle: "Apache Airflow 1.10.8 & 1.10.9" +author: "Kaxil Naik" +twitter: "kaxil" +github: "kaxil" +linkedin: "kaxil" +description: "We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow." +tags: ["release"] +date: 2020-02-23 +--- + +We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow. + +Airflow 1.10.8 contains 160 commits since 1.10.7 and includes 4 new features, 42 improvements, 36 bug fixes, and several doc changes. + +We released 1.10.9 on the same day as one of the Flask dependencies (Werkzeug) released 1.0 which broke Airflow 1.10.8. + +**Details**: +* **PyPI**: [https://pypi.org/project/apache-airflow/1.10.9/](https://pypi.org/project/apache-airflow/1.10.9/) +* **Docs**: [https://airflow.apache.org/docs/1.10.9/](https://airflow.apache.org/docs/1.10.9/) +* **Changelog (1.10.8)**: [http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07](http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07) +* **Changelog (1.10.9)**: [http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10](http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10) + +Some of the noteworthy new features (user-facing) are: + +- [Add tags to DAGs and use it for filtering in the UI](https://github.com/apache/airflow/pull/6489) +- [New Executor: DebugExecutor for Local debugging from your IDE](http://airflow.apache.org/docs/1.10.9/executor/debug.html) +- [Allow passing conf in "Add DAG Run" view](https://github.com/apache/airflow/pull/7281) Review comment: Is this Add DAG run or is it Trigger dag run? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow-site] ashb commented on a change in pull request #252: Add blogpost for 1.10.8 and 1.10.9
ashb commented on a change in pull request #252: Add blogpost for 1.10.8 and 1.10.9 URL: https://github.com/apache/airflow-site/pull/252#discussion_r383039090 ## File path: landing-pages/site/content/en/blog/airflow-1.10.8-1.10.9.md ## @@ -0,0 +1,77 @@ +--- +title: "Apache Airflow 1.10.8 & 1.10.9" +linkTitle: "Apache Airflow 1.10.8 & 1.10.9" +author: "Kaxil Naik" +twitter: "kaxil" +github: "kaxil" +linkedin: "kaxil" +description: "We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow." +tags: ["release"] +date: 2020-02-23 +--- + +We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow. + +Airflow 1.10.8 contains 160 commits since 1.10.7 and includes 4 new features, 42 improvements, 36 bug fixes, and several doc changes. + +We released 1.10.9 on the same day as one of the Flask dependencies (Werkzeug) released 1.0 which broke Airflow 1.10.8. + +**Details**: +* **PyPI**: [https://pypi.org/project/apache-airflow/1.10.9/](https://pypi.org/project/apache-airflow/1.10.9/) +* **Docs**: [https://airflow.apache.org/docs/1.10.9/](https://airflow.apache.org/docs/1.10.9/) +* **Changelog (1.10.8)**: [http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07](http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07) +* **Changelog (1.10.9)**: [http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10](http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10) + +Some of the noteworthy new features (user-facing) are: + +- [Add tags to DAGs and use it for filtering in the UI](https://github.com/apache/airflow/pull/6489) Review comment: ```suggestion - [Add tags to DAGs and use it for filtering in the UI (RBAC only)](https://github.com/apache/airflow/pull/6489) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow-site] kaxil commented on a change in pull request #252: Add blogpost for 1.10.8 and 1.10.9
kaxil commented on a change in pull request #252: Add blogpost for 1.10.8 and 1.10.9 URL: https://github.com/apache/airflow-site/pull/252#discussion_r383039016 ## File path: landing-pages/site/content/en/blog/airflow-1.10.8-1.10.9.md ## @@ -0,0 +1,77 @@ +--- +title: "Apache Airflow 1.10.8 & 1.10.9" +linkTitle: "Apache Airflow 1.10.8 & 1.10.9" +author: "Kaxil Naik" +twitter: "kaxil" +github: "kaxil" +linkedin: "kaxil" +description: "We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow." +tags: ["release"] +date: 2020-02-23 Review comment: We can change the date to the date of the release ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow-site] kaxil opened a new pull request #252: Add blogpost for 1.10.8 and 1.10.9
kaxil opened a new pull request #252: Add blogpost for 1.10.8 and 1.10.9 URL: https://github.com/apache/airflow-site/pull/252 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7428: [AIRFLOW-4526] Poll k8s logs to avoid problem with kubernetes logs --follow
codecov-io edited a comment on issue #7428: [AIRFLOW-4526] Poll k8s logs to avoid problem with kubernetes logs --follow URL: https://github.com/apache/airflow/pull/7428#issuecomment-590111637 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7428?src=pr=h1) Report > Merging [#7428](https://codecov.io/gh/apache/airflow/pull/7428?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/a812f48052d0839ea4e9d6ddc6b6e6be78608ce7?src=pr=desc) will **decrease** coverage by `0.16%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7428?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7428 +/- ## == - Coverage 86.81% 86.64% -0.17% == Files 893 893 Lines 4219042226 +36 == - Hits3662636587 -39 - Misses 5564 5639 +75 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7428?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `94.38% <100%> (+2.12%)` | :arrow_up: | | [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0%> (-64.16%)` | :arrow_down: | | [airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==) | `30.43% <0%> (-45.66%)` | :arrow_down: | | [airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5) | `55% <0%> (-45%)` | :arrow_down: | | [airflow/providers/apache/hive/hooks/hive.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvaG9va3MvaGl2ZS5weQ==) | `76.02% <0%> (-1.54%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `90.98% <0%> (-1.13%)` | :arrow_down: | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `90.9% <0%> (-0.83%)` | :arrow_down: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `90.09% <0%> (-0.15%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7428?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7428?src=pr=footer). Last update [a812f48...aef6ac4](https://codecov.io/gh/apache/airflow/pull/7428?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7428: [AIRFLOW-4526] Poll k8s logs to avoid problem with kubernetes logs --follow
codecov-io edited a comment on issue #7428: [AIRFLOW-4526] Poll k8s logs to avoid problem with kubernetes logs --follow URL: https://github.com/apache/airflow/pull/7428#issuecomment-590111637 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7428?src=pr=h1) Report > Merging [#7428](https://codecov.io/gh/apache/airflow/pull/7428?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/a812f48052d0839ea4e9d6ddc6b6e6be78608ce7?src=pr=desc) will **decrease** coverage by `0.16%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7428?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7428 +/- ## == - Coverage 86.81% 86.64% -0.17% == Files 893 893 Lines 4219042226 +36 == - Hits3662636587 -39 - Misses 5564 5639 +75 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7428?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `94.38% <100%> (+2.12%)` | :arrow_up: | | [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0%> (-64.16%)` | :arrow_down: | | [airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==) | `30.43% <0%> (-45.66%)` | :arrow_down: | | [airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5) | `55% <0%> (-45%)` | :arrow_down: | | [airflow/providers/apache/hive/hooks/hive.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvaG9va3MvaGl2ZS5weQ==) | `76.02% <0%> (-1.54%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `90.98% <0%> (-1.13%)` | :arrow_down: | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `90.9% <0%> (-0.83%)` | :arrow_down: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `90.09% <0%> (-0.15%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7428?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7428?src=pr=footer). Last update [a812f48...aef6ac4](https://codecov.io/gh/apache/airflow/pull/7428?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7428: [AIRFLOW-4526] Poll k8s logs to avoid problem with kubernetes logs --follow
codecov-io edited a comment on issue #7428: [AIRFLOW-4526] Poll k8s logs to avoid problem with kubernetes logs --follow URL: https://github.com/apache/airflow/pull/7428#issuecomment-590111637 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7428?src=pr=h1) Report > Merging [#7428](https://codecov.io/gh/apache/airflow/pull/7428?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/a812f48052d0839ea4e9d6ddc6b6e6be78608ce7?src=pr=desc) will **decrease** coverage by `0.16%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7428?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7428 +/- ## == - Coverage 86.81% 86.64% -0.17% == Files 893 893 Lines 4219042226 +36 == - Hits3662636587 -39 - Misses 5564 5639 +75 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7428?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `94.38% <100%> (+2.12%)` | :arrow_up: | | [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0%> (-64.16%)` | :arrow_down: | | [airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==) | `30.43% <0%> (-45.66%)` | :arrow_down: | | [airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5) | `55% <0%> (-45%)` | :arrow_down: | | [airflow/providers/apache/hive/hooks/hive.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvaG9va3MvaGl2ZS5weQ==) | `76.02% <0%> (-1.54%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `90.98% <0%> (-1.13%)` | :arrow_down: | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `90.9% <0%> (-0.83%)` | :arrow_down: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `90.09% <0%> (-0.15%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7428?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7428?src=pr=footer). Last update [a812f48...aef6ac4](https://codecov.io/gh/apache/airflow/pull/7428?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[airflow-site] 01/01: Add blogpost for 1.10.8 and 1.10.9
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a commit to branch add-release-notes in repository https://gitbox.apache.org/repos/asf/airflow-site.git commit 2822caab1a07e57cc217752ab79a5409eb9e8e45 Author: Kaxil Naik AuthorDate: Sun Feb 23 20:44:59 2020 + Add blogpost for 1.10.8 and 1.10.9 --- .../site/content/en/blog/airflow-1.10.8-1.10.9.md | 77 ++ 1 file changed, 77 insertions(+) diff --git a/landing-pages/site/content/en/blog/airflow-1.10.8-1.10.9.md b/landing-pages/site/content/en/blog/airflow-1.10.8-1.10.9.md new file mode 100644 index 000..210ad00 --- /dev/null +++ b/landing-pages/site/content/en/blog/airflow-1.10.8-1.10.9.md @@ -0,0 +1,77 @@ +--- +title: "Apache Airflow 1.10.8 & 1.10.9" +linkTitle: "Apache Airflow 1.10.8 & 1.10.9" +author: "Kaxil Naik" +twitter: "kaxil" +github: "kaxil" +linkedin: "kaxil" +description: "We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow." +tags: ["release"] +date: 2020-02-23 +--- + +We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow. + +Airflow 1.10.8 contains 160 commits since 1.10.7 and includes 4 new features, 42 improvements, 36 bug fixes, and several doc changes. + +We released 1.10.9 on the same day as one of the Flask dependencies (Werkzeug) released 1.0 which broke Airflow 1.10.8. + +**Details**: +* **PyPI**: [https://pypi.org/project/apache-airflow/1.10.9/](https://pypi.org/project/apache-airflow/1.10.9/) +* **Docs**: [https://airflow.apache.org/docs/1.10.9/](https://airflow.apache.org/docs/1.10.9/) +* **Changelog (1.10.8)**: [http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07](http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07) +* **Changelog (1.10.9)**: [http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10](http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10) + +Some of the noteworthy new features (user-facing) are: + +- [Add tags to DAGs and use it for filtering in the UI](https://github.com/apache/airflow/pull/6489) +- [New Executor: DebugExecutor for Local debugging from your IDE](http://airflow.apache.org/docs/1.10.9/executor/debug.html) +- [Allow passing conf in "Add DAG Run" view](https://github.com/apache/airflow/pull/7281) +- [Allow dags to run for future execution dates](https://github.com/apache/airflow/pull/7038) +- [Dedicated page in documentation for all configs in airflow.cfg](https://airflow.apache.org/docs/1.10.9/configurations-ref.html) + +### Add tags to DAGs and use it for filtering in the UI + +In order to filter DAGs (e.g by team), you can add tags in each dag. The filter is saved in a cookie and can be reset by the reset button. + +For example: + +In your Dag file, pass a list of tags you want to add to DAG object: + +```python +dag = DAG( +dag_id='example_dag_tag', +schedule_interval='0 0 * * *', +tags=['example'] +) +``` + +**Screenshot**: + + +**Note**: This feature is only available for the RBAC UI (enabled using `rbac=True` in `[webserver]` section in your `airflow.cfg`). + + +## Special Note / Deprecations + +### Python 2 +Python 2 has reached end of its life on Jan 2020. Airflow Master no longer supports Python 2. +Airflow 1.10.* would be the last series to support Python 2. + +We strongly recommend users to use Python >= 3.6 + +### Use Airflow RBAC UI +Airflow 1.10.9 ships with 2 UIs, the default is non-RBAC Flask-admin based UI and Flask-appbuilder based UI. + +The Flask-AppBuilder (FAB) based UI is allows Role-based Access Control and has more advanced features compared to +the legacy Flask-admin based UI. This UI can be enabled by setting `rbac=True` in `[webserver]` section in your `airflow.cfg`. + +Flask-admin based UI is deprecated and new features won't be ported to it. This UI will still be the default +for 1.10.* series but would no longer be available from Airflow 2.0 + + +## List of Contributors + +According to git shortlog, the following people contributed to the 1.10.8 and 1.10.9 release. Thank you to all contributors! + +Anita Fronczak, Ash Berlin-Taylor, BasPH, Bharat Kashyap, Bharath Palaksha, Bhavika Tekwani, Bjorn Olsen, Brian Phillips, Cooper Gillan, Daniel Cohen, Daniel Imberman, Daniel Standish, Gabriel Eckers, Hossein Torabi, Igor Khrol, Jacob, Jarek Potiuk, Jay, Jiajie Zhong, Jithin Sukumar, Kamil Breguła, Kaxil Naik, Kousuke Saruta, Mustafa Gök, Noël Bardelot, Oluwafemi Sule, Pete DeJoy, QP Hou, Qian Yu, Robin Edwards, Ry Walker, Steven van Rossum, Tomek Urbaszek, Xinbin Huang, Yuen-Kuei Hsueh, [...]
[airflow-site] branch add-release-notes created (now 2822caa)
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a change to branch add-release-notes in repository https://gitbox.apache.org/repos/asf/airflow-site.git. at 2822caa Add blogpost for 1.10.8 and 1.10.9 This branch includes the following new commits: new 2822caa Add blogpost for 1.10.8 and 1.10.9 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[GitHub] [airflow] codecov-io edited a comment on issue #7428: [AIRFLOW-4526] Poll k8s logs to avoid problem with kubernetes logs --follow
codecov-io edited a comment on issue #7428: [AIRFLOW-4526] Poll k8s logs to avoid problem with kubernetes logs --follow URL: https://github.com/apache/airflow/pull/7428#issuecomment-590111637 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7428?src=pr=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@aa711f1`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `92.11%`. [](https://codecov.io/gh/apache/airflow/pull/7428?src=pr=tree) ```diff @@Coverage Diff@@ ## master#7428 +/- ## = Coverage ? 86.47% = Files ? 893 Lines ?42226 Branches ?0 = Hits ?36514 Misses? 5712 Partials ?0 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7428?src=pr=tree) | Coverage Δ | | |---|---|---| | [...cncf/kubernetes/example\_dags/example\_kubernetes.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL2V4YW1wbGVfZGFncy9leGFtcGxlX2t1YmVybmV0ZXMucHk=) | `78.57% <ø> (ø)` | | | [...low/providers/yandex/hooks/yandexcloud\_dataproc.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMveWFuZGV4L2hvb2tzL3lhbmRleGNsb3VkX2RhdGFwcm9jLnB5) | `100% <ø> (ø)` | | | [airflow/ti\_deps/deps/dagrun\_exists\_dep.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvZGFncnVuX2V4aXN0c19kZXAucHk=) | `88.88% <ø> (ø)` | | | [.../providers/google/suite/operators/gcs\_to\_gdrive.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL3N1aXRlL29wZXJhdG9ycy9nY3NfdG9fZ2RyaXZlLnB5) | `100% <ø> (ø)` | | | [...w/providers/apache/pig/example\_dags/example\_pig.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL3BpZy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9waWcucHk=) | `100% <ø> (ø)` | | | [airflow/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9fX2luaXRfXy5weQ==) | `100% <ø> (ø)` | | | [airflow/ti\_deps/deps/not\_in\_retry\_period\_dep.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvbm90X2luX3JldHJ5X3BlcmlvZF9kZXAucHk=) | `100% <ø> (ø)` | | | [...providers/google/cloud/example\_dags/example\_dlp.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2V4YW1wbGVfZGFncy9leGFtcGxlX2RscC5weQ==) | `100% <ø> (ø)` | | | [airflow/models/dagbag.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnYmFnLnB5) | `89.56% <ø> (ø)` | | | [airflow/ti\_deps/deps/dag\_ti\_slots\_available\_dep.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvZGFnX3RpX3Nsb3RzX2F2YWlsYWJsZV9kZXAucHk=) | `100% <ø> (ø)` | | | ... and [161 more](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7428?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7428?src=pr=footer). Last update [aa711f1...17f60c9](https://codecov.io/gh/apache/airflow/pull/7428?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7428: [AIRFLOW-4526] Poll k8s logs to avoid problem with kubernetes logs --follow
codecov-io commented on issue #7428: [AIRFLOW-4526] Poll k8s logs to avoid problem with kubernetes logs --follow URL: https://github.com/apache/airflow/pull/7428#issuecomment-590111637 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7428?src=pr=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@aa711f1`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `92.11%`. [](https://codecov.io/gh/apache/airflow/pull/7428?src=pr=tree) ```diff @@Coverage Diff@@ ## master#7428 +/- ## = Coverage ? 86.47% = Files ? 893 Lines ?42226 Branches ?0 = Hits ?36514 Misses? 5712 Partials ?0 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7428?src=pr=tree) | Coverage Δ | | |---|---|---| | [...cncf/kubernetes/example\_dags/example\_kubernetes.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL2V4YW1wbGVfZGFncy9leGFtcGxlX2t1YmVybmV0ZXMucHk=) | `78.57% <ø> (ø)` | | | [...low/providers/yandex/hooks/yandexcloud\_dataproc.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMveWFuZGV4L2hvb2tzL3lhbmRleGNsb3VkX2RhdGFwcm9jLnB5) | `100% <ø> (ø)` | | | [airflow/ti\_deps/deps/dagrun\_exists\_dep.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvZGFncnVuX2V4aXN0c19kZXAucHk=) | `88.88% <ø> (ø)` | | | [.../providers/google/suite/operators/gcs\_to\_gdrive.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL3N1aXRlL29wZXJhdG9ycy9nY3NfdG9fZ2RyaXZlLnB5) | `100% <ø> (ø)` | | | [...w/providers/apache/pig/example\_dags/example\_pig.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL3BpZy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9waWcucHk=) | `100% <ø> (ø)` | | | [airflow/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9fX2luaXRfXy5weQ==) | `100% <ø> (ø)` | | | [airflow/ti\_deps/deps/not\_in\_retry\_period\_dep.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvbm90X2luX3JldHJ5X3BlcmlvZF9kZXAucHk=) | `100% <ø> (ø)` | | | [...providers/google/cloud/example\_dags/example\_dlp.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2V4YW1wbGVfZGFncy9leGFtcGxlX2RscC5weQ==) | `100% <ø> (ø)` | | | [airflow/models/dagbag.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnYmFnLnB5) | `89.56% <ø> (ø)` | | | [airflow/ti\_deps/deps/dag\_ti\_slots\_available\_dep.py](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvZGFnX3RpX3Nsb3RzX2F2YWlsYWJsZV9kZXAucHk=) | `100% <ø> (ø)` | | | ... and [161 more](https://codecov.io/gh/apache/airflow/pull/7428/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7428?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7428?src=pr=footer). Last update [aa711f1...17f60c9](https://codecov.io/gh/apache/airflow/pull/7428?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #7489: [AIRFLOW-6869][WIP] Bulk fetch DAGRuns for _process_task_instances
mik-laj commented on a change in pull request #7489: [AIRFLOW-6869][WIP] Bulk
fetch DAGRuns for _process_task_instances
URL: https://github.com/apache/airflow/pull/7489#discussion_r383032005
##
File path: airflow/jobs/scheduler_job.py
##
@@ -699,23 +727,30 @@ def _process_dags(self, dagbag, dags, tis_out):
self.log.info("Not processing DAG %s since it's paused",
dag.dag_id)
continue
-self.log.info("Processing %s", dag.dag_id)
+dag_id = dag.dag_id
+self.log.info("Processing %s", dag_id)
+dag_runs_for_dag = dag_runs_by_dag_id[dag_id] if dag_id in
dag_runs_by_dag_id else []
# Only creates DagRun for DAGs that are not subdag since
# DagRun of subdags are created when SubDagOperator executes.
if not dag.is_subdag:
dag_run = self.create_dag_run(dag)
if dag_run:
+dag_runs_for_dag.append(dag_run)
expected_start_date =
dag.following_schedule(dag_run.execution_date)
if expected_start_date:
schedule_delay = dag_run.start_date -
expected_start_date
Stats.timing(
'dagrun.schedule_delay.{dag_id}'.format(dag_id=dag.dag_id),
schedule_delay)
-self.log.info("Created %s", dag_run)
-self._process_task_instances(dag, tis_out)
-if check_slas:
-self.manage_slas(dag)
+self.log.info("Created %s", dag_run)
+
+if dag_runs_for_dag:
+tis_out.extend(self._process_task_instances(dag,
dag_runs_for_dag))
+if check_slas:
+self.manage_slas(dag)
Review comment:
If the task in DAGRun was executed then there must be at least one active
DagRun. `DagRun.update_state` method is called by `_process_task_instances.
First, we fetch the runs that are running, and then change from running to
another state. If no run was running, then no task should be running.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on issue #7484: [AIRFLOW-6864][depends on 6863] Make airflow/jobs pylint compatible
ashb commented on issue #7484: [AIRFLOW-6864][depends on 6863] Make airflow/jobs pylint compatible URL: https://github.com/apache/airflow/pull/7484#issuecomment-590101310 Good spot Kamil! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
