[GitHub] bolkedebruin closed pull request #4361: [AIRFLOW-3558] Improve default tox flake8 excludes
bolkedebruin closed pull request #4361: [AIRFLOW-3558] Improve default tox flake8 excludes URL: https://github.com/apache/incubator-airflow/pull/4361 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/.flake8 b/.flake8 index 2723df1f10..9c4bbcd661 100644 --- a/.flake8 +++ b/.flake8 @@ -1,3 +1,4 @@ [flake8] max-line-length = 110 ignore = E731 +exclude = .git, env, venv, .venv, VENV, .tox, .eggs, .* This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3558) Have tox flake8 skip ignored and hidden directories
[ https://issues.apache.org/jira/browse/AIRFLOW-3558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16727826#comment-16727826 ] ASF GitHub Bot commented on AIRFLOW-3558: - bolkedebruin commented on pull request #4361: [AIRFLOW-3558] Improve default tox flake8 excludes URL: https://github.com/apache/incubator-airflow/pull/4361 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Have tox flake8 skip ignored and hidden directories > --- > > Key: AIRFLOW-3558 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3558 > Project: Apache Airflow > Issue Type: Improvement >Reporter: holdenk >Assignee: holdenk >Priority: Trivial > > By default if you run tox with the flake8 target in Airflow it checks all of > the directories, this includes .eggs, env, etc. all of which are ignored by > our gitignore but cat by flake8 and gives a bunch of errors for non-Airflow > code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3356) Scheduler gets stuck for certain DAGs
[ https://issues.apache.org/jira/browse/AIRFLOW-3356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16727825#comment-16727825 ] jack commented on AIRFLOW-3356: --- I'm having the same issue also on 1.9 Some tasks are stuck on running even though they are not (tasks that takes 1-4 minutes to execute get stuck for hours)... Only clearing them solve this and then they are re-scheduled. > Scheduler gets stuck for certain DAGs > - > > Key: AIRFLOW-3356 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3356 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Affects Versions: 1.9.0 >Reporter: John Smodic >Priority: Critical > > I observe the scheduler getting stuck for certain DAGs: > Nov 15 19:11:48 ip-172-16-13-120 python3.6[1319]: File Path PID Runtime Last > Runtime Last Run > Nov 15 19:11:48 ip-172-16-13-120 python3.6[1319]: > /home/ubuntu/airflow/dags/stuck_dag.py 14241 *19977.55s* 1.05s > 2018-11-15T13:38:47 > Nov 15 19:11:48 ip-172-16-13-120 python3.6[1319]: > /home/ubuntu/airflow/dags/not_stuck_dag.py 19906 0.05s 1.05s > 2018-11-15T19:11:44 > > The "Runtime" of the stuck DAG's scheduling process is huge and I can't tell > what it's doing. There's no mention of that DAG in the scheduler logs > otherwise. > > The mapped process looks like this: > ubuntu 14241 0.0 0.3 371132 63232 ? S 13:38 0:00 /usr/bin/python3.6 > /usr/local/bin/airflow scheduler > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] kaxil commented on issue #4363: [AIRFLOW-3560] Add WeekEnd & DayOfWeek Sensors
kaxil commented on issue #4363: [AIRFLOW-3560] Add WeekEnd & DayOfWeek Sensors URL: https://github.com/apache/incubator-airflow/pull/4363#issuecomment-449617519 @feng-tao Not Really, the TimeDeltaSensor adds X sec/min/days but doesn't necessarily check what day it is. DayOfWeekSensor here would give the user explicit option to specify that. WeekEnd Just checks if it is Sat or Sun This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kaxil commented on a change in pull request #4359: [AIRFLOW-3150] Make execution_date templated in TriggerDagRunOperator
kaxil commented on a change in pull request #4359: [AIRFLOW-3150] Make execution_date templated in TriggerDagRunOperator URL: https://github.com/apache/incubator-airflow/pull/4359#discussion_r243753098 ## File path: airflow/operators/dagrun_operator.py ## @@ -64,10 +66,26 @@ def __init__( super(TriggerDagRunOperator, self).__init__(*args, **kwargs) self.python_callable = python_callable self.trigger_dag_id = trigger_dag_id -self.execution_date = execution_date + +if isinstance(execution_date, datetime.datetime): +self.execution_date = execution_date.isoformat() +elif isinstance(execution_date, six.string_types): +self.execution_date = execution_date +elif execution_date is None: +self.execution_date = execution_date Review comment: It uses the current time when execution date is None This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kaxil commented on a change in pull request #4359: [AIRFLOW-3150] Make execution_date templated in TriggerDagRunOperator
kaxil commented on a change in pull request #4359: [AIRFLOW-3150] Make execution_date templated in TriggerDagRunOperator URL: https://github.com/apache/incubator-airflow/pull/4359#discussion_r243753086 ## File path: airflow/operators/dagrun_operator.py ## @@ -64,10 +66,26 @@ def __init__( super(TriggerDagRunOperator, self).__init__(*args, **kwargs) self.python_callable = python_callable self.trigger_dag_id = trigger_dag_id -self.execution_date = execution_date + +if isinstance(execution_date, datetime.datetime): +self.execution_date = execution_date.isoformat() +elif isinstance(execution_date, six.string_types): +self.execution_date = execution_date +elif execution_date is None: +self.execution_date = execution_date Review comment: No, it is the default value as well and will also break backwards compatiblility. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-1684) Branching based on XCOM variable
[ https://issues.apache.org/jira/browse/AIRFLOW-1684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16727813#comment-16727813 ] ASF GitHub Bot commented on AIRFLOW-1684: - eladkal commented on pull request #4365: [AIRFLOW-1684] - Branching based on XCom variable (Docs) URL: https://github.com/apache/incubator-airflow/pull/4365 Elaborate how to use branching with XComs ### Jira - [ ] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW-1684) issues and references them in the PR title. ### Description - Elaborate how to use branching with XComs This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Branching based on XCOM variable > > > Key: AIRFLOW-1684 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1684 > Project: Apache Airflow > Issue Type: New Feature > Components: xcom >Affects Versions: 1.7.0 > Environment: Centos 7, Airflow1.7 >Reporter: Virendhar Sivaraman >Assignee: Elad >Priority: Major > > I would like to branch my dag based on a XCOM variable. > Steps: > 1. Populate XCOM in bash > 2. pull the XCOM variable in a BranchPythonOperator and branch it out based > on the XCOM variable > I've tried the documentation and researched on the internet - haven't been > successful. > This feature will be helpful if its not available yet. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] eladkal commented on issue #4365: [AIRFLOW-1684] - Branching based on XCom variable (Docs)
eladkal commented on issue #4365: [AIRFLOW-1684] - Branching based on XCom variable (Docs) URL: https://github.com/apache/incubator-airflow/pull/4365#issuecomment-449617129 @feluelle follow our chat on Slack. Finally had the time to submit PR :) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] eladkal opened a new pull request #4365: [AIRFLOW-1684] - Branching based on XCom variable (Docs)
eladkal opened a new pull request #4365: [AIRFLOW-1684] - Branching based on XCom variable (Docs) URL: https://github.com/apache/incubator-airflow/pull/4365 Elaborate how to use branching with XComs ### Jira - [ ] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW-1684) issues and references them in the PR title. ### Description - Elaborate how to use branching with XComs This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (AIRFLOW-1684) Branching based on XCOM variable
[ https://issues.apache.org/jira/browse/AIRFLOW-1684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elad reassigned AIRFLOW-1684: - Assignee: Elad > Branching based on XCOM variable > > > Key: AIRFLOW-1684 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1684 > Project: Apache Airflow > Issue Type: New Feature > Components: xcom >Affects Versions: 1.7.0 > Environment: Centos 7, Airflow1.7 >Reporter: Virendhar Sivaraman >Assignee: Elad >Priority: Major > > I would like to branch my dag based on a XCOM variable. > Steps: > 1. Populate XCOM in bash > 2. pull the XCOM variable in a BranchPythonOperator and branch it out based > on the XCOM variable > I've tried the documentation and researched on the internet - haven't been > successful. > This feature will be helpful if its not available yet. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] feng-tao commented on issue #4363: [AIRFLOW-3560] Add WeekEnd & DayOfWeek Sensors
feng-tao commented on issue #4363: [AIRFLOW-3560] Add WeekEnd & DayOfWeek Sensors URL: https://github.com/apache/incubator-airflow/pull/4363#issuecomment-449615720 is it possible to use TimeDeltaSensor for this purpose? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] feng-tao commented on a change in pull request #4359: [AIRFLOW-3150] Make execution_date templated in TriggerDagRunOperator
feng-tao commented on a change in pull request #4359: [AIRFLOW-3150] Make execution_date templated in TriggerDagRunOperator URL: https://github.com/apache/incubator-airflow/pull/4359#discussion_r243752391 ## File path: airflow/operators/dagrun_operator.py ## @@ -64,10 +66,26 @@ def __init__( super(TriggerDagRunOperator, self).__init__(*args, **kwargs) self.python_callable = python_callable self.trigger_dag_id = trigger_dag_id -self.execution_date = execution_date + +if isinstance(execution_date, datetime.datetime): +self.execution_date = execution_date.isoformat() +elif isinstance(execution_date, six.string_types): +self.execution_date = execution_date +elif execution_date is None: +self.execution_date = execution_date Review comment: if execution_date is None, should we raise error ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jmcarp opened a new pull request #4364: [AIRFLOW-3550] Standardize GKE hook.
jmcarp opened a new pull request #4364: [AIRFLOW-3550] Standardize GKE hook. URL: https://github.com/apache/incubator-airflow/pull/4364 Refactor `GKEClusterHook` to subclass `GoogleCloudBaseHook` and authenticate with connection credentials. Make sure you have checked _all_ steps below. ### Jira - [x] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-3550 - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. ### Description - [x] Here are some details about my PR, including screenshots of any UI changes: Refactor `GKEClusterHook` to subclass `GoogleCloudBaseHook` and authenticate with connection credentials. ### Tests - [x] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: ### Commits - [x] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [x] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. - All the public functions and the classes in the PR contain docstrings that explain what it does ### Code Quality - [x] Passes `flake8` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3550) GKEClusterHook doesn't use gcp_conn_id
[ https://issues.apache.org/jira/browse/AIRFLOW-3550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16727807#comment-16727807 ] ASF GitHub Bot commented on AIRFLOW-3550: - jmcarp commented on pull request #4364: [AIRFLOW-3550] Standardize GKE hook. URL: https://github.com/apache/incubator-airflow/pull/4364 Refactor `GKEClusterHook` to subclass `GoogleCloudBaseHook` and authenticate with connection credentials. Make sure you have checked _all_ steps below. ### Jira - [x] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-3550 - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. ### Description - [x] Here are some details about my PR, including screenshots of any UI changes: Refactor `GKEClusterHook` to subclass `GoogleCloudBaseHook` and authenticate with connection credentials. ### Tests - [x] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: ### Commits - [x] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [x] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. - All the public functions and the classes in the PR contain docstrings that explain what it does ### Code Quality - [x] Passes `flake8` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > GKEClusterHook doesn't use gcp_conn_id > -- > > Key: AIRFLOW-3550 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3550 > Project: Apache Airflow > Issue Type: Bug > Components: contrib >Affects Versions: 1.10.0, 1.10.1 >Reporter: Wilson Lian >Priority: Major > > The hook doesn't inherit from GoogleCloudBaseHook. API calls are made using > the default service account (if present). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] stale[bot] commented on issue #4133: [AIRFLOW-3270] Allow passwordless-binding for LDAP auth backend
stale[bot] commented on issue #4133: [AIRFLOW-3270] Allow passwordless-binding for LDAP auth backend URL: https://github.com/apache/incubator-airflow/pull/4133#issuecomment-449611637 This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3560) Add Sensor that polls until a day of the week
[ https://issues.apache.org/jira/browse/AIRFLOW-3560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16727764#comment-16727764 ] ASF GitHub Bot commented on AIRFLOW-3560: - kaxil commented on pull request #4363: [AIRFLOW-3560] Add WeekEnd & DayOfWeek Sensors URL: https://github.com/apache/incubator-airflow/pull/4363 Make sure you have checked _all_ steps below. ### Jira - [x] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-3560 ### Description - [x] Here are some details about my PR, including screenshots of any UI changes: One of the use-cases we had is we wanted to run certain tasks only on Weekends or certain days of the weeks. Along the way, I have seen more people requiring the same. ### Tests - [x] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: * `DayOfWeekSensorTests` * `WeekEndSensorTests` ### Commits - [x] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [x] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. - All the public functions and the classes in the PR contain docstrings that explain what it does ### Code Quality - [x] Passes `flake8` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Sensor that polls until a day of the week > - > > Key: AIRFLOW-3560 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3560 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Kaxil Naik >Assignee: Kaxil Naik >Priority: Minor > Fix For: 1.10.2 > > > One of the use-case we have is we want to run certain tasks only on Weekends -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] kaxil opened a new pull request #4363: [AIRFLOW-3560] Add WeekEnd & DayOfWeek Sensors
kaxil opened a new pull request #4363: [AIRFLOW-3560] Add WeekEnd & DayOfWeek Sensors URL: https://github.com/apache/incubator-airflow/pull/4363 Make sure you have checked _all_ steps below. ### Jira - [x] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-3560 ### Description - [x] Here are some details about my PR, including screenshots of any UI changes: One of the use-cases we had is we wanted to run certain tasks only on Weekends or certain days of the weeks. Along the way, I have seen more people requiring the same. ### Tests - [x] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: * `DayOfWeekSensorTests` * `WeekEndSensorTests` ### Commits - [x] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [x] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. - All the public functions and the classes in the PR contain docstrings that explain what it does ### Code Quality - [x] Passes `flake8` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-3560) Add Sensor that polls until a day of the week
Kaxil Naik created AIRFLOW-3560: --- Summary: Add Sensor that polls until a day of the week Key: AIRFLOW-3560 URL: https://issues.apache.org/jira/browse/AIRFLOW-3560 Project: Apache Airflow Issue Type: New Feature Reporter: Kaxil Naik Assignee: Kaxil Naik Fix For: 1.10.2 One of the use-case we have is we want to run certain tasks only on Weekends -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] codecov-io commented on issue #4359: [AIRFLOW-3150] Make execution_date templated in TriggerDagRunOperator
codecov-io commented on issue #4359: [AIRFLOW-3150] Make execution_date templated in TriggerDagRunOperator URL: https://github.com/apache/incubator-airflow/pull/4359#issuecomment-449597919 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4359?src=pr=h1) Report > Merging [#4359](https://codecov.io/gh/apache/incubator-airflow/pull/4359?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/0e365665957d09c1aa612014b40994e3634ef70e?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `93.33%`. [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-airflow/pull/4359/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4359?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4359 +/- ## == + Coverage 78.12% 78.13% +<.01% == Files 202 202 Lines 1648316495 +12 == + Hits1287812889 +11 - Misses 3605 3606 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4359?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/operators/dagrun\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4359/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZGFncnVuX29wZXJhdG9yLnB5) | `94.73% <93.33%> (-1.42%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4359?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4359?src=pr=footer). Last update [0e36566...670c29a](https://codecov.io/gh/apache/incubator-airflow/pull/4359?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4359: [AIRFLOW-3150] Make execution_date templated in TriggerDagRunOperator
codecov-io edited a comment on issue #4359: [AIRFLOW-3150] Make execution_date templated in TriggerDagRunOperator URL: https://github.com/apache/incubator-airflow/pull/4359#issuecomment-449597919 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4359?src=pr=h1) Report > Merging [#4359](https://codecov.io/gh/apache/incubator-airflow/pull/4359?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/0e365665957d09c1aa612014b40994e3634ef70e?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `93.33%`. [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-airflow/pull/4359/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4359?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4359 +/- ## == + Coverage 78.12% 78.13% +<.01% == Files 202 202 Lines 1648316495 +12 == + Hits1287812889 +11 - Misses 3605 3606 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4359?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/operators/dagrun\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4359/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZGFncnVuX29wZXJhdG9yLnB5) | `94.73% <93.33%> (-1.42%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4359?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4359?src=pr=footer). Last update [0e36566...670c29a](https://codecov.io/gh/apache/incubator-airflow/pull/4359?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4362: [AIRFLOW-3559] Add missing options to DatadogHook.
codecov-io edited a comment on issue #4362: [AIRFLOW-3559] Add missing options to DatadogHook. URL: https://github.com/apache/incubator-airflow/pull/4362#issuecomment-449595748 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4362?src=pr=h1) Report > Merging [#4362](https://codecov.io/gh/apache/incubator-airflow/pull/4362?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/0e365665957d09c1aa612014b40994e3634ef70e?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-airflow/pull/4362/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4362?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#4362 +/- ## === Coverage 78.12% 78.12% === Files 202 202 Lines 1648316483 === Hits1287812878 Misses 3605 3605 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4362?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4362?src=pr=footer). Last update [0e36566...359f778](https://codecov.io/gh/apache/incubator-airflow/pull/4362?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #4362: [AIRFLOW-3559] Add missing options to DatadogHook.
codecov-io commented on issue #4362: [AIRFLOW-3559] Add missing options to DatadogHook. URL: https://github.com/apache/incubator-airflow/pull/4362#issuecomment-449595748 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4362?src=pr=h1) Report > Merging [#4362](https://codecov.io/gh/apache/incubator-airflow/pull/4362?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/0e365665957d09c1aa612014b40994e3634ef70e?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-airflow/pull/4362/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4362?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#4362 +/- ## === Coverage 78.12% 78.12% === Files 202 202 Lines 1648316483 === Hits1287812878 Misses 3605 3605 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4362?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4362?src=pr=footer). Last update [0e36566...359f778](https://codecov.io/gh/apache/incubator-airflow/pull/4362?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-1077) Subdags can deadlock
[ https://issues.apache.org/jira/browse/AIRFLOW-1077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16727537#comment-16727537 ] ASF GitHub Bot commented on AIRFLOW-1077: - stale[bot] closed pull request #2367: [AIRFLOW-1077] Warn about subdag deadlock case URL: https://github.com/apache/incubator-airflow/pull/2367 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/docs/concepts.rst b/docs/concepts.rst index 33a6ea44c7..56fbd2a531 100644 --- a/docs/concepts.rst +++ b/docs/concepts.rst @@ -457,10 +457,10 @@ Not like this, where the join task is skipped .. image:: img/branch_bad.png -SubDAGs +SubDags === -SubDAGs are perfect for repeating patterns. Defining a function that returns a +SubDags are perfect for repeating patterns. Defining a function that returns a DAG object is a nice design pattern when using Airflow. Airbnb uses the *stage-check-exchange* pattern when loading data. Data is staged @@ -472,13 +472,13 @@ As another example, consider the following DAG: .. image:: img/subdag_before.png -We can combine all of the parallel ``task-*`` operators into a single SubDAG, +We can combine all of the parallel ``task-*`` operators into a single SubDag, so that the resulting DAG resembles the following: .. image:: img/subdag_after.png -Note that SubDAG operators should contain a factory method that returns a DAG -object. This will prevent the SubDAG from being treated like a separate DAG in +Note that SubDag operators should contain a factory method that returns a DAG +object. This will prevent the SubDag from being treated like a separate DAG in the main UI. For example: .. code:: python @@ -503,7 +503,7 @@ the main UI. For example: return dag -This SubDAG can then be referenced in your main DAG file: +This SubDag can then be referenced in your main DAG file: .. code:: python @@ -531,29 +531,36 @@ This SubDAG can then be referenced in your main DAG file: ) You can zoom into a SubDagOperator from the graph view of the main DAG to show -the tasks contained within the SubDAG: +the tasks contained within the SubDag: .. image:: img/subdag_zoom.png -Some other tips when using SubDAGs: +Some other tips when using SubDags: -- by convention, a SubDAG's ``dag_id`` should be prefixed by its parent and +- by convention, a SubDag's ``dag_id`` should be prefixed by its parent and a dot. As in ``parent.child`` -- share arguments between the main DAG and the SubDAG by passing arguments to - the SubDAG operator (as demonstrated above) -- SubDAGs must have a schedule and be enabled. If the SubDAG's schedule is - set to ``None`` or ``@once``, the SubDAG will succeed without having done +- share arguments between the main DAG and the SubDag by passing arguments to + the SubDag operator (as demonstrated above) +- SubDags must have a schedule and be enabled. If the SubDag's schedule is + set to ``None`` or ``@once``, the SubDag will succeed without having done anything - clearing a SubDagOperator also clears the state of the tasks within - marking success on a SubDagOperator does not affect the state of the tasks within -- refrain from using ``depends_on_past=True`` in tasks within the SubDAG as +- refrain from using ``depends_on_past=True`` in tasks within the SubDag as this can be confusing -- it is possible to specify an executor for the SubDAG. It is common to use - the SequentialExecutor if you want to run the SubDAG in-process and +- it is possible to specify an executor for the SubDag. It is common to use + the SequentialExecutor if you want to run the SubDag in-process and effectively limit its parallelism to one. Using LocalExecutor can be problematic as it may over-subscribe your worker, running multiple tasks in a single slot +- do not create more SubDags then your concurrency limit or the scheduler + will deadlock. Each SubDags counts towards your concurrency limit. For + example, if you have a concurrency limit of 16 and you have 25 SubDags, + the 16 SubDags will be scheduled, effectively blocking any of the tasks + within the given SubDags. You can work around this by setting the SubDag's + executor to SequentialExecutor. This allows multiple SubDag to run + concurrently without locking the tasks within the SubDag See ``airflow/example_dags`` for a demonstration. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org >
[GitHub] stale[bot] closed pull request #2367: [AIRFLOW-1077] Warn about subdag deadlock case
stale[bot] closed pull request #2367: [AIRFLOW-1077] Warn about subdag deadlock case URL: https://github.com/apache/incubator-airflow/pull/2367 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/docs/concepts.rst b/docs/concepts.rst index 33a6ea44c7..56fbd2a531 100644 --- a/docs/concepts.rst +++ b/docs/concepts.rst @@ -457,10 +457,10 @@ Not like this, where the join task is skipped .. image:: img/branch_bad.png -SubDAGs +SubDags === -SubDAGs are perfect for repeating patterns. Defining a function that returns a +SubDags are perfect for repeating patterns. Defining a function that returns a DAG object is a nice design pattern when using Airflow. Airbnb uses the *stage-check-exchange* pattern when loading data. Data is staged @@ -472,13 +472,13 @@ As another example, consider the following DAG: .. image:: img/subdag_before.png -We can combine all of the parallel ``task-*`` operators into a single SubDAG, +We can combine all of the parallel ``task-*`` operators into a single SubDag, so that the resulting DAG resembles the following: .. image:: img/subdag_after.png -Note that SubDAG operators should contain a factory method that returns a DAG -object. This will prevent the SubDAG from being treated like a separate DAG in +Note that SubDag operators should contain a factory method that returns a DAG +object. This will prevent the SubDag from being treated like a separate DAG in the main UI. For example: .. code:: python @@ -503,7 +503,7 @@ the main UI. For example: return dag -This SubDAG can then be referenced in your main DAG file: +This SubDag can then be referenced in your main DAG file: .. code:: python @@ -531,29 +531,36 @@ This SubDAG can then be referenced in your main DAG file: ) You can zoom into a SubDagOperator from the graph view of the main DAG to show -the tasks contained within the SubDAG: +the tasks contained within the SubDag: .. image:: img/subdag_zoom.png -Some other tips when using SubDAGs: +Some other tips when using SubDags: -- by convention, a SubDAG's ``dag_id`` should be prefixed by its parent and +- by convention, a SubDag's ``dag_id`` should be prefixed by its parent and a dot. As in ``parent.child`` -- share arguments between the main DAG and the SubDAG by passing arguments to - the SubDAG operator (as demonstrated above) -- SubDAGs must have a schedule and be enabled. If the SubDAG's schedule is - set to ``None`` or ``@once``, the SubDAG will succeed without having done +- share arguments between the main DAG and the SubDag by passing arguments to + the SubDag operator (as demonstrated above) +- SubDags must have a schedule and be enabled. If the SubDag's schedule is + set to ``None`` or ``@once``, the SubDag will succeed without having done anything - clearing a SubDagOperator also clears the state of the tasks within - marking success on a SubDagOperator does not affect the state of the tasks within -- refrain from using ``depends_on_past=True`` in tasks within the SubDAG as +- refrain from using ``depends_on_past=True`` in tasks within the SubDag as this can be confusing -- it is possible to specify an executor for the SubDAG. It is common to use - the SequentialExecutor if you want to run the SubDAG in-process and +- it is possible to specify an executor for the SubDag. It is common to use + the SequentialExecutor if you want to run the SubDag in-process and effectively limit its parallelism to one. Using LocalExecutor can be problematic as it may over-subscribe your worker, running multiple tasks in a single slot +- do not create more SubDags then your concurrency limit or the scheduler + will deadlock. Each SubDags counts towards your concurrency limit. For + example, if you have a concurrency limit of 16 and you have 25 SubDags, + the 16 SubDags will be scheduled, effectively blocking any of the tasks + within the given SubDags. You can work around this by setting the SubDag's + executor to SequentialExecutor. This allows multiple SubDag to run + concurrently without locking the tasks within the SubDag See ``airflow/example_dags`` for a demonstration. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #4361: [AIRFLOW-3558] Improve default tox flake8 excludes
codecov-io commented on issue #4361: [AIRFLOW-3558] Improve default tox flake8 excludes URL: https://github.com/apache/incubator-airflow/pull/4361#issuecomment-449592064 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4361?src=pr=h1) Report > Merging [#4361](https://codecov.io/gh/apache/incubator-airflow/pull/4361?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/0e365665957d09c1aa612014b40994e3634ef70e?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-airflow/pull/4361/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4361?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#4361 +/- ## === Coverage 78.12% 78.12% === Files 202 202 Lines 1648316483 === Hits1287812878 Misses 3605 3605 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4361?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4361?src=pr=footer). Last update [0e36566...a18c8af](https://codecov.io/gh/apache/incubator-airflow/pull/4361?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4361: [AIRFLOW-3558] Improve default tox flake8 excludes
codecov-io edited a comment on issue #4361: [AIRFLOW-3558] Improve default tox flake8 excludes URL: https://github.com/apache/incubator-airflow/pull/4361#issuecomment-449592064 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4361?src=pr=h1) Report > Merging [#4361](https://codecov.io/gh/apache/incubator-airflow/pull/4361?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/0e365665957d09c1aa612014b40994e3634ef70e?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-airflow/pull/4361/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4361?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#4361 +/- ## === Coverage 78.12% 78.12% === Files 202 202 Lines 1648316483 === Hits1287812878 Misses 3605 3605 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4361?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4361?src=pr=footer). Last update [0e36566...a18c8af](https://codecov.io/gh/apache/incubator-airflow/pull/4361?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-3150) Make execution_date a template field in TriggerDagRunOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-3150: Fix Version/s: 1.10.2 > Make execution_date a template field in TriggerDagRunOperator > - > > Key: AIRFLOW-3150 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3150 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Reporter: Kyle Hamlin >Assignee: Kaxil Naik >Priority: Minor > Labels: easy-fix > Fix For: 1.10.2 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] jmcarp opened a new pull request #4362: [AIRFLOW-3559] Add missing options to DatadogHook.
jmcarp opened a new pull request #4362: [AIRFLOW-3559] Add missing options to DatadogHook. URL: https://github.com/apache/incubator-airflow/pull/4362 Make sure you have checked _all_ steps below. ### Jira - [x] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-3559 - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. ### Description - [x] Here are some details about my PR, including screenshots of any UI changes: Adds missing arguments to `DatadogHook`. ### Tests - [x] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: Backfills missing tests for `DatadogHook` ### Commits - [x] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [x] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. - All the public functions and the classes in the PR contain docstrings that explain what it does ### Code Quality - [x] Passes `flake8` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3559) Add missing options to DatadogHook
[ https://issues.apache.org/jira/browse/AIRFLOW-3559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16727533#comment-16727533 ] ASF GitHub Bot commented on AIRFLOW-3559: - jmcarp opened a new pull request #4362: [AIRFLOW-3559] Add missing options to DatadogHook. URL: https://github.com/apache/incubator-airflow/pull/4362 Make sure you have checked _all_ steps below. ### Jira - [x] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-3559 - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. ### Description - [x] Here are some details about my PR, including screenshots of any UI changes: Adds missing arguments to `DatadogHook`. ### Tests - [x] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: Backfills missing tests for `DatadogHook` ### Commits - [x] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [x] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. - All the public functions and the classes in the PR contain docstrings that explain what it does ### Code Quality - [x] Passes `flake8` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add missing options to DatadogHook > -- > > Key: AIRFLOW-3559 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3559 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Josh Carp >Assignee: Josh Carp >Priority: Trivial > > The DataDog hook is missing a few options for creating events and metrics. > I'll add those options and backfill unit tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3559) Add missing options to DatadogHook
Josh Carp created AIRFLOW-3559: -- Summary: Add missing options to DatadogHook Key: AIRFLOW-3559 URL: https://issues.apache.org/jira/browse/AIRFLOW-3559 Project: Apache Airflow Issue Type: Improvement Reporter: Josh Carp Assignee: Josh Carp The DataDog hook is missing a few options for creating events and metrics. I'll add those options and backfill unit tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3558) Have tox flake8 skip ignored and hidden directories
[ https://issues.apache.org/jira/browse/AIRFLOW-3558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16727526#comment-16727526 ] ASF GitHub Bot commented on AIRFLOW-3558: - holdenk opened a new pull request #4361: [AIRFLOW-3558] Improve default tox flake8 excludes URL: https://github.com/apache/incubator-airflow/pull/4361 Right now our gitignore skips a bunch of temporary Python directories but our flake8 config will still test against them, leading to unnecessary error messages. This changes the excludes to skip the common directories that can cause false flake8 failures. Make sure you have checked _all_ steps below. ### Jira - [ X ] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-XXX - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. ### Description - [ X ] Here are some details about my PR, including screenshots of any UI changes: Right now our gitignore skips a bunch of temporary Python directories but our flake8 config will still test against them, leading to unnecessary error messages. This changes the excludes to skip the common directories that can cause false flake8 failures. This should not impact end users. ### Tests - [ X ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: The existing flake8 env still runs from tox ### Commits - [ X ] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [ X ] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. - All the public functions and the classes in the PR contain docstrings that explain what it does No new functionality. ### Code Quality - [ X ] Passes `flake8` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Have tox flake8 skip ignored and hidden directories > --- > > Key: AIRFLOW-3558 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3558 > Project: Apache Airflow > Issue Type: Improvement >Reporter: holdenk >Assignee: holdenk >Priority: Trivial > > By default if you run tox with the flake8 target in Airflow it checks all of > the directories, this includes .eggs, env, etc. all of which are ignored by > our gitignore but cat by flake8 and gives a bunch of errors for non-Airflow > code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] holdenk opened a new pull request #4361: [AIRFLOW-3558] Improve default tox flake8 excludes
holdenk opened a new pull request #4361: [AIRFLOW-3558] Improve default tox flake8 excludes URL: https://github.com/apache/incubator-airflow/pull/4361 Right now our gitignore skips a bunch of temporary Python directories but our flake8 config will still test against them, leading to unnecessary error messages. This changes the excludes to skip the common directories that can cause false flake8 failures. Make sure you have checked _all_ steps below. ### Jira - [ X ] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-XXX - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. ### Description - [ X ] Here are some details about my PR, including screenshots of any UI changes: Right now our gitignore skips a bunch of temporary Python directories but our flake8 config will still test against them, leading to unnecessary error messages. This changes the excludes to skip the common directories that can cause false flake8 failures. This should not impact end users. ### Tests - [ X ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: The existing flake8 env still runs from tox ### Commits - [ X ] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [ X ] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. - All the public functions and the classes in the PR contain docstrings that explain what it does No new functionality. ### Code Quality - [ X ] Passes `flake8` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-3558) Have tox flake8 skip ignored and hidden directories
holdenk created AIRFLOW-3558: Summary: Have tox flake8 skip ignored and hidden directories Key: AIRFLOW-3558 URL: https://issues.apache.org/jira/browse/AIRFLOW-3558 Project: Apache Airflow Issue Type: Improvement Reporter: holdenk Assignee: holdenk By default if you run tox with the flake8 target in Airflow it checks all of the directories, this includes .eggs, env, etc. all of which are ignored by our gitignore but cat by flake8 and gives a bunch of errors for non-Airflow code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] holdenk opened a new pull request #4360: [AIRFLOW-1191] Simplify override of spark submit command
holdenk opened a new pull request #4360: [AIRFLOW-1191] Simplify override of spark submit command URL: https://github.com/apache/incubator-airflow/pull/4360 This will better support distros which ship spark 1 & 2 ( and eventually 3) Make sure you have checked _all_ steps below. ### Jira - [ X ] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-XXX - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. ### Description - [ X ] Here are some details about my PR, including screenshots of any UI changes: Adds a spark_binary param to the spark submit operator to allow folks to more easily configure the operator to use a different binary, as is needed for some distros of the Hadoop ecosystem which ship multiple version of Spark. ### Tests - [ X ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: Updates the existing test_spark_submit_operator to check for override ### Commits - [ X ] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [ X ] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. - All the public functions and the classes in the PR contain docstrings that explain what it does docstring update ### Code Quality - [ ] Passes `flake8` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-1191) Contrib Spark Submit hook should permit override of spark-submit cmd
[ https://issues.apache.org/jira/browse/AIRFLOW-1191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16727522#comment-16727522 ] ASF GitHub Bot commented on AIRFLOW-1191: - holdenk opened a new pull request #4360: [AIRFLOW-1191] Simplify override of spark submit command URL: https://github.com/apache/incubator-airflow/pull/4360 This will better support distros which ship spark 1 & 2 ( and eventually 3) Make sure you have checked _all_ steps below. ### Jira - [ X ] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-XXX - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. ### Description - [ X ] Here are some details about my PR, including screenshots of any UI changes: Adds a spark_binary param to the spark submit operator to allow folks to more easily configure the operator to use a different binary, as is needed for some distros of the Hadoop ecosystem which ship multiple version of Spark. ### Tests - [ X ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: Updates the existing test_spark_submit_operator to check for override ### Commits - [ X ] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [ X ] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. - All the public functions and the classes in the PR contain docstrings that explain what it does docstring update ### Code Quality - [ ] Passes `flake8` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Contrib Spark Submit hook should permit override of spark-submit cmd > > > Key: AIRFLOW-1191 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1191 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib, hooks >Affects Versions: 1.8.1 > Environment: Cloudera based Spark parcel >Reporter: Vianney FOUCAULT >Assignee: Vianney FOUCAULT >Priority: Major > Fix For: 1.10.0 > > > Using Cloudera based cluster with spark 2 parcel that rename spark-submit to > spark2-submit > It should be possible to change the spark submit cmd without specifying a env > var -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-850) Airflow should support a general purpose PythonSensor
[ https://issues.apache.org/jira/browse/AIRFLOW-850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-850. Resolution: Fixed Fix Version/s: 1.10.2 Resolved by https://github.com/apache/incubator-airflow/pull/4349 > Airflow should support a general purpose PythonSensor > - > > Key: AIRFLOW-850 > URL: https://issues.apache.org/jira/browse/AIRFLOW-850 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Affects Versions: 1.8.0 >Reporter: Daniel Gies >Assignee: Daniel Gies >Priority: Major > Fix For: 1.10.2 > > > Today I found myself trying to use a sensor to postpone execution until data > for the current execution date appeared in a file. It occurred to me that > having a general purpose PythonSensor would allow developers to use the > sensor paradigm with arbitrary code. > We should add a PythonSensor to the core sensors module which takes a > python_callable and optional args like the PythonOperator does. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-850) Airflow should support a general purpose PythonSensor
[ https://issues.apache.org/jira/browse/AIRFLOW-850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16727516#comment-16727516 ] ASF GitHub Bot commented on AIRFLOW-850: kaxil closed pull request #4349: [AIRFLOW-850] Add a PythonSensor URL: https://github.com/apache/incubator-airflow/pull/4349 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/airflow/contrib/sensors/python_sensor.py b/airflow/contrib/sensors/python_sensor.py new file mode 100644 index 00..68bc7497ea --- /dev/null +++ b/airflow/contrib/sensors/python_sensor.py @@ -0,0 +1,81 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +from airflow.sensors.base_sensor_operator import BaseSensorOperator +from airflow.utils.decorators import apply_defaults + + +class PythonSensor(BaseSensorOperator): +""" +Waits for a Python callable to return True. + +User could put input argument in templates_dict +e.g templates_dict = {'start_ds': 1970} +and access the argument by calling `kwargs['templates_dict']['start_ds']` +in the the callable + +:param python_callable: A reference to an object that is callable +:type python_callable: python callable +:param op_kwargs: a dictionary of keyword arguments that will get unpacked +in your function +:type op_kwargs: dict +:param op_args: a list of positional arguments that will get unpacked when +calling your callable +:type op_args: list +:param provide_context: if set to true, Airflow will pass a set of +keyword arguments that can be used in your function. This set of +kwargs correspond exactly to what you can use in your jinja +templates. For this to work, you need to define `**kwargs` in your +function header. +:type provide_context: bool +:param templates_dict: a dictionary where the values are templates that +will get templated by the Airflow engine sometime between +``__init__`` and ``execute`` takes place and are made available +in your callable's context after the template has been applied. +:type templates_dict: dict of str +""" + +template_fields = ('templates_dict',) +template_ext = tuple() + +@apply_defaults +def __init__( +self, +python_callable, +op_args=None, +op_kwargs=None, +provide_context=False, +templates_dict=None, +*args, **kwargs): +super(PythonSensor, self).__init__(*args, **kwargs) +self.python_callable = python_callable +self.op_args = op_args or [] +self.op_kwargs = op_kwargs or {} +self.provide_context = provide_context +self.templates_dict = templates_dict + +def poke(self, context): +if self.provide_context: +context.update(self.op_kwargs) +context['templates_dict'] = self.templates_dict +self.op_kwargs = context + +self.log.info("Poking callable: " + str(self.python_callable)) +return_value = self.python_callable(*self.op_args, **self.op_kwargs) +return bool(return_value) diff --git a/docs/code.rst b/docs/code.rst index 61414ecbd6..e890adffec 100644 --- a/docs/code.rst +++ b/docs/code.rst @@ -256,6 +256,7 @@ Sensors .. autoclass:: airflow.contrib.sensors.imap_attachment_sensor.ImapAttachmentSensor .. autoclass:: airflow.contrib.sensors.jira_sensor.JiraSensor .. autoclass:: airflow.contrib.sensors.pubsub_sensor.PubSubPullSensor +.. autoclass:: airflow.contrib.sensors.python_sensor.PythonSensor .. autoclass:: airflow.contrib.sensors.qubole_sensor.QuboleSensor .. autoclass:: airflow.contrib.sensors.redis_key_sensor.RedisKeySensor .. autoclass:: airflow.contrib.sensors.sagemaker_base_sensor.SageMakerBaseSensor diff --git a/tests/contrib/sensors/test_python_sensor.py b/tests/contrib/sensors/test_python_sensor.py new file mode 100644 index
[GitHub] kaxil closed pull request #4349: [AIRFLOW-850] Add a PythonSensor
kaxil closed pull request #4349: [AIRFLOW-850] Add a PythonSensor URL: https://github.com/apache/incubator-airflow/pull/4349 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/airflow/contrib/sensors/python_sensor.py b/airflow/contrib/sensors/python_sensor.py new file mode 100644 index 00..68bc7497ea --- /dev/null +++ b/airflow/contrib/sensors/python_sensor.py @@ -0,0 +1,81 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +from airflow.sensors.base_sensor_operator import BaseSensorOperator +from airflow.utils.decorators import apply_defaults + + +class PythonSensor(BaseSensorOperator): +""" +Waits for a Python callable to return True. + +User could put input argument in templates_dict +e.g templates_dict = {'start_ds': 1970} +and access the argument by calling `kwargs['templates_dict']['start_ds']` +in the the callable + +:param python_callable: A reference to an object that is callable +:type python_callable: python callable +:param op_kwargs: a dictionary of keyword arguments that will get unpacked +in your function +:type op_kwargs: dict +:param op_args: a list of positional arguments that will get unpacked when +calling your callable +:type op_args: list +:param provide_context: if set to true, Airflow will pass a set of +keyword arguments that can be used in your function. This set of +kwargs correspond exactly to what you can use in your jinja +templates. For this to work, you need to define `**kwargs` in your +function header. +:type provide_context: bool +:param templates_dict: a dictionary where the values are templates that +will get templated by the Airflow engine sometime between +``__init__`` and ``execute`` takes place and are made available +in your callable's context after the template has been applied. +:type templates_dict: dict of str +""" + +template_fields = ('templates_dict',) +template_ext = tuple() + +@apply_defaults +def __init__( +self, +python_callable, +op_args=None, +op_kwargs=None, +provide_context=False, +templates_dict=None, +*args, **kwargs): +super(PythonSensor, self).__init__(*args, **kwargs) +self.python_callable = python_callable +self.op_args = op_args or [] +self.op_kwargs = op_kwargs or {} +self.provide_context = provide_context +self.templates_dict = templates_dict + +def poke(self, context): +if self.provide_context: +context.update(self.op_kwargs) +context['templates_dict'] = self.templates_dict +self.op_kwargs = context + +self.log.info("Poking callable: " + str(self.python_callable)) +return_value = self.python_callable(*self.op_args, **self.op_kwargs) +return bool(return_value) diff --git a/docs/code.rst b/docs/code.rst index 61414ecbd6..e890adffec 100644 --- a/docs/code.rst +++ b/docs/code.rst @@ -256,6 +256,7 @@ Sensors .. autoclass:: airflow.contrib.sensors.imap_attachment_sensor.ImapAttachmentSensor .. autoclass:: airflow.contrib.sensors.jira_sensor.JiraSensor .. autoclass:: airflow.contrib.sensors.pubsub_sensor.PubSubPullSensor +.. autoclass:: airflow.contrib.sensors.python_sensor.PythonSensor .. autoclass:: airflow.contrib.sensors.qubole_sensor.QuboleSensor .. autoclass:: airflow.contrib.sensors.redis_key_sensor.RedisKeySensor .. autoclass:: airflow.contrib.sensors.sagemaker_base_sensor.SageMakerBaseSensor diff --git a/tests/contrib/sensors/test_python_sensor.py b/tests/contrib/sensors/test_python_sensor.py new file mode 100644 index 00..6c852e748a --- /dev/null +++ b/tests/contrib/sensors/test_python_sensor.py @@ -0,0 +1,72 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the
[jira] [Commented] (AIRFLOW-3557) Various typos
[ https://issues.apache.org/jira/browse/AIRFLOW-3557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16727513#comment-16727513 ] ASF GitHub Bot commented on AIRFLOW-3557: - kaxil closed pull request #4357: [AIRFLOW-3557] Fix various typos URL: https://github.com/apache/incubator-airflow/pull/4357 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/CHANGELOG.txt b/CHANGELOG.txt index abb0563d71..98a1103792 100644 --- a/CHANGELOG.txt +++ b/CHANGELOG.txt @@ -24,7 +24,7 @@ Improvements: [AIRFLOW-2622] Add "confirm=False" option to SFTPOperator [AIRFLOW-2662] support affinity & nodeSelector policies for kubernetes executor/operator [AIRFLOW-2709] Improve error handling in Databricks hook -[AIRFLOW-2723] Update lxml dependancy to >= 4.0. +[AIRFLOW-2723] Update lxml dependency to >= 4.0. [AIRFLOW-2763] No precheck mechanism in place during worker initialisation for the connection to metadata database [AIRFLOW-2789] Add ability to create single node cluster to DataprocClusterCreateOperator [AIRFLOW-2797] Add ability to create Google Dataproc cluster with custom image @@ -269,7 +269,7 @@ AIRFLOW 1.10.0, 2018-08-03 [AIRFLOW-2429] Make Airflow flake8 compliant [AIRFLOW-2491] Resolve flask version conflict [AIRFLOW-2484] Remove duplicate key in MySQL to GCS Op -[ARIFLOW-2458] Add cassandra-to-gcs operator +[AIRFLOW-2458] Add cassandra-to-gcs operator [AIRFLOW-2477] Improve time units for task duration and landing times charts for RBAC UI [AIRFLOW-2474] Only import snakebite if using py2 [AIRFLOW-48] Parse connection uri querystring @@ -1504,7 +1504,7 @@ AIRFLOW 1.8.0, 2017-03-12 [AIRFLOW-784] Pin funcsigs to 1.0.0 [AIRFLOW-624] Fix setup.py to not import airflow.version as version [AIRFLOW-779] Task should fail with specific message when deleted -[AIRFLOW-778] Fix completey broken MetastorePartitionSensor +[AIRFLOW-778] Fix completely broken MetastorePartitionSensor [AIRFLOW-739] Set pickle_info log to debug [AIRFLOW-771] Make S3 logs append instead of clobber [AIRFLOW-773] Fix flaky datetime addition in api test diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 556a5d847b..2a60f1dc3c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -166,10 +166,10 @@ There are three ways to setup an Apache Airflow development environment. tox -e py35-backend_mysql ``` - If you wish to run individual tests inside of docker enviroment you can do as follows: + If you wish to run individual tests inside of Docker environment you can do as follows: ```bash -# From the container (with your desired enviroment) with druid hook +# From the container (with your desired environment) with druid hook tox -e py35-backend_mysql -- tests/hooks/test_druid_hook.py ``` diff --git a/airflow/contrib/hooks/bigquery_hook.py b/airflow/contrib/hooks/bigquery_hook.py index 5cab013b28..30a16305db 100644 --- a/airflow/contrib/hooks/bigquery_hook.py +++ b/airflow/contrib/hooks/bigquery_hook.py @@ -1594,7 +1594,7 @@ def insert_all(self, project_id, dataset_id, table_id, self.log.info('All row(s) inserted successfully: {}:{}.{}'.format( dataset_project_id, dataset_id, table_id)) else: -error_msg = '{} insert error(s) occured: {}:{}.{}. Details: {}'.format( +error_msg = '{} insert error(s) occurred: {}:{}.{}. Details: {}'.format( len(resp['insertErrors']), dataset_project_id, dataset_id, table_id, resp['insertErrors']) if fail_on_error: diff --git a/airflow/contrib/hooks/emr_hook.py b/airflow/contrib/hooks/emr_hook.py index f9fd3f04de..fcdf4ac848 100644 --- a/airflow/contrib/hooks/emr_hook.py +++ b/airflow/contrib/hooks/emr_hook.py @@ -23,7 +23,7 @@ class EmrHook(AwsHook): """ -Interact with AWS EMR. emr_conn_id is only neccessary for using the +Interact with AWS EMR. emr_conn_id is only necessary for using the create_job_flow method. """ diff --git a/airflow/executors/celery_executor.py b/airflow/executors/celery_executor.py index 98ce6efba7..10694ea4b7 100644 --- a/airflow/executors/celery_executor.py +++ b/airflow/executors/celery_executor.py @@ -74,7 +74,7 @@ def execute_command(command_to_exec): class ExceptionWithTraceback(object): """ -Wrapper class used to propogate exceptions to parent processes from subprocesses. +Wrapper class used to propagate exceptions to parent processes from subprocesses. :param exception: The exception to wrap :type exception: Exception :param traceback: The stacktrace to wrap diff --git a/airflow/sensors/base_sensor_operator.py
[jira] [Resolved] (AIRFLOW-3557) Various typos
[ https://issues.apache.org/jira/browse/AIRFLOW-3557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-3557. - Resolution: Fixed Fix Version/s: 2.0.0 Resolved by https://github.com/apache/incubator-airflow/pull/4357 > Various typos > - > > Key: AIRFLOW-3557 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3557 > Project: Apache Airflow > Issue Type: Bug >Reporter: Bas Harenslak >Priority: Major > Fix For: 2.0.0 > > > Fix various typos, checked with > [misspell|https://github.com/client9/misspell]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] kaxil closed pull request #4357: [AIRFLOW-3557] Fix various typos
kaxil closed pull request #4357: [AIRFLOW-3557] Fix various typos URL: https://github.com/apache/incubator-airflow/pull/4357 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/CHANGELOG.txt b/CHANGELOG.txt index abb0563d71..98a1103792 100644 --- a/CHANGELOG.txt +++ b/CHANGELOG.txt @@ -24,7 +24,7 @@ Improvements: [AIRFLOW-2622] Add "confirm=False" option to SFTPOperator [AIRFLOW-2662] support affinity & nodeSelector policies for kubernetes executor/operator [AIRFLOW-2709] Improve error handling in Databricks hook -[AIRFLOW-2723] Update lxml dependancy to >= 4.0. +[AIRFLOW-2723] Update lxml dependency to >= 4.0. [AIRFLOW-2763] No precheck mechanism in place during worker initialisation for the connection to metadata database [AIRFLOW-2789] Add ability to create single node cluster to DataprocClusterCreateOperator [AIRFLOW-2797] Add ability to create Google Dataproc cluster with custom image @@ -269,7 +269,7 @@ AIRFLOW 1.10.0, 2018-08-03 [AIRFLOW-2429] Make Airflow flake8 compliant [AIRFLOW-2491] Resolve flask version conflict [AIRFLOW-2484] Remove duplicate key in MySQL to GCS Op -[ARIFLOW-2458] Add cassandra-to-gcs operator +[AIRFLOW-2458] Add cassandra-to-gcs operator [AIRFLOW-2477] Improve time units for task duration and landing times charts for RBAC UI [AIRFLOW-2474] Only import snakebite if using py2 [AIRFLOW-48] Parse connection uri querystring @@ -1504,7 +1504,7 @@ AIRFLOW 1.8.0, 2017-03-12 [AIRFLOW-784] Pin funcsigs to 1.0.0 [AIRFLOW-624] Fix setup.py to not import airflow.version as version [AIRFLOW-779] Task should fail with specific message when deleted -[AIRFLOW-778] Fix completey broken MetastorePartitionSensor +[AIRFLOW-778] Fix completely broken MetastorePartitionSensor [AIRFLOW-739] Set pickle_info log to debug [AIRFLOW-771] Make S3 logs append instead of clobber [AIRFLOW-773] Fix flaky datetime addition in api test diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 556a5d847b..2a60f1dc3c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -166,10 +166,10 @@ There are three ways to setup an Apache Airflow development environment. tox -e py35-backend_mysql ``` - If you wish to run individual tests inside of docker enviroment you can do as follows: + If you wish to run individual tests inside of Docker environment you can do as follows: ```bash -# From the container (with your desired enviroment) with druid hook +# From the container (with your desired environment) with druid hook tox -e py35-backend_mysql -- tests/hooks/test_druid_hook.py ``` diff --git a/airflow/contrib/hooks/bigquery_hook.py b/airflow/contrib/hooks/bigquery_hook.py index 5cab013b28..30a16305db 100644 --- a/airflow/contrib/hooks/bigquery_hook.py +++ b/airflow/contrib/hooks/bigquery_hook.py @@ -1594,7 +1594,7 @@ def insert_all(self, project_id, dataset_id, table_id, self.log.info('All row(s) inserted successfully: {}:{}.{}'.format( dataset_project_id, dataset_id, table_id)) else: -error_msg = '{} insert error(s) occured: {}:{}.{}. Details: {}'.format( +error_msg = '{} insert error(s) occurred: {}:{}.{}. Details: {}'.format( len(resp['insertErrors']), dataset_project_id, dataset_id, table_id, resp['insertErrors']) if fail_on_error: diff --git a/airflow/contrib/hooks/emr_hook.py b/airflow/contrib/hooks/emr_hook.py index f9fd3f04de..fcdf4ac848 100644 --- a/airflow/contrib/hooks/emr_hook.py +++ b/airflow/contrib/hooks/emr_hook.py @@ -23,7 +23,7 @@ class EmrHook(AwsHook): """ -Interact with AWS EMR. emr_conn_id is only neccessary for using the +Interact with AWS EMR. emr_conn_id is only necessary for using the create_job_flow method. """ diff --git a/airflow/executors/celery_executor.py b/airflow/executors/celery_executor.py index 98ce6efba7..10694ea4b7 100644 --- a/airflow/executors/celery_executor.py +++ b/airflow/executors/celery_executor.py @@ -74,7 +74,7 @@ def execute_command(command_to_exec): class ExceptionWithTraceback(object): """ -Wrapper class used to propogate exceptions to parent processes from subprocesses. +Wrapper class used to propagate exceptions to parent processes from subprocesses. :param exception: The exception to wrap :type exception: Exception :param traceback: The stacktrace to wrap diff --git a/airflow/sensors/base_sensor_operator.py b/airflow/sensors/base_sensor_operator.py index 1dc59dd230..eaa6306770 100644 --- a/airflow/sensors/base_sensor_operator.py +++ b/airflow/sensors/base_sensor_operator.py @@ -48,7 +48,7 @@ class BaseSensorOperator(BaseOperator, SkipMixin): When set to ``poke`` the
[jira] [Commented] (AIRFLOW-3150) Make execution_date a template field in TriggerDagRunOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16727510#comment-16727510 ] ASF GitHub Bot commented on AIRFLOW-3150: - kaxil opened a new pull request #4359: [AIRFLOW-3150] Make execution_date templated in TriggerDagRunOperator URL: https://github.com/apache/incubator-airflow/pull/4359 Make sure you have checked _all_ steps below. ### Jira - [x] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-3150 ### Description - [x] Here are some details about my PR, including screenshots of any UI changes: ### Tests - [x] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: * `test_trigger_dagrun_with_str_execution_date` ### Commits - [x] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [x] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. - All the public functions and the classes in the PR contain docstrings that explain what it does ### Code Quality - [x] Passes `flake8` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Make execution_date a template field in TriggerDagRunOperator > - > > Key: AIRFLOW-3150 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3150 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Reporter: Kyle Hamlin >Assignee: Kaxil Naik >Priority: Minor > Labels: easy-fix > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] kaxil opened a new pull request #4359: [AIRFLOW-3150] Make execution_date templated in TriggerDagRunOperator
kaxil opened a new pull request #4359: [AIRFLOW-3150] Make execution_date templated in TriggerDagRunOperator URL: https://github.com/apache/incubator-airflow/pull/4359 Make sure you have checked _all_ steps below. ### Jira - [x] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-3150 ### Description - [x] Here are some details about my PR, including screenshots of any UI changes: ### Tests - [x] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: * `test_trigger_dagrun_with_str_execution_date` ### Commits - [x] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [x] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. - All the public functions and the classes in the PR contain docstrings that explain what it does ### Code Quality - [x] Passes `flake8` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] stale[bot] commented on issue #4127: Bug Fix: Secrets object and key separated by ":"
stale[bot] commented on issue #4127: Bug Fix: Secrets object and key separated by ":" URL: https://github.com/apache/incubator-airflow/pull/4127#issuecomment-449581166 This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] XD-DENG commented on issue #4355: [AIRFLOW-XXX] Update log level and return value
XD-DENG commented on issue #4355: [AIRFLOW-XXX] Update log level and return value URL: https://github.com/apache/incubator-airflow/pull/4355#issuecomment-449577116 A JIRA ticket would be needed, given this is a code change. You can get the details in the original PR template. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4357: [AIRFLOW-3557] Fix various typos
codecov-io edited a comment on issue #4357: [AIRFLOW-3557] Fix various typos URL: https://github.com/apache/incubator-airflow/pull/4357#issuecomment-449563258 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4357?src=pr=h1) Report > Merging [#4357](https://codecov.io/gh/apache/incubator-airflow/pull/4357?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/701b877a2eb4aac151a643f6eb735abf303a4e5e?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-airflow/pull/4357/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4357?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#4357 +/- ## === Coverage 78.12% 78.12% === Files 202 202 Lines 1648316483 === Hits1287812878 Misses 3605 3605 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4357?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/sensors/base\_sensor\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4357/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL2Jhc2Vfc2Vuc29yX29wZXJhdG9yLnB5) | `97.87% <ø> (ø)` | :arrow_up: | | [airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/incubator-airflow/pull/4357/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5) | `76.92% <ø> (ø)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4357?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4357?src=pr=footer). Last update [701b877...8badf67](https://codecov.io/gh/apache/incubator-airflow/pull/4357?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #4357: [AIRFLOW-3557] Fix various typos
codecov-io commented on issue #4357: [AIRFLOW-3557] Fix various typos URL: https://github.com/apache/incubator-airflow/pull/4357#issuecomment-449563258 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4357?src=pr=h1) Report > Merging [#4357](https://codecov.io/gh/apache/incubator-airflow/pull/4357?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/701b877a2eb4aac151a643f6eb735abf303a4e5e?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-airflow/pull/4357/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4357?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#4357 +/- ## === Coverage 78.12% 78.12% === Files 202 202 Lines 1648316483 === Hits1287812878 Misses 3605 3605 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4357?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/sensors/base\_sensor\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4357/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL2Jhc2Vfc2Vuc29yX29wZXJhdG9yLnB5) | `97.87% <ø> (ø)` | :arrow_up: | | [airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/incubator-airflow/pull/4357/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5) | `76.92% <ø> (ø)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4357?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4357?src=pr=footer). Last update [701b877...8badf67](https://codecov.io/gh/apache/incubator-airflow/pull/4357?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4357: [AIRFLOW-3557] Fix various typos
codecov-io edited a comment on issue #4357: [AIRFLOW-3557] Fix various typos URL: https://github.com/apache/incubator-airflow/pull/4357#issuecomment-449563258 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4357?src=pr=h1) Report > Merging [#4357](https://codecov.io/gh/apache/incubator-airflow/pull/4357?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/701b877a2eb4aac151a643f6eb735abf303a4e5e?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-airflow/pull/4357/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4357?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#4357 +/- ## === Coverage 78.12% 78.12% === Files 202 202 Lines 1648316483 === Hits1287812878 Misses 3605 3605 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4357?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/sensors/base\_sensor\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4357/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL2Jhc2Vfc2Vuc29yX29wZXJhdG9yLnB5) | `97.87% <ø> (ø)` | :arrow_up: | | [airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/incubator-airflow/pull/4357/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5) | `76.92% <ø> (ø)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4357?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4357?src=pr=footer). Last update [701b877...8badf67](https://codecov.io/gh/apache/incubator-airflow/pull/4357?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #4358: [AIRFLOW-XXX] Correct instructions for Travis CI on fork
codecov-io commented on issue #4358: [AIRFLOW-XXX] Correct instructions for Travis CI on fork URL: https://github.com/apache/incubator-airflow/pull/4358#issuecomment-449563026 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4358?src=pr=h1) Report > Merging [#4358](https://codecov.io/gh/apache/incubator-airflow/pull/4358?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/701b877a2eb4aac151a643f6eb735abf303a4e5e?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-airflow/pull/4358/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4358?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#4358 +/- ## === Coverage 78.12% 78.12% === Files 202 202 Lines 1648316483 === Hits1287812878 Misses 3605 3605 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4358?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4358?src=pr=footer). Last update [701b877...b4aef93](https://codecov.io/gh/apache/incubator-airflow/pull/4358?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4358: [AIRFLOW-XXX] Correct instructions for Travis CI on fork
codecov-io edited a comment on issue #4358: [AIRFLOW-XXX] Correct instructions for Travis CI on fork URL: https://github.com/apache/incubator-airflow/pull/4358#issuecomment-449563026 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4358?src=pr=h1) Report > Merging [#4358](https://codecov.io/gh/apache/incubator-airflow/pull/4358?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/701b877a2eb4aac151a643f6eb735abf303a4e5e?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-airflow/pull/4358/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4358?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#4358 +/- ## === Coverage 78.12% 78.12% === Files 202 202 Lines 1648316483 === Hits1287812878 Misses 3605 3605 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4358?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4358?src=pr=footer). Last update [701b877...b4aef93](https://codecov.io/gh/apache/incubator-airflow/pull/4358?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] BasPH opened a new pull request #4358: [AIRFLOW-XXX] Correct instructions for Travis CI on fork
BasPH opened a new pull request #4358: [AIRFLOW-XXX] Correct instructions for Travis CI on fork URL: https://github.com/apache/incubator-airflow/pull/4358 Make sure you have checked _all_ steps below. ### Jira - [x] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-XXX - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. AIRFLOW-XXX ### Description - [x] Here are some details about my PR, including screenshots of any UI changes: The instructions in CONTRIBUTING.md pointed to somebodies private Travis CI setup, this PR corrects that. ### Tests - [x] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: No tests required. ### Commits - [x] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [x] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. - All the public functions and the classes in the PR contain docstrings that explain what it does ### Code Quality - [x] Passes `flake8` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4356: [AIRFLOW-3556] Add cross join set downstream function
codecov-io edited a comment on issue #4356: [AIRFLOW-3556] Add cross join set downstream function URL: https://github.com/apache/incubator-airflow/pull/4356#issuecomment-449515658 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4356?src=pr=h1) Report > Merging [#4356](https://codecov.io/gh/apache/incubator-airflow/pull/4356?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/f2113906d052df818cf14be8c513fada91caf26a?src=pr=desc) will **increase** coverage by `62%`. > The diff coverage is `100%`. [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-airflow/pull/4356/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4356?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4356 +/- ## == + Coverage 16.13% 78.14% +62% == Files 202 202 Lines 1648316486 +3 == + Hits 266012883 +10223 + Misses 13823 3603 -10220 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4356?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/helpers.py](https://codecov.io/gh/apache/incubator-airflow/pull/4356/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9oZWxwZXJzLnB5) | `82.6% <100%> (+55.94%)` | :arrow_up: | | [airflow/plugins\_manager.py](https://codecov.io/gh/apache/incubator-airflow/pull/4356/diff?src=pr=tree#diff-YWlyZmxvdy9wbHVnaW5zX21hbmFnZXIucHk=) | `93.1% <0%> (+1.14%)` | :arrow_up: | | [airflow/executors/dask\_executor.py](https://codecov.io/gh/apache/incubator-airflow/pull/4356/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvZGFza19leGVjdXRvci5weQ==) | `2% <0%> (+2%)` | :arrow_up: | | [airflow/exceptions.py](https://codecov.io/gh/apache/incubator-airflow/pull/4356/diff?src=pr=tree#diff-YWlyZmxvdy9leGNlcHRpb25zLnB5) | `100% <0%> (+2.85%)` | :arrow_up: | | [airflow/utils/operator\_resources.py](https://codecov.io/gh/apache/incubator-airflow/pull/4356/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9vcGVyYXRvcl9yZXNvdXJjZXMucHk=) | `86.95% <0%> (+4.34%)` | :arrow_up: | | [airflow/executors/\_\_init\_\_.py](https://codecov.io/gh/apache/incubator-airflow/pull/4356/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvX19pbml0X18ucHk=) | `63.46% <0%> (+5.76%)` | :arrow_up: | | [airflow/utils/module\_loading.py](https://codecov.io/gh/apache/incubator-airflow/pull/4356/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9tb2R1bGVfbG9hZGluZy5weQ==) | `100% <0%> (+10.52%)` | :arrow_up: | | [airflow/utils/decorators.py](https://codecov.io/gh/apache/incubator-airflow/pull/4356/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kZWNvcmF0b3JzLnB5) | `91.66% <0%> (+14.58%)` | :arrow_up: | | [airflow/settings.py](https://codecov.io/gh/apache/incubator-airflow/pull/4356/diff?src=pr=tree#diff-YWlyZmxvdy9zZXR0aW5ncy5weQ==) | `80.41% <0%> (+14.68%)` | :arrow_up: | | [airflow/hooks/oracle\_hook.py](https://codecov.io/gh/apache/incubator-airflow/pull/4356/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9vcmFjbGVfaG9vay5weQ==) | `15.47% <0%> (+15.47%)` | :arrow_up: | | ... and [162 more](https://codecov.io/gh/apache/incubator-airflow/pull/4356/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4356?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4356?src=pr=footer). Last update [f211390...0bbf833](https://codecov.io/gh/apache/incubator-airflow/pull/4356?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3556) Add a "cross join" function for setting dependencies between two lists of tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-3556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16727313#comment-16727313 ] ASF GitHub Bot commented on AIRFLOW-3556: - BasPH opened a new pull request #4356: [AIRFLOW-3556] Add cross join set downstream function URL: https://github.com/apache/incubator-airflow/pull/4356 Make sure you have checked _all_ steps below. ### Jira - [x] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-3556 - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. ### Description - [x] Here are some details about my PR, including screenshots of any UI changes: Add function to set "cross join style" downstream dependencies between two list of tasks. For example: ``` cross_downstream(from_tasks=[t1, t2, t3], to_tasks=[t4, t5, t6]) Sets dependencies: t1 --> t4 \ / t2 -X> t5 / \ t3 --> t6 Equivalent to: t1.set_downstream(t4) t1.set_downstream(t5) t1.set_downstream(t6) t2.set_downstream(t4) t2.set_downstream(t5) t2.set_downstream(t6) t3.set_downstream(t4) t3.set_downstream(t5) t3.set_downstream(t6) ``` ### Tests - [x] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: HelpersTest.test_cross_downstream() ### Commits - [x] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [x] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. - All the public functions and the classes in the PR contain docstrings that explain what it does ### Code Quality - [x] Passes `flake8` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add a "cross join" function for setting dependencies between two lists of > tasks > --- > > Key: AIRFLOW-3556 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3556 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Bas Harenslak >Priority: Major > > Similar to airflow.utils.helpers.chain(), it would be useful to have a helper > function that sets downstream dependencies in a cross join fashion between > two lists of tasks. > For example: > {code} > cross_downstream(from_tasks=[t1, t2, t3], to_tasks=[t4, t5, t6]) > Sets dependencies: > t1 --> t4 >\ / > t2 -X> t5 >/ \ > t3 --> t6 > Equivalent to: > t1.set_downstream(t4) > t1.set_downstream(t5) > t1.set_downstream(t6) > t2.set_downstream(t4) > t2.set_downstream(t5) > t2.set_downstream(t6) > t3.set_downstream(t4) > t3.set_downstream(t5) > t3.set_downstream(t6){code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] BasPH opened a new pull request #4356: [AIRFLOW-3556] Add cross join set downstream function
BasPH opened a new pull request #4356: [AIRFLOW-3556] Add cross join set downstream function URL: https://github.com/apache/incubator-airflow/pull/4356 Make sure you have checked _all_ steps below. ### Jira - [x] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-3556 - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. ### Description - [x] Here are some details about my PR, including screenshots of any UI changes: Add function to set "cross join style" downstream dependencies between two list of tasks. For example: ``` cross_downstream(from_tasks=[t1, t2, t3], to_tasks=[t4, t5, t6]) Sets dependencies: t1 --> t4 \ / t2 -X> t5 / \ t3 --> t6 Equivalent to: t1.set_downstream(t4) t1.set_downstream(t5) t1.set_downstream(t6) t2.set_downstream(t4) t2.set_downstream(t5) t2.set_downstream(t6) t3.set_downstream(t4) t3.set_downstream(t5) t3.set_downstream(t6) ``` ### Tests - [x] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: HelpersTest.test_cross_downstream() ### Commits - [x] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [x] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. - All the public functions and the classes in the PR contain docstrings that explain what it does ### Code Quality - [x] Passes `flake8` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3557) Various typos
[ https://issues.apache.org/jira/browse/AIRFLOW-3557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16727311#comment-16727311 ] ASF GitHub Bot commented on AIRFLOW-3557: - BasPH opened a new pull request #4357: [AIRFLOW-3557] Fix various typos URL: https://github.com/apache/incubator-airflow/pull/4357 Make sure you have checked _all_ steps below. ### Jira - [x] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-3557 - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. ### Description - [x] Here are some details about my PR, including screenshots of any UI changes: Check source code with [misspell](https://github.com/client9/misspell) and fixed various typos. ### Tests - [x] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: No changes to code. ### Commits - [x] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [x] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. - All the public functions and the classes in the PR contain docstrings that explain what it does No new functionality. ### Code Quality - [x] Passes `flake8` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Various typos > - > > Key: AIRFLOW-3557 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3557 > Project: Apache Airflow > Issue Type: Bug >Reporter: Bas Harenslak >Priority: Major > > Fix various typos, checked with > [misspell|https://github.com/client9/misspell]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] BasPH opened a new pull request #4357: [AIRFLOW-3557] Fix various typos
BasPH opened a new pull request #4357: [AIRFLOW-3557] Fix various typos URL: https://github.com/apache/incubator-airflow/pull/4357 Make sure you have checked _all_ steps below. ### Jira - [x] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-3557 - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. ### Description - [x] Here are some details about my PR, including screenshots of any UI changes: Check source code with [misspell](https://github.com/client9/misspell) and fixed various typos. ### Tests - [x] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: No changes to code. ### Commits - [x] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [x] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. - All the public functions and the classes in the PR contain docstrings that explain what it does No new functionality. ### Code Quality - [x] Passes `flake8` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3556) Add a "cross join" function for setting dependencies between two lists of tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-3556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16727310#comment-16727310 ] ASF GitHub Bot commented on AIRFLOW-3556: - BasPH closed pull request #4356: [AIRFLOW-3556] Add cross join set downstream function URL: https://github.com/apache/incubator-airflow/pull/4356 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/airflow/utils/helpers.py b/airflow/utils/helpers.py index 328147c1cf..5f8c88879c 100644 --- a/airflow/utils/helpers.py +++ b/airflow/utils/helpers.py @@ -169,6 +169,37 @@ def chain(*tasks): up_task.set_downstream(down_task) +def cross_downstream(from_tasks, to_tasks): +""" +Set downstream dependencies for all tasks in from_tasks to all tasks in to_tasks. +E.g.: cross_downstream(from_tasks=[t1, t2, t3], to_tasks=[t4, t5, t6]) +Is equivalent to: + +t1 --> t4 + \ / +t2 -X> t5 + / \ +t3 --> t6 + +t1.set_downstream(t4) +t1.set_downstream(t5) +t1.set_downstream(t6) +t2.set_downstream(t4) +t2.set_downstream(t5) +t2.set_downstream(t6) +t3.set_downstream(t4) +t3.set_downstream(t5) +t3.set_downstream(t6) + +:param from_tasks: List of tasks to start from. +:type from_tasks: List[airflow.models.BaseOperator] +:param to_tasks: List of tasks to set as downstream dependencies. +:type to_tasks: List[airflow.models.BaseOperator] +""" +for task in from_tasks: +task.set_downstream(to_tasks) + + def pprinttable(rows): """Returns a pretty ascii table from tuples diff --git a/tests/utils/test_helpers.py b/tests/utils/test_helpers.py index 4cb3e1a1fc..837a79acba 100644 --- a/tests/utils/test_helpers.py +++ b/tests/utils/test_helpers.py @@ -20,11 +20,16 @@ import logging import multiprocessing import os -import psutil import signal import time import unittest +from datetime import datetime + +import psutil +import six +from airflow import DAG +from airflow.operators.dummy_operator import DummyOperator from airflow.utils import helpers @@ -210,6 +215,16 @@ def test_is_container(self): # Pass an object that is not iter nor a string. self.assertFalse(helpers.is_container(10)) +def test_cross_downstream(self): +"""Test if all dependencies between tasks are all set correctly.""" +dag = DAG(dag_id="test_dag", start_date=datetime.now()) +start_tasks = [DummyOperator(task_id="t{i}".format(i=i), dag=dag) for i in range(1, 4)] +end_tasks = [DummyOperator(task_id="t{i}".format(i=i), dag=dag) for i in range(4, 7)] +helpers.cross_downstream(from_tasks=start_tasks, to_tasks=end_tasks) + +for start_task in start_tasks: +six.assertCountEqual(self, start_task.get_direct_relatives(upstream=False), end_tasks) + if __name__ == '__main__': unittest.main() This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add a "cross join" function for setting dependencies between two lists of > tasks > --- > > Key: AIRFLOW-3556 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3556 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Bas Harenslak >Priority: Major > > Similar to airflow.utils.helpers.chain(), it would be useful to have a helper > function that sets downstream dependencies in a cross join fashion between > two lists of tasks. > For example: > {code} > cross_downstream(from_tasks=[t1, t2, t3], to_tasks=[t4, t5, t6]) > Sets dependencies: > t1 --> t4 >\ / > t2 -X> t5 >/ \ > t3 --> t6 > Equivalent to: > t1.set_downstream(t4) > t1.set_downstream(t5) > t1.set_downstream(t6) > t2.set_downstream(t4) > t2.set_downstream(t5) > t2.set_downstream(t6) > t3.set_downstream(t4) > t3.set_downstream(t5) > t3.set_downstream(t6){code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] BasPH closed pull request #4356: [AIRFLOW-3556] Add cross join set downstream function
BasPH closed pull request #4356: [AIRFLOW-3556] Add cross join set downstream function URL: https://github.com/apache/incubator-airflow/pull/4356 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/airflow/utils/helpers.py b/airflow/utils/helpers.py index 328147c1cf..5f8c88879c 100644 --- a/airflow/utils/helpers.py +++ b/airflow/utils/helpers.py @@ -169,6 +169,37 @@ def chain(*tasks): up_task.set_downstream(down_task) +def cross_downstream(from_tasks, to_tasks): +""" +Set downstream dependencies for all tasks in from_tasks to all tasks in to_tasks. +E.g.: cross_downstream(from_tasks=[t1, t2, t3], to_tasks=[t4, t5, t6]) +Is equivalent to: + +t1 --> t4 + \ / +t2 -X> t5 + / \ +t3 --> t6 + +t1.set_downstream(t4) +t1.set_downstream(t5) +t1.set_downstream(t6) +t2.set_downstream(t4) +t2.set_downstream(t5) +t2.set_downstream(t6) +t3.set_downstream(t4) +t3.set_downstream(t5) +t3.set_downstream(t6) + +:param from_tasks: List of tasks to start from. +:type from_tasks: List[airflow.models.BaseOperator] +:param to_tasks: List of tasks to set as downstream dependencies. +:type to_tasks: List[airflow.models.BaseOperator] +""" +for task in from_tasks: +task.set_downstream(to_tasks) + + def pprinttable(rows): """Returns a pretty ascii table from tuples diff --git a/tests/utils/test_helpers.py b/tests/utils/test_helpers.py index 4cb3e1a1fc..837a79acba 100644 --- a/tests/utils/test_helpers.py +++ b/tests/utils/test_helpers.py @@ -20,11 +20,16 @@ import logging import multiprocessing import os -import psutil import signal import time import unittest +from datetime import datetime + +import psutil +import six +from airflow import DAG +from airflow.operators.dummy_operator import DummyOperator from airflow.utils import helpers @@ -210,6 +215,16 @@ def test_is_container(self): # Pass an object that is not iter nor a string. self.assertFalse(helpers.is_container(10)) +def test_cross_downstream(self): +"""Test if all dependencies between tasks are all set correctly.""" +dag = DAG(dag_id="test_dag", start_date=datetime.now()) +start_tasks = [DummyOperator(task_id="t{i}".format(i=i), dag=dag) for i in range(1, 4)] +end_tasks = [DummyOperator(task_id="t{i}".format(i=i), dag=dag) for i in range(4, 7)] +helpers.cross_downstream(from_tasks=start_tasks, to_tasks=end_tasks) + +for start_task in start_tasks: +six.assertCountEqual(self, start_task.get_direct_relatives(upstream=False), end_tasks) + if __name__ == '__main__': unittest.main() This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-3557) Various typos
Bas Harenslak created AIRFLOW-3557: -- Summary: Various typos Key: AIRFLOW-3557 URL: https://issues.apache.org/jira/browse/AIRFLOW-3557 Project: Apache Airflow Issue Type: Bug Reporter: Bas Harenslak Fix various typos, checked with [misspell|https://github.com/client9/misspell]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)