[GitHub] KevinYang21 commented on issue #3873: [Airflow-2760] Decouple DAG parsing loop from scheduler loop
KevinYang21 commented on issue #3873: [Airflow-2760] Decouple DAG parsing loop from scheduler loop URL: https://github.com/apache/incubator-airflow/pull/3873#issuecomment-425301416 Updated docstring in DagFileProcessor[Agent|Manager] with more details and added docstring for method `execute_helper()` with details on how the scheduler loop works and link to the graphic representation. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] akosel commented on issue #3958: [AIRFLOW-3124] Fix RBAC webserver debug mode
akosel commented on issue #3958: [AIRFLOW-3124] Fix RBAC webserver debug mode URL: https://github.com/apache/incubator-airflow/pull/3958#issuecomment-425282532 > LGTM, +1 on @ashb 's feedback. Do you want to check whether the CI failure is false positive? It looks like CI is currently in a bad state and @kaxil is currently running some tests to get it fixed. So the failures look unrelated, but we can wait until that is fixed up to be sure. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-airflow-ci] branch kaxil-fix deleted (was 2e3fbff)
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a change to branch kaxil-fix in repository https://gitbox.apache.org/repos/asf/incubator-airflow-ci.git. was 2e3fbff Remove requirements This change permanently discards the following revisions: discard 2e3fbff Remove requirements
[GitHub] kaxil closed pull request #3965: [DoNotMerge] Testing CI
kaxil closed pull request #3965: [DoNotMerge] Testing CI URL: https://github.com/apache/incubator-airflow/pull/3965 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/scripts/ci/docker-compose.yml b/scripts/ci/docker-compose.yml index 101ad95297..4b5f409358 100644 --- a/scripts/ci/docker-compose.yml +++ b/scripts/ci/docker-compose.yml @@ -62,11 +62,10 @@ services: domainname: example.com airflow-testing: -image: airflowci/incubator-airflow-ci:latest +image: airflowci/incubator-airflow-ci:kaxil-fix init: true environment: - USER=airflow - - ADDITIONAL_PATH=~/.local/bin - SLUGIFY_USES_TEXT_UNIDECODE=yes - TOX_ENV - PYTHON_VERSION diff --git a/scripts/ci/run-ci.sh b/scripts/ci/run-ci.sh index f16ab5284d..1a65bf1ef6 100755 --- a/scripts/ci/run-ci.sh +++ b/scripts/ci/run-ci.sh @@ -32,8 +32,8 @@ else PIP=pip fi -sudo -H $PIP install --upgrade pip -sudo -H $PIP install tox +sudo $PIP install --upgrade pip +sudo $PIP install tox cd $AIRFLOW_ROOT && $PIP --version && tox --version diff --git a/setup.py b/setup.py index 1d7885f88c..39015c01a9 100644 --- a/setup.py +++ b/setup.py @@ -226,7 +226,7 @@ def write_version(filename=os.path.join(*['airflow', sendgrid = ['sendgrid>=5.2.0'] slack = ['slackclient>=1.0.0'] mongo = ['pymongo>=3.6.0'] -snowflake = ['snowflake-connector-python>=1.5.2', +snowflake = ['snowflake-connector-python==1.6.9', 'snowflake-sqlalchemy>=1.1.0'] ssh = ['paramiko>=2.1.1', 'pysftp>=0.2.9', 'sshtunnel>=0.1.4,<0.2'] statsd = ['statsd>=3.0.1, <4.0'] This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] feng-tao commented on issue #3958: [AIRFLOW-3124] Fix RBAC webserver debug mode
feng-tao commented on issue #3958: [AIRFLOW-3124] Fix RBAC webserver debug mode URL: https://github.com/apache/incubator-airflow/pull/3958#issuecomment-425277282 LGTM, +1 on @ashb 's feedback. Do you want to check whether the CI failure is false positive? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] feng-tao commented on issue #3966: [AIRFLOW-3125] Monitor Task Instances creation rates
feng-tao commented on issue #3966: [AIRFLOW-3125] Monitor Task Instances creation rates URL: https://github.com/apache/incubator-airflow/pull/3966#issuecomment-425277014 LGTM. Will wait for the CI. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mingyexia opened a new pull request #3966: [AIRFLOW-3125] Monitor Task Instances creation rates
mingyexia opened a new pull request #3966: [AIRFLOW-3125] Monitor Task
Instances creation rates
URL: https://github.com/apache/incubator-airflow/pull/3966
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following
[AIRFLOW-3125](https://issues.apache.org/jira/browse/AIRFLOW-3125)
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
Montor Task Instances creation rates by Operator type.
These stats can provide some visibility on how much workload Airflow is
getting. They can be used for resource allocation in the long run (i.e. to
determine when we should scale up workers) and debugging in scenarios like the
creation rate of certain type of Task Instances spikes.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Just adding stats.
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-3125) Add monitoring on Task Instance creation rate
[
https://issues.apache.org/jira/browse/AIRFLOW-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631191#comment-16631191
]
ASF GitHub Bot commented on AIRFLOW-3125:
-
mingyexia opened a new pull request #3966: [AIRFLOW-3125] Monitor Task
Instances creation rates
URL: https://github.com/apache/incubator-airflow/pull/3966
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following
[AIRFLOW-3125](https://issues.apache.org/jira/browse/AIRFLOW-3125)
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
Montor Task Instances creation rates by Operator type.
These stats can provide some visibility on how much workload Airflow is
getting. They can be used for resource allocation in the long run (i.e. to
determine when we should scale up workers) and debugging in scenarios like the
creation rate of certain type of Task Instances spikes.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Just adding stats.
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add monitoring on Task Instance creation rate
> -
>
> Key: AIRFLOW-3125
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3125
> Project: Apache Airflow
> Issue Type: Improvement
>Reporter: Mingye Xia
>Assignee: Mingye Xia
>Priority: Major
>
> Monitoring on Task Instance creation rate can give us some visibility on how
> much workload we are putting on Airflow. It can be used for resource
> allocation in the long run (i.e. to determine when we should scale up
> workers) and and debugging in scenarios like creation rate for certain types
> of Task Instances spike.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] kaxil opened a new pull request #3965: [DoNotMerge] Testing CI
kaxil opened a new pull request #3965: [DoNotMerge] Testing CI
URL: https://github.com/apache/incubator-airflow/pull/3965
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-3079) initdb fails on Microsoft SQL Server
[
https://issues.apache.org/jira/browse/AIRFLOW-3079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631168#comment-16631168
]
ASF GitHub Bot commented on AIRFLOW-3079:
-
BradHolmes opened a new pull request #3964: [AIRFLOW-3079] Improve migration
scripts to support MSSQL Server
URL: https://github.com/apache/incubator-airflow/pull/3964
There were two problems for MSSQL. First, 'timestamp' data type in MSSQL
Server is essentially a row-id,
and not a timezone enabled date/time stamp. Second, alembic creates invalid
SQL when applying the 0/1 constraint
to boolean values. MSSQL should enforce this constrait by simply asserting
a boolean value.
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses
[AIRFLOW-3079](https://issues.apache.org/jira/browse/AIRFLOW-3079)
### Description
- [x] See commit message.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
* No tests added as I only adjusted the migration scripts to work for
MSSQL.
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
NONE
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> initdb fails on Microsoft SQL Server
>
>
> Key: AIRFLOW-3079
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3079
> Project: Apache Airflow
> Issue Type: Bug
> Components: database
>Affects Versions: 1.10.0
>Reporter: Morten Post
>Priority: Major
>
> airflow initdb fails using Microsoft SQL Server 17 backend. Problem does not
> exist in 1.9.0.
> [*@ airflow]$ airflow initdb
> [2018-09-17 14:08:28,744] \{settings.py:174} INFO - setting.configure_orm():
> Using pool settings. pool_size=5, pool_recycle=1800
> [2018-09-17 14:08:28,865] \{__init__.py:51} INFO - Using executor
> SequentialExecutor
> DB: DB: mssql+pyodbc://***/Airflow?driver=ODBC Driver 17 for SQL
> Server
> [2018-09-17 14:08:28,967] \{db.py:338} INFO - Creating tables
> INFO [alembic.runtime.migration] Context impl MSSQLImpl.
> INFO [alembic.runtime.migration] Will assume transactional DDL.
> INFO [alembic.runtime.migration] Running upgrade -> e3a246e0dc1, current
> schema
> INFO [alembic.runtime.migration] Running upgrade e3a246e0dc1 -> 1507a7289a2f,
> create is_encrypted
> INFO [alembic.runtime.migration] Running upgrade 1507a7289a2f ->
> 13eb55f81627, maintain history for compatibility with earlier migrations
> INFO [alembic.runtime.migration] Running upgrade 13eb55f81627 ->
> 338e90f54d61, More logging into task_isntance
> INFO [alembic.runtime.migration] Running upgrade 338e90f54d61 -> 52d714495f0,
> job_id indices
> INFO [alembic.runtime.migration] Running upgrade 52d714495f0 -> 502898887f84,
> Adding extra to Log
> INFO [alembic.runtime.migration] Running upgrade 502898887f84 ->
> 1b38cef5b76e, add dagrun
> INFO [alembic.runtime.migration] Running upgrade 1b38cef5b76e ->
> 2e541a1dcfed, task_duration
> INFO [alembic.runtime.migration] Running upgrade 2e541a1dcfed ->
> 40e67319e3a9, dagrun_config
> INFO [alembic.runtime.migration] Running upgrade 40e67319e3a9 ->
> 561833c1c74b, add password column to user
> INFO [alembic.runtime.migration] Running upgrade 561833c1c74b -> 4446e08588,
> dagrun start end
> INFO [alembic.runtime.migration] Running upgrade 4446e08588 -> bbc73705a13e,
> Add notification_sent column to sla_miss
> INFO [alembic.runtime.migration] Running upgrade bbc73705a13e ->
> bba5a7cfc896, Add a column to track the encryption state of the 'Extra' field
> in connection
> INFO [alembic.runtime.migration] Running upgrade
[GitHub] BradHolmes opened a new pull request #3964: [AIRFLOW-3079] Improve migration scripts to support MSSQL Server
BradHolmes opened a new pull request #3964: [AIRFLOW-3079] Improve migration
scripts to support MSSQL Server
URL: https://github.com/apache/incubator-airflow/pull/3964
There were two problems for MSSQL. First, 'timestamp' data type in MSSQL
Server is essentially a row-id,
and not a timezone enabled date/time stamp. Second, alembic creates invalid
SQL when applying the 0/1 constraint
to boolean values. MSSQL should enforce this constrait by simply asserting
a boolean value.
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses
[AIRFLOW-3079](https://issues.apache.org/jira/browse/AIRFLOW-3079)
### Description
- [x] See commit message.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
* No tests added as I only adjusted the migration scripts to work for
MSSQL.
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
NONE
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[incubator-airflow-ci] branch kaxil-fix created (now 2e3fbff)
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a change to branch kaxil-fix in repository https://gitbox.apache.org/repos/asf/incubator-airflow-ci.git. at 2e3fbff Remove requirements This branch includes the following new commits: new 2e3fbff Remove requirements The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[incubator-airflow-ci] 01/01: Remove requirements
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a commit to branch kaxil-fix in repository https://gitbox.apache.org/repos/asf/incubator-airflow-ci.git commit 2e3fbff3489085d6cb4973aa339acf4faf523fcc Author: Kaxil Naik AuthorDate: Fri Sep 28 00:07:42 2018 +0100 Remove requirements --- Dockerfile | 9 + 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/Dockerfile b/Dockerfile index 4ea5ea6..eb491ac 100644 --- a/Dockerfile +++ b/Dockerfile @@ -116,17 +116,10 @@ RUN adduser airflow && \ echo "airflow ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/airflow && \ chmod 0440 /etc/sudoers.d/airflow -# Install Python requirements -RUN sudo -H pip install --upgrade pip && \ -sudo -H pip install wheel tox && \ -sudo -H pip3 install --upgrade pip && \ -sudo -H pip3 install wheel tox && \ -rm -rf ~/.cache - EXPOSE 8080 WORKDIR /home/airflow -ENV PATH "$PATH:/tmp/hive/bin:$ADDITIONAL_PATH" +ENV PATH "$PATH:/tmp/hive/bin" USER airflow
[GitHub] kaxil commented on issue #3957: [AIRFLOW-2952] Fix Kubernetes CI
kaxil commented on issue #3957: [AIRFLOW-2952] Fix Kubernetes CI URL: https://github.com/apache/incubator-airflow/pull/3957#issuecomment-425269121 @r39132 That what we are trying it diagnose. More details in this PR: https://github.com/apache/incubator-airflow-ci/pull/3#issuecomment-425074580 This is not related to this PR. But we need to sort this issue out. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] r39132 commented on issue #3957: [AIRFLOW-2952] Fix Kubernetes CI
r39132 commented on issue #3957: [AIRFLOW-2952] Fix Kubernetes CI URL: https://github.com/apache/incubator-airflow/pull/3957#issuecomment-425268509 Why are the 2.7 docker (non-kub) tests failiing? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] r39132 edited a comment on issue #3957: [AIRFLOW-2952] Fix Kubernetes CI
r39132 edited a comment on issue #3957: [AIRFLOW-2952] Fix Kubernetes CI URL: https://github.com/apache/incubator-airflow/pull/3957#issuecomment-425268509 Why are the 2.7 docker (non-k8) tests failiing? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] r39132 edited a comment on issue #3957: [AIRFLOW-2952] Fix Kubernetes CI
r39132 edited a comment on issue #3957: [AIRFLOW-2952] Fix Kubernetes CI URL: https://github.com/apache/incubator-airflow/pull/3957#issuecomment-425268509 Why are the 2.7 docker (non-k8) tests failing? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-3126) kubernetes executor not allowing new PV
Rahul Singh created AIRFLOW-3126: Summary: kubernetes executor not allowing new PV Key: AIRFLOW-3126 URL: https://issues.apache.org/jira/browse/AIRFLOW-3126 Project: Apache Airflow Issue Type: Bug Affects Versions: 1.10.0 Reporter: Rahul Singh Kubernetes executor allows only two PV one for dag and other for logs , in project scenario there are multiple PV needed , like separate PV to store data , PV to store common script files etc . Current implementation is very static to only two PV. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3100) Add instructions how to run individual tests in the docker environment
[ https://issues.apache.org/jira/browse/AIRFLOW-3100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-3100. - Resolution: Fixed Resolved by https://github.com/apache/incubator-airflow/pull/3933 > Add instructions how to run individual tests in the docker environment > --- > > Key: AIRFLOW-3100 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3100 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Holden Karau's magical unicorn >Assignee: holdenk >Priority: Trivial > > Since the docker env is now how we expect folks to run there tests, for > individuals who just want to verify and individual test we should tell them > how to do that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3101) Fix docker-compose local testing
[
https://issues.apache.org/jira/browse/AIRFLOW-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik resolved AIRFLOW-3101.
-
Resolution: Fixed
Resolved by https://github.com/apache/incubator-airflow/pull/3933
> Fix docker-compose local testing
>
>
> Key: AIRFLOW-3101
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3101
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: holdenk
>Assignee: holdenk
>Priority: Trivial
>
> When testing with docker compose I ran into the following issues:
> 1) No tox installed
> 2) path did not include /home/airflow/.local/bin instead included
> /home/\{systemuser}/.local/bin but $HOME inside of the system pointed to
> /home/airflow, so pip installed packages where not easily callable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-3125) Add monitoring on Task Instance creation rate
Mingye Xia created AIRFLOW-3125: --- Summary: Add monitoring on Task Instance creation rate Key: AIRFLOW-3125 URL: https://issues.apache.org/jira/browse/AIRFLOW-3125 Project: Apache Airflow Issue Type: Improvement Reporter: Mingye Xia Assignee: Mingye Xia Monitoring on Task Instance creation rate can give us some visibility on how much workload we are putting on Airflow. It can be used for resource allocation in the long run (i.e. to determine when we should scale up workers) and and debugging in scenarios like creation rate for certain types of Task Instances spike. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3100) Add instructions how to run individual tests in the docker environment
[ https://issues.apache.org/jira/browse/AIRFLOW-3100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631118#comment-16631118 ] ASF GitHub Bot commented on AIRFLOW-3100: - kaxil closed pull request #3933: [AIRFLOW-3100][AIRFLOW-3101] Improve docker compose local testing URL: https://github.com/apache/incubator-airflow/pull/3933 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 152d5d9aab..f114c66585 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -146,9 +146,18 @@ There are three ways to setup an Apache Airflow development environment. # From the container pip install -e .[devel] # Run all the tests with python and mysql through tox + pip install tox tox -e py35-backend_mysql ``` + If you wish to run individual tests inside of docker enviroment you can do as follows: + + ```bash +# From the container (with your desired enviroment) with druid hook +tox -e py35-backend_mysql -- tests/hooks/test_druid_hook.py + ``` + + ### Running unit tests To run tests locally, once your unit test environment is setup (directly on your diff --git a/scripts/ci/docker-compose.yml b/scripts/ci/docker-compose.yml index 4accf119f6..101ad95297 100644 --- a/scripts/ci/docker-compose.yml +++ b/scripts/ci/docker-compose.yml @@ -66,6 +66,7 @@ services: init: true environment: - USER=airflow + - ADDITIONAL_PATH=~/.local/bin - SLUGIFY_USES_TEXT_UNIDECODE=yes - TOX_ENV - PYTHON_VERSION diff --git a/scripts/ci/run-ci.sh b/scripts/ci/run-ci.sh index 1a65bf1ef6..f16ab5284d 100755 --- a/scripts/ci/run-ci.sh +++ b/scripts/ci/run-ci.sh @@ -32,8 +32,8 @@ else PIP=pip fi -sudo $PIP install --upgrade pip -sudo $PIP install tox +sudo -H $PIP install --upgrade pip +sudo -H $PIP install tox cd $AIRFLOW_ROOT && $PIP --version && tox --version This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add instructions how to run individual tests in the docker environment > --- > > Key: AIRFLOW-3100 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3100 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Holden Karau's magical unicorn >Assignee: holdenk >Priority: Trivial > > Since the docker env is now how we expect folks to run there tests, for > individuals who just want to verify and individual test we should tell them > how to do that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] kaxil commented on issue #3933: [AIRFLOW-3100][AIRFLOW-3101] Improve docker compose local testing
kaxil commented on issue #3933: [AIRFLOW-3100][AIRFLOW-3101] Improve docker compose local testing URL: https://github.com/apache/incubator-airflow/pull/3933#issuecomment-425262263 Thanks @holdenk This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kaxil closed pull request #3933: [AIRFLOW-3100][AIRFLOW-3101] Improve docker compose local testing
kaxil closed pull request #3933: [AIRFLOW-3100][AIRFLOW-3101] Improve docker compose local testing URL: https://github.com/apache/incubator-airflow/pull/3933 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 152d5d9aab..f114c66585 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -146,9 +146,18 @@ There are three ways to setup an Apache Airflow development environment. # From the container pip install -e .[devel] # Run all the tests with python and mysql through tox + pip install tox tox -e py35-backend_mysql ``` + If you wish to run individual tests inside of docker enviroment you can do as follows: + + ```bash +# From the container (with your desired enviroment) with druid hook +tox -e py35-backend_mysql -- tests/hooks/test_druid_hook.py + ``` + + ### Running unit tests To run tests locally, once your unit test environment is setup (directly on your diff --git a/scripts/ci/docker-compose.yml b/scripts/ci/docker-compose.yml index 4accf119f6..101ad95297 100644 --- a/scripts/ci/docker-compose.yml +++ b/scripts/ci/docker-compose.yml @@ -66,6 +66,7 @@ services: init: true environment: - USER=airflow + - ADDITIONAL_PATH=~/.local/bin - SLUGIFY_USES_TEXT_UNIDECODE=yes - TOX_ENV - PYTHON_VERSION diff --git a/scripts/ci/run-ci.sh b/scripts/ci/run-ci.sh index 1a65bf1ef6..f16ab5284d 100755 --- a/scripts/ci/run-ci.sh +++ b/scripts/ci/run-ci.sh @@ -32,8 +32,8 @@ else PIP=pip fi -sudo $PIP install --upgrade pip -sudo $PIP install tox +sudo -H $PIP install --upgrade pip +sudo -H $PIP install tox cd $AIRFLOW_ROOT && $PIP --version && tox --version This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (AIRFLOW-2911) Add job cancellation capability to Dataflow hook
[ https://issues.apache.org/jira/browse/AIRFLOW-2911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pablo Estrada reassigned AIRFLOW-2911: -- Assignee: Pablo Estrada > Add job cancellation capability to Dataflow hook > > > Key: AIRFLOW-2911 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2911 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib, Dataflow, gcp >Reporter: Wilson Lian >Assignee: Pablo Estrada >Priority: Minor > > The hook currently only supports starting and waiting on a job. One might > want to cancel a job when, for example, it exceeds a certain timeout. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2911) Add job cancellation capability to Dataflow hook
[ https://issues.apache.org/jira/browse/AIRFLOW-2911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630993#comment-16630993 ] Pablo Estrada commented on AIRFLOW-2911: I'll take a look at this. > Add job cancellation capability to Dataflow hook > > > Key: AIRFLOW-2911 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2911 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib, Dataflow, gcp >Reporter: Wilson Lian >Assignee: Pablo Estrada >Priority: Minor > > The hook currently only supports starting and waiting on a job. One might > want to cancel a job when, for example, it exceeds a certain timeout. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] codecov-io edited a comment on issue #3963: [AIRFLOW-XXX] Fix SlackWebhookOperator execute method comment
codecov-io edited a comment on issue #3963: [AIRFLOW-XXX] Fix SlackWebhookOperator execute method comment URL: https://github.com/apache/incubator-airflow/pull/3963#issuecomment-425225069 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3963?src=pr=h1) Report > Merging [#3963](https://codecov.io/gh/apache/incubator-airflow/pull/3963?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/f4f8027cbf61ce2ed6a9989facf6c99dffb12f66?src=pr=desc) will **increase** coverage by `0.03%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3963?src=pr=tree) ```diff @@Coverage Diff @@ ## master#3963 +/- ## == + Coverage 74.88% 74.91% +0.03% == Files 200 200 Lines 1597415974 == + Hits1196211967 +5 + Misses 4012 4007 -5 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3963?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/www\_rbac/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3963/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==) | `72.04% <0%> (+0.07%)` | :arrow_up: | | [airflow/configuration.py](https://codecov.io/gh/apache/incubator-airflow/pull/3963/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.25% <0%> (+0.74%)` | :arrow_up: | | [airflow/www\_rbac/app.py](https://codecov.io/gh/apache/incubator-airflow/pull/3963/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy9hcHAucHk=) | `97.77% <0%> (+2.22%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3963?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3963?src=pr=footer). Last update [f4f8027...883bee4](https://codecov.io/gh/apache/incubator-airflow/pull/3963?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #3963: [AIRFLOW-XXX] Fix SlackWebhookOperator execute method comment
codecov-io commented on issue #3963: [AIRFLOW-XXX] Fix SlackWebhookOperator execute method comment URL: https://github.com/apache/incubator-airflow/pull/3963#issuecomment-425225069 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3963?src=pr=h1) Report > Merging [#3963](https://codecov.io/gh/apache/incubator-airflow/pull/3963?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/f4f8027cbf61ce2ed6a9989facf6c99dffb12f66?src=pr=desc) will **increase** coverage by `0.03%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3963?src=pr=tree) ```diff @@Coverage Diff @@ ## master#3963 +/- ## == + Coverage 74.88% 74.91% +0.03% == Files 200 200 Lines 1597415974 == + Hits1196211967 +5 + Misses 4012 4007 -5 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3963?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/www\_rbac/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3963/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==) | `72.04% <0%> (+0.07%)` | :arrow_up: | | [airflow/configuration.py](https://codecov.io/gh/apache/incubator-airflow/pull/3963/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.25% <0%> (+0.74%)` | :arrow_up: | | [airflow/www\_rbac/app.py](https://codecov.io/gh/apache/incubator-airflow/pull/3963/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy9hcHAucHk=) | `97.77% <0%> (+2.22%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3963?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3963?src=pr=footer). Last update [f4f8027...883bee4](https://codecov.io/gh/apache/incubator-airflow/pull/3963?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kaxil closed pull request #3963: [AIRFLOW-XXX] Fix SlackWebhookOperator execute method comment
kaxil closed pull request #3963: [AIRFLOW-XXX] Fix SlackWebhookOperator execute method comment URL: https://github.com/apache/incubator-airflow/pull/3963 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/airflow/contrib/operators/slack_webhook_operator.py b/airflow/contrib/operators/slack_webhook_operator.py index 5263b5c0f2..64f2448a55 100644 --- a/airflow/contrib/operators/slack_webhook_operator.py +++ b/airflow/contrib/operators/slack_webhook_operator.py @@ -77,7 +77,7 @@ def __init__(self, def execute(self, context): """ -Call the SparkSqlHook to run the provided sql query +Call the SlackWebhookHook to post the provided Slack message """ self.hook = SlackWebhookHook( self.http_conn_id, This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jcao219 commented on a change in pull request #3930: [AIRFLOW-2548] Output plugin import errors to web UI
jcao219 commented on a change in pull request #3930: [AIRFLOW-2548] Output
plugin import errors to web UI
URL: https://github.com/apache/incubator-airflow/pull/3930#discussion_r221033419
##
File path: airflow/plugins_manager.py
##
@@ -97,6 +99,7 @@ def validate(cls):
except Exception as e:
log.exception(e)
log.error('Failed to import plugin %s', filepath)
+import_errors[filepath] = str(e)
Review comment:
I believe the only way to get the Airflow UI to reload a broken plugin is to
restart the webserver.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] nritholtz opened a new pull request #3963: [AIRFLOW-XXX] Fix SlackWebhookOperator execute method comment
nritholtz opened a new pull request #3963: [AIRFLOW-XXX] Fix
SlackWebhookOperator execute method comment
URL: https://github.com/apache/incubator-airflow/pull/3963
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
The SlackWebhookOperator's `execute` method doc refers to `SparkSqlHook`
instead of `SlackWebhookHook`
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
No tests - documentation fix only.
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] akosel commented on issue #2372: [AIRFLOW-393] Add callback for FTP downloads
akosel commented on issue #2372: [AIRFLOW-393] Add callback for FTP downloads URL: https://github.com/apache/incubator-airflow/pull/2372#issuecomment-425177464 Better late than never...I'm not sure if @skudriashev is still a maintainer of Airflow, but if not, could I get another review/stamp? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] akosel commented on a change in pull request #2372: [AIRFLOW-393] Add callback for FTP downloads
akosel commented on a change in pull request #2372: [AIRFLOW-393] Add callback for FTP downloads URL: https://github.com/apache/incubator-airflow/pull/2372#discussion_r221007675 ## File path: airflow/contrib/hooks/ftp_hook.py ## @@ -165,11 +172,15 @@ def retrieve_file(self, remote_full_path, local_full_path_or_buffer): else: output_handle = local_full_path_or_buffer +if not callback: Review comment: Added This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ron819 commented on issue #3249: [AIRFLOW-2354] Change task instance run validation to not exclude das…
ron819 commented on issue #3249: [AIRFLOW-2354] Change task instance run validation to not exclude das… URL: https://github.com/apache/incubator-airflow/pull/3249#issuecomment-425108075 @drajguru you have some conflicts to resolve This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] XD-DENG commented on issue #3939: [AIRFLOW-3104] Add .airflowignore info into doc
XD-DENG commented on issue #3939: [AIRFLOW-3104] Add .airflowignore info into doc URL: https://github.com/apache/incubator-airflow/pull/3939#issuecomment-425101563 Hi @ashb , the revision is ready to mention directories plus all its subfolders would not be scanned by Airflow at all if its name matches pattern in `.airflowignore`, which improves efficiency of DAG finding. PTAL. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] akshayi1 commented on a change in pull request #3952: [AIRFLOW-XXX] Update GCS logging docs for latest code
akshayi1 commented on a change in pull request #3952: [AIRFLOW-XXX] Update GCS
logging docs for latest code
URL: https://github.com/apache/incubator-airflow/pull/3952#discussion_r220928415
##
File path: docs/howto/write-logs.rst
##
@@ -89,54 +89,21 @@ Writing Logs to Google Cloud Storage
Follow the steps below to enable Google Cloud Storage logging.
-#. Airflow's logging system requires a custom .py file to be located in the
``PYTHONPATH``, so that it's importable from Airflow. Start by creating a
directory to store the config file. ``$AIRFLOW_HOME/config`` is recommended.
-#. Create empty files called ``$AIRFLOW_HOME/config/log_config.py`` and
``$AIRFLOW_HOME/config/__init__.py``.
-#. Copy the contents of ``airflow/config_templates/airflow_local_settings.py``
into the ``log_config.py`` file that was just created in the step above.
-#. Customize the following portions of the template:
-
-.. code-block:: bash
-
-# Add this variable to the top of the file. Note the trailing slash.
-GCS_LOG_FOLDER = 'gs:///'
-
-# Rename DEFAULT_LOGGING_CONFIG to LOGGING CONFIG
-LOGGING_CONFIG = ...
-
-# Add a GCSTaskHandler to the 'handlers' block of the LOGGING_CONFIG
variable
-'gcs.task': {
-'class': 'airflow.utils.log.gcs_task_handler.GCSTaskHandler',
-'formatter': 'airflow.task',
-'base_log_folder': os.path.expanduser(BASE_LOG_FOLDER),
-'gcs_log_folder': GCS_LOG_FOLDER,
-'filename_template': FILENAME_TEMPLATE,
-},
-
-# Update the airflow.task and airflow.task_runner blocks to be
'gcs.task' instead of 'file.task'.
-'loggers': {
-'airflow.task': {
-'handlers': ['gcs.task'],
-...
-},
-'airflow.task_runner': {
-'handlers': ['gcs.task'],
-...
-},
-'airflow': {
-'handlers': ['console'],
-...
-},
-}
-
-#. Make sure a Google Cloud Platform connection hook has been defined in
Airflow. The hook should have read and write access to the Google Cloud Storage
bucket defined above in ``GCS_LOG_FOLDER``.
-
-#. Update ``$AIRFLOW_HOME/airflow.cfg`` to contain:
+To enable this feature, ``airflow.cfg`` must be configured as in this
+example:
-.. code-block:: bash
+.. code-block:: bash
-task_log_reader = gcs.task
-logging_config_class = log_config.LOGGING_CONFIG
-remote_log_conn_id =
+[core]
+# Airflow can store logs remotely in AWS S3. Users must supply a remote
+# location URL (starting with either 's3://...') and an Airflow connection
+# id that provides access to the storage location.
+remote_logging_enabled = True
Review comment:
Fixed the comment block and the config property.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] kaxil commented on issue #3: Add make
kaxil commented on issue #3: Add make URL: https://github.com/apache/incubator-airflow-ci/pull/3#issuecomment-425082644 I have merged this one. Good catch @johnhofman regarding python3 being used. Let's figure out how we can resolve it now. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kaxil closed pull request #3: Add make
kaxil closed pull request #3: Add make
URL: https://github.com/apache/incubator-airflow-ci/pull/3
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/Dockerfile b/Dockerfile
index af724d9..4ea5ea6 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -63,6 +63,7 @@ RUN apt-get update && apt-get install --no-install-recommends
-y \
python-pkg-resources \
python3-setuptools \
python3-pkg-resources \
+ make \
nodejs \
vim \
less \
@@ -93,9 +94,10 @@ RUN sed -i 's/^# en_US.UTF-8 UTF-8$/en_US.UTF-8 UTF-8/g'
/etc/locale.gen \
&& update-locale LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8
# Install Hadoop
+# --absolute-names is a work around to avoid this issue
https://github.com/docker/hub-feedback/issues/727
RUN cd /tmp && \
wget -q
https://archive.cloudera.com/cdh5/cdh/5/hadoop-${HADOOP_VERSION}-cdh5.11.0.tar.gz
&& \
-tar xzf hadoop-${HADOOP_VERSION}-cdh5.11.0.tar.gz --strip-components 1 -C
$HADOOP_HOME && \
+tar xzf hadoop-${HADOOP_VERSION}-cdh5.11.0.tar.gz --absolute-names
--strip-components 1 -C $HADOOP_HOME && \
rm hadoop-${HADOOP_VERSION}-cdh5.11.0.tar.gz
# Install Hive
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[incubator-airflow-ci] 01/01: Merge pull request #3 from johnhofman/feature/add-make
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-airflow-ci.git commit 7488589aef4d05d2958b050238b341639b82ad11 Merge: 19e3c59 0e80d1c Author: Kaxil Naik AuthorDate: Thu Sep 27 14:03:30 2018 +0100 Merge pull request #3 from johnhofman/feature/add-make Add make Dockerfile | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
[incubator-airflow-ci] branch master updated (19e3c59 -> 7488589)
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-airflow-ci.git. from 19e3c59 Merge pull request #2 from holdenk/install-some-requirements-for-testing add 0e80d1c Add make new 7488589 Merge pull request #3 from johnhofman/feature/add-make The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: Dockerfile | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
[GitHub] ckljohn commented on issue #3855: [AIRFLOW-3022] Add volume mount to KubernetesExecutorConfig
ckljohn commented on issue #3855: [AIRFLOW-3022] Add volume mount to KubernetesExecutorConfig URL: https://github.com/apache/incubator-airflow/pull/3855#issuecomment-425075376 @dimberman Done. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] johnhofman commented on a change in pull request #3: WIP: Add make
johnhofman commented on a change in pull request #3: WIP: Add make
URL: https://github.com/apache/incubator-airflow-ci/pull/3#discussion_r220906331
##
File path: Dockerfile
##
@@ -95,7 +96,7 @@ RUN sed -i 's/^# en_US.UTF-8 UTF-8$/en_US.UTF-8 UTF-8/g'
/etc/locale.gen \
# Install Hadoop
RUN cd /tmp && \
wget -q
https://archive.cloudera.com/cdh5/cdh/5/hadoop-${HADOOP_VERSION}-cdh5.11.0.tar.gz
&& \
-tar xzf hadoop-${HADOOP_VERSION}-cdh5.11.0.tar.gz --strip-components 1 -C
$HADOOP_HOME && \
+tar xzf hadoop-${HADOOP_VERSION}-cdh5.11.0.tar.gz --absolute-names
--strip-components 1 -C $HADOOP_HOME && \
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] johnhofman commented on issue #3: WIP: Add make
johnhofman commented on issue #3: WIP: Add make URL: https://github.com/apache/incubator-airflow-ci/pull/3#issuecomment-425074580 I tried using `pip2` or `pip3` explicitly but it doesn't change the test results. I think the system python of the container has changed from Python2.7 to to Python3.5, probably from the updates to the xenial base image and/or the apt repositories. This means all the python based tools like `tox` and `virtualenv` are now running in Python3.5, and the tests are assuming they run Python2.7. I don't know which of these tools causes the problem, but I would guess `virtualenv` since it is the `PythonVirtualenvOperator` that is failing. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] bart-eijk commented on issue #3961: [AIRFLOW-2794] Add WasbDeleteBlobOperator
bart-eijk commented on issue #3961: [AIRFLOW-2794] Add WasbDeleteBlobOperator URL: https://github.com/apache/incubator-airflow/pull/3961#issuecomment-425072820 Rebased on master after `click` was pinned to version 6.7. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb commented on issue #3: WIP: Add make
ashb commented on issue #3: WIP: Add make URL: https://github.com/apache/incubator-airflow-ci/pull/3#issuecomment-425071900 (Not familiar with this image, so this suggestion might not make sense) Could we fix this by explicitly mentioning `pip2` or `pip3`? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] johnhofman commented on issue #3: WIP: Add make
johnhofman commented on issue #3: WIP: Add make URL: https://github.com/apache/incubator-airflow-ci/pull/3#issuecomment-425071349 @kaxil The latest version of `pip` has change to Python 3. ``` + pip --version pip 18.0 from /usr/local/lib/python3.5/dist-packages/pip (python 3.5) + tox --version 3.4.0 imported from /usr/local/lib/python3.5/dist-packages/tox/__init__.py ``` where using the previous image it was Python 2 ``` + pip --version pip 18.0 from /usr/local/lib/python2.7/dist-packages/pip (python 2.7) + tox --version 3.4.0 imported from /usr/local/lib/python2.7/dist-packages/tox/__init__.pyc ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kaxil edited a comment on issue #3: WIP: Add make
kaxil edited a comment on issue #3: WIP: Add make URL: https://github.com/apache/incubator-airflow-ci/pull/3#issuecomment-425068024 Still not entirely sure on why the tests related to `PythonVirtualenvOperator` is failing. It is somehow using `protocol=3` for `pickle.dump` causing it to fail. Can you just add the comment that @ashb asked, we should merge this PR once that is done. Let's figure out the `PythonVirtualenvOperator` in a separate PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kaxil commented on issue #3: WIP: Add make
kaxil commented on issue #3: WIP: Add make URL: https://github.com/apache/incubator-airflow-ci/pull/3#issuecomment-425068024 Still not entirely sure on why the tests related to `PythonVirtualenvOperator` is failing. It is somehow using `protocol=3` for `pickle.dump` causing it to fail. Can you just add the comment that @ashb asked, we should merge this PR than. Let's figure out the `PythonVirtualenvOperator` in a separate PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kaxil closed pull request #3962: [AIRFLOW-XXX] Pin Click to 6.7 to Fix CI
kaxil closed pull request #3962: [AIRFLOW-XXX] Pin Click to 6.7 to Fix CI URL: https://github.com/apache/incubator-airflow/pull/3962 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/setup.py b/setup.py index aecc218170..1d7885f88c 100644 --- a/setup.py +++ b/setup.py @@ -239,7 +239,7 @@ def write_version(filename=os.path.join(*['airflow', + cassandra + mongo devel = [ -'click', +'click==6.7', 'freezegun', 'jira', 'lxml>=4.0.0', This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb commented on a change in pull request #3: WIP: Add make
ashb commented on a change in pull request #3: WIP: Add make
URL: https://github.com/apache/incubator-airflow-ci/pull/3#discussion_r220892562
##
File path: Dockerfile
##
@@ -95,7 +96,7 @@ RUN sed -i 's/^# en_US.UTF-8 UTF-8$/en_US.UTF-8 UTF-8/g'
/etc/locale.gen \
# Install Hadoop
RUN cd /tmp && \
wget -q
https://archive.cloudera.com/cdh5/cdh/5/hadoop-${HADOOP_VERSION}-cdh5.11.0.tar.gz
&& \
-tar xzf hadoop-${HADOOP_VERSION}-cdh5.11.0.tar.gz --strip-components 1 -C
$HADOOP_HOME && \
+tar xzf hadoop-${HADOOP_VERSION}-cdh5.11.0.tar.gz --absolute-names
--strip-components 1 -C $HADOOP_HOME && \
Review comment:
Lets keep it, but just add a comment in to the docker file why we have it.
(Cos of dockerfile semantics it should be like:
```
# --absolute-names is cos of X
RUN cd /tmp && \
...
```
)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] kaxil commented on issue #3: WIP: Add make
kaxil commented on issue #3: WIP: Add make URL: https://github.com/apache/incubator-airflow-ci/pull/3#issuecomment-425051387 @johnhofman I am looking into it now. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] johnhofman commented on issue #3: WIP: Add make
johnhofman commented on issue #3: WIP: Add make URL: https://github.com/apache/incubator-airflow-ci/pull/3#issuecomment-425049876 The `tests.operators.test_virtualenv_operator.TestPythonVirtualenvOperator` tests are now failing for the `py27_xxx_env_docker` variants. Anybody have any insight into what dependencies could have broken those tests? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] johnhofman commented on a change in pull request #3: WIP: Add make
johnhofman commented on a change in pull request #3: WIP: Add make
URL: https://github.com/apache/incubator-airflow-ci/pull/3#discussion_r220877173
##
File path: Dockerfile
##
@@ -95,7 +96,7 @@ RUN sed -i 's/^# en_US.UTF-8 UTF-8$/en_US.UTF-8 UTF-8/g'
/etc/locale.gen \
# Install Hadoop
RUN cd /tmp && \
wget -q
https://archive.cloudera.com/cdh5/cdh/5/hadoop-${HADOOP_VERSION}-cdh5.11.0.tar.gz
&& \
-tar xzf hadoop-${HADOOP_VERSION}-cdh5.11.0.tar.gz --strip-components 1 -C
$HADOOP_HOME && \
+tar xzf hadoop-${HADOOP_VERSION}-cdh5.11.0.tar.gz --absolute-names
--strip-components 1 -C $HADOOP_HOME && \
Review comment:
I ran into [this](https://github.com/docker/hub-feedback/issues/727) issue
when building this docker image. It seems to be a sporadic issue related to
using tar in docker, I haven't looked deeper than someone mentioned that
`--absolute-names` helps, which I tried and it builds now.
There are no absolute paths in the hadoop tar so as you say this actually
has no effect, but it must change something internal to tar since my build
fails without it:
```
Step 21/29 : RUN cd /tmp && wget -q
https://archive.cloudera.com/cdh5/cdh/5/hadoop-${HADOOP_VERSION}-cdh5.11.0.tar.gz
&& tar xzf hadoop-${HADOOP_VERSION}-cdh5.11.0.tar.gz --strip-components 1
-C $HADOOP_HOME && rm hadoop-${HADOOP_VERSION}-cdh5.11.0.tar.gz
---> Running in 406a118eff9e
tar: share/hadoop/mapreduce1: Directory renamed before its status could be
extracted
tar: Exiting with failure status due to previous errors
```
It may not be a problem for the automated build of this repository so I can
remove it. Thoughts?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (AIRFLOW-3122) Logs not being printed to STDOUT when running `airflow test` in airflow >= 1.10
[
https://issues.apache.org/jira/browse/AIRFLOW-3122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor updated AIRFLOW-3122:
---
Fix Version/s: 1.10.1
We should fix this for 1.10.1
> Logs not being printed to STDOUT when running `airflow test` in airflow >=
> 1.10
> ---
>
> Key: AIRFLOW-3122
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3122
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.10.0
> Environment: Linux
>Reporter: Fred Israel
>Priority: Major
> Fix For: 1.10.1
>
> Attachments: airflow_bug.zip
>
>
>
> In Version 1.10 it seems like there is a regression, as logs in bash operator
> are not being sent to stdout in `airflow test` command. This looks wrong to
> me, as it makes the airflow test command a bit useless and impossible to
> debug. Take a look at my tests below:
> (code is in attachment, inside docker for easy testing)
> dag.py
>
> {code:java}
> from datetime import datetime
> from airflow import DAG
> from airflow.operators.bash_operator import BashOperator
> dag = DAG('test')
> start = datetime.now()
> with dag:
> BashOperator(task_id='test', bash_command='echo TEST', start_date=start)
> BashOperator(task_id='test_failed', bash_command='echo TEST ; exit 1',
> start_date=start)
> {code}
>
>
> $ VERSION=1.10.0 docker-compose run --rm bug airflow test test test 2000-1-1
> {code:java}
> [2018-09-26 20:23:21,614] {__init__.py:51} INFO - Using executor
> SequentialExecutor
> [2018-09-26 20:23:21,756] {models.py:258} INFO - Filling up the DagBag from
> /root/airflow/dags
> [2018-09-26 20:23:21,842] {example_kubernetes_operator.py:54} WARNING - Could
> not import KubernetesPodOperator: No module named 'kubernetes'
> [2018-09-26 20:23:21,842] {example_kubernetes_operator.py:55} WARNING -
> Install kubernetes dependencies with: pip install airflow['kubernetes']{code}
> $ VERSION=1.9.0 docker-compose run --rm bug airflow test test test 2000-1-1
> {code:java}
> [2018-09-26 20:23:28,941] {__init__.py:45} INFO - Using executor
> SequentialExecutor
> [2018-09-26 20:23:28,994] {models.py:189} INFO - Filling up the DagBag from
> /root/airflow/dags
> [2018-09-26 20:23:29,094] {bash_operator.py:70} INFO - Tmp dir root location:
> /tmp
> [2018-09-26 20:23:29,094] {bash_operator.py:80} INFO - Temporary script
> location: /tmp/airflowtmphywv1usk//tmp/airflowtmphywv1usk/testoxl3mg4r
> [2018-09-26 20:23:29,095] {bash_operator.py:88} INFO - Running command: echo
> TEST
> [2018-09-26 20:23:29,098] {bash_operator.py:97} INFO - Output:
> [2018-09-26 20:23:29,099] {bash_operator.py:101} INFO - TEST
> [2018-09-26 20:23:29,099] {bash_operator.py:105} INFO - Command exited with
> return code 0{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] kaxil commented on issue #3: WIP: Add make
kaxil commented on issue #3: WIP: Add make URL: https://github.com/apache/incubator-airflow-ci/pull/3#issuecomment-425038642 @johnhofman Looks like the tests are going through. And should be fine. Can you just address the comments that @ashb has in this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kaxil edited a comment on issue #3: WIP: Add make
kaxil edited a comment on issue #3: WIP: Add make URL: https://github.com/apache/incubator-airflow-ci/pull/3#issuecomment-425038642 @johnhofman Looks like the tests are going through. And should be fine. Can you just address the comments that @ashb has in this PR? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kaxil commented on issue #3962: [AIRFLOW-XXX] Pin Click to 6.7 to Fix CI
kaxil commented on issue #3962: [AIRFLOW-XXX] Pin Click to 6.7 to Fix CI URL: https://github.com/apache/incubator-airflow/pull/3962#issuecomment-425038340 This PR in-conjunction with https://github.com/apache/incubator-airflow-ci/pull/3 should fix the issue This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] johnhofman edited a comment on issue #3: WIP: Add make
johnhofman edited a comment on issue #3: WIP: Add make URL: https://github.com/apache/incubator-airflow-ci/pull/3#issuecomment-425035420 I have added `WIP` because the tests on my fork failed. Because this fork includes other changes I won't know if it is the new ci image or my airflow changes. I will re-run the tests with the branch that @kaxil just patched to fix the `click` dependency issue. https://travis-ci.org/johnhofman/incubator-airflow/builds/434014630 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] johnhofman commented on issue #3: WIP: Add make
johnhofman commented on issue #3: WIP: Add make URL: https://github.com/apache/incubator-airflow-ci/pull/3#issuecomment-425035420 I have added `WIP` because the tests on my fork failed. Because this fork includes other changes I won't know if it is the new ci image or my airflow changes. I will re-run the tests with the branch that @kaxil just patched to fix the `click` dependency issue. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb commented on a change in pull request #3: Add make
ashb commented on a change in pull request #3: Add make
URL: https://github.com/apache/incubator-airflow-ci/pull/3#discussion_r220856381
##
File path: Dockerfile
##
@@ -95,7 +96,7 @@ RUN sed -i 's/^# en_US.UTF-8 UTF-8$/en_US.UTF-8 UTF-8/g'
/etc/locale.gen \
# Install Hadoop
RUN cd /tmp && \
wget -q
https://archive.cloudera.com/cdh5/cdh/5/hadoop-${HADOOP_VERSION}-cdh5.11.0.tar.gz
&& \
-tar xzf hadoop-${HADOOP_VERSION}-cdh5.11.0.tar.gz --strip-components 1 -C
$HADOOP_HOME && \
+tar xzf hadoop-${HADOOP_VERSION}-cdh5.11.0.tar.gz --absolute-names
--strip-components 1 -C $HADOOP_HOME && \
Review comment:
What's this flag change for?
It sounds someone counter-intuative when combined with -C -- absolute-names
means any absolute paths in the tar will be extracted to there, but with
strip-components, and -C mean we want to extract into to a specific subdir...?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] johnhofman commented on issue #3: Add make
johnhofman commented on issue #3: Add make URL: https://github.com/apache/incubator-airflow-ci/pull/3#issuecomment-425032070 @kaxil Cool! That will fix the `click` issue, but I think your build will still fail until this PR is also merged. Combined we should get a green build again. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] bart-eijk commented on a change in pull request #3961: [AIRFLOW-2794] Add WasbDeleteBlobOperator
bart-eijk commented on a change in pull request #3961: [AIRFLOW-2794] Add WasbDeleteBlobOperator URL: https://github.com/apache/incubator-airflow/pull/3961#discussion_r220861811 ## File path: setup.py ## @@ -239,7 +239,7 @@ def write_version(filename=os.path.join(*['airflow', + cassandra + mongo devel = [ -'click', +'click==6.7', Review comment: Good point. I see that seconds after you posted this kaxil did so: https://github.com/apache/incubator-airflow/pull/3962 Let's wait until this PR is merged, and I'll rebase it on master. :) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kaxil commented on issue #3: Add make
kaxil commented on issue #3: Add make URL: https://github.com/apache/incubator-airflow-ci/pull/3#issuecomment-425030427 @johnhofman I have raised a PR to pin click to 6.7 .However, I am not entirely sure if that would solve the issue. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb commented on a change in pull request #3961: [AIRFLOW-2794] Add WasbDeleteBlobOperator
ashb commented on a change in pull request #3961: [AIRFLOW-2794] Add WasbDeleteBlobOperator URL: https://github.com/apache/incubator-airflow/pull/3961#discussion_r220861435 ## File path: setup.py ## @@ -239,7 +239,7 @@ def write_version(filename=os.path.join(*['airflow', + cassandra + mongo devel = [ -'click', +'click==6.7', Review comment: Ah NM someone has just created this as a PR. Rebase on to master in a few minutes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-2966) KubernetesExecutor + namespace quotas kills scheduler if the pod can't be launched
[
https://issues.apache.org/jira/browse/AIRFLOW-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Hofman updated AIRFLOW-2966:
-
External issue URL: https://github.com/apache/incubator-airflow/pull/3960
> KubernetesExecutor + namespace quotas kills scheduler if the pod can't be
> launched
> --
>
> Key: AIRFLOW-2966
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2966
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 2.0.0
> Environment: Kubernetes 1.9.8
>Reporter: John Hofman
>Priority: Major
>
> When running Airflow in Kubernetes with the KubernetesExecutor and resource
> quota's set on the namespace Airflow is deployed in. If the scheduler tries
> to launch a pod into the namespace that exceeds the namespace limits it gets
> an ApiException, and crashes the scheduler.
> This stack trace is an example of the ApiException from the kubernetes client:
> {code:java}
> [2018-08-27 09:51:08,516] {pod_launcher.py:58} ERROR - Exception when
> attempting to create Namespaced Pod.
> Traceback (most recent call last):
> File "/src/apache-airflow/airflow/contrib/kubernetes/pod_launcher.py", line
> 55, in run_pod_async
> resp = self._client.create_namespaced_pod(body=req, namespace=pod.namespace)
> File
> "/usr/local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py",
> line 6057, in create_namespaced_pod
> (data) = self.create_namespaced_pod_with_http_info(namespace, body, **kwargs)
> File
> "/usr/local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py",
> line 6142, in create_namespaced_pod_with_http_info
> collection_formats=collection_formats)
> File
> "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py",
> line 321, in call_api
> _return_http_data_only, collection_formats, _preload_content,
> _request_timeout)
> File
> "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py",
> line 155, in __call_api
> _request_timeout=_request_timeout)
> File
> "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py",
> line 364, in request
> body=body)
> File "/usr/local/lib/python3.6/site-packages/kubernetes/client/rest.py", line
> 266, in POST
> body=body)
> File "/usr/local/lib/python3.6/site-packages/kubernetes/client/rest.py", line
> 222, in request
> raise ApiException(http_resp=r)
> kubernetes.client.rest.ApiException: (403)
> Reason: Forbidden
> HTTP response headers: HTTPHeaderDict({'Audit-Id':
> 'b00e2cbb-bdb2-41f3-8090-824aee79448c', 'Content-Type': 'application/json',
> 'Date': 'Mon, 27 Aug 2018 09:51:08 GMT', 'Content-Length': '410'})
> HTTP response body:
> {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods
> \"podname-ec366e89ef934d91b2d3ffe96234a725\" is forbidden: exceeded quota:
> compute-resources, requested: limits.memory=4Gi, used: limits.memory=6508Mi,
> limited:
> limits.memory=10Gi","reason":"Forbidden","details":{"name":"podname-ec366e89ef934d91b2d3ffe96234a725","kind":"pods"},"code":403}{code}
>
> I would expect the scheduler to catch the Exception and at least mark the
> task as failed, or better yet retry the task later.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] kaxil opened a new pull request #3962: [AIRFLOW-XXX] Pin Click to 6.7 to Fix CI
kaxil opened a new pull request #3962: [AIRFLOW-XXX] Pin Click to 6.7 to Fix CI
URL: https://github.com/apache/incubator-airflow/pull/3962
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ashb commented on a change in pull request #3961: [AIRFLOW-2794] Add WasbDeleteBlobOperator
ashb commented on a change in pull request #3961: [AIRFLOW-2794] Add WasbDeleteBlobOperator URL: https://github.com/apache/incubator-airflow/pull/3961#discussion_r220860891 ## File path: setup.py ## @@ -239,7 +239,7 @@ def write_version(filename=os.path.join(*['airflow', + cassandra + mongo devel = [ -'click', +'click==6.7', Review comment: Sorry to be a pain, but can you make this change a separate PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-2794) Add delete support for Azure blob
[ https://issues.apache.org/jira/browse/AIRFLOW-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bart Eijk updated AIRFLOW-2794: --- External issue URL: https://github.com/apache/incubator-airflow/pull/3961 > Add delete support for Azure blob > - > > Key: AIRFLOW-2794 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2794 > Project: Apache Airflow > Issue Type: Wish > Components: hooks, operators >Reporter: Bart Eijk >Assignee: Bart Eijk >Priority: Trivial > > As a developer, I would like to have the ability to create tasks that can > delete files in Azure blob storage. > Nice to have: the ability to delete a "folder", i.e. a prefix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2794) Add delete support for Azure blob
[ https://issues.apache.org/jira/browse/AIRFLOW-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630077#comment-16630077 ] ASF GitHub Bot commented on AIRFLOW-2794: - bart-eijk opened a new pull request #3961: [AIRFLOW-2794] Add WasbDeleteBlobOperator URL: https://github.com/apache/incubator-airflow/pull/3961 ### Description * Add `delete_file` method to WasbHook * Add WasbDeleteBlobOperator *Additional changes* The `click` dependency was recently updated to `7.0`, which breaks the build. I have pinpointed it to `6.7` in order to (temporarily) fix this. ### Tests * Add 4 tests to tests/contrib/hooks/test_wasb_hook.py * Create new tests in tests/contrib/operators/test_wasb_delete_blob_operator.py ### Code Quality On my manually patched local installation, running ./scripts/ci/flake8-diff.sh exits without any problems. Currently, the Travis CI build will fail due to a new version of `pynacl` being released. This is addressed in the following PR by John: https://github.com/apache/incubator-airflow-ci/pull/3 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add delete support for Azure blob > - > > Key: AIRFLOW-2794 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2794 > Project: Apache Airflow > Issue Type: Wish > Components: hooks, operators >Reporter: Bart Eijk >Assignee: Bart Eijk >Priority: Trivial > > As a developer, I would like to have the ability to create tasks that can > delete files in Azure blob storage. > Nice to have: the ability to delete a "folder", i.e. a prefix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] bart-eijk opened a new pull request #3961: [AIRFLOW-2794] Add WasbDeleteBlobOperator
bart-eijk opened a new pull request #3961: [AIRFLOW-2794] Add WasbDeleteBlobOperator URL: https://github.com/apache/incubator-airflow/pull/3961 ### Description * Add `delete_file` method to WasbHook * Add WasbDeleteBlobOperator *Additional changes* The `click` dependency was recently updated to `7.0`, which breaks the build. I have pinpointed it to `6.7` in order to (temporarily) fix this. ### Tests * Add 4 tests to tests/contrib/hooks/test_wasb_hook.py * Create new tests in tests/contrib/operators/test_wasb_delete_blob_operator.py ### Code Quality On my manually patched local installation, running ./scripts/ci/flake8-diff.sh exits without any problems. Currently, the Travis CI build will fail due to a new version of `pynacl` being released. This is addressed in the following PR by John: https://github.com/apache/incubator-airflow-ci/pull/3 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3118) DAGs not successful on new installation
[
https://issues.apache.org/jira/browse/AIRFLOW-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630074#comment-16630074
]
Kaxil Naik commented on AIRFLOW-3118:
-
Yes, this has to go in 1.10.1 . I will try and fix this by Sunday.
> DAGs not successful on new installation
> ---
>
> Key: AIRFLOW-3118
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3118
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG
>Affects Versions: 1.10.0
> Environment: Ubuntu 18.04
> Python 3.6
>Reporter: Brylie Christopher Oxley
>Assignee: Kaxil Naik
>Priority: Blocker
> Attachments: Screenshot_20180926_161837.png,
> image-2018-09-26-12-39-03-094.png
>
>
> When trying out Airflow, on localhost, none of the DAG runs are getting to
> the 'success' state. They are getting stuck in 'running', or I manually label
> them as failed:
> !image-2018-09-26-12-39-03-094.png!
> h2. Steps to reproduce
> # create new conda environment
> ** conda create -n airflow
> ** source activate airflow
> # install airflow
> ** pip install apache-airflow
> # initialize Airflow db
> ** airflow initdb
> # disable default paused setting in airflow.cfg
> ** dags_are_paused_at_creation = False
> # {color:#6a8759}run airflow and airflow scheduler (in separate
> terminal){color}
> ** {color:#6a8759}airflow scheduler{color}
> ** {color:#6a8759}airflow webserver{color}
> # {color:#6a8759}unpause example_bash_operator{color}
> ** {color:#6a8759}airflow unpause example_bash_operator{color}
> # {color:#6a8759}log in to Airflow UI{color}
> # {color:#6a8759}turn on example_bash_operator{color}
> # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color}
> h2. {color:#6a8759}Observed result{color}
> {color:#6a8759}The `example_bash_operator` never leaves the "running"
> state.{color}
> h2. {color:#6a8759}Expected result{color}
> {color:#6a8759}The `example_bash_operator` would quickly enter the "success"
> state{color}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] XD-DENG commented on a change in pull request #3939: [AIRFLOW-3104] Add .airflowignore info into doc
XD-DENG commented on a change in pull request #3939: [AIRFLOW-3104] Add .airflowignore info into doc URL: https://github.com/apache/incubator-airflow/pull/3939#discussion_r220859759 ## File path: docs/concepts.rst ## @@ -863,3 +863,30 @@ do the same, but then it is more to use a virtualenv and pip. to be available on the system if a module needs those. In other words only pure python modules can be packaged. + +.airflowignore +'' + +A ``.airflowignore`` file specifies the directories or files in ``DAG_FOLDER`` +that Airflow should intentionally ignore. Each line in ``.airflowignore`` +specifies a regular expression pattern, and directories or files whose names +(not DAG id) match any of the patterns would be ignored (under the hood, +``re.findall()`` is used to match the pattern). Overall it works like a +``.gitignore`` file. + +``.airflowignore`` file should be put in your ``DAG_FOLDER``. +For example, you can prepare a ``.airflowignore`` file with contents + +.. code:: + +project_a +tenant_[\d] + + +Then files like "project_a_dag_1.py", "TESTING_project_a.py", "tenant_1.py", +"project_a/dag_1.py", and "tenant_1/dag_1.py" in your ``DAG_FOLDER`` would be ignored. Review comment: Worth mentioning to users. Sorry I missed to cover this. Let me modify later and ping you then. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3118) DAGs not successful on new installation
[
https://issues.apache.org/jira/browse/AIRFLOW-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630057#comment-16630057
]
Ash Berlin-Taylor commented on AIRFLOW-3118:
Ah, if there are _no_ dags at all in there we don't load the example ones
properly? Fix for 1.10.1?
> DAGs not successful on new installation
> ---
>
> Key: AIRFLOW-3118
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3118
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG
>Affects Versions: 1.10.0
> Environment: Ubuntu 18.04
> Python 3.6
>Reporter: Brylie Christopher Oxley
>Assignee: Kaxil Naik
>Priority: Blocker
> Attachments: Screenshot_20180926_161837.png,
> image-2018-09-26-12-39-03-094.png
>
>
> When trying out Airflow, on localhost, none of the DAG runs are getting to
> the 'success' state. They are getting stuck in 'running', or I manually label
> them as failed:
> !image-2018-09-26-12-39-03-094.png!
> h2. Steps to reproduce
> # create new conda environment
> ** conda create -n airflow
> ** source activate airflow
> # install airflow
> ** pip install apache-airflow
> # initialize Airflow db
> ** airflow initdb
> # disable default paused setting in airflow.cfg
> ** dags_are_paused_at_creation = False
> # {color:#6a8759}run airflow and airflow scheduler (in separate
> terminal){color}
> ** {color:#6a8759}airflow scheduler{color}
> ** {color:#6a8759}airflow webserver{color}
> # {color:#6a8759}unpause example_bash_operator{color}
> ** {color:#6a8759}airflow unpause example_bash_operator{color}
> # {color:#6a8759}log in to Airflow UI{color}
> # {color:#6a8759}turn on example_bash_operator{color}
> # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color}
> h2. {color:#6a8759}Observed result{color}
> {color:#6a8759}The `example_bash_operator` never leaves the "running"
> state.{color}
> h2. {color:#6a8759}Expected result{color}
> {color:#6a8759}The `example_bash_operator` would quickly enter the "success"
> state{color}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] ashb commented on a change in pull request #3939: [AIRFLOW-3104] Add .airflowignore info into doc
ashb commented on a change in pull request #3939: [AIRFLOW-3104] Add .airflowignore info into doc URL: https://github.com/apache/incubator-airflow/pull/3939#discussion_r220855874 ## File path: docs/concepts.rst ## @@ -863,3 +863,30 @@ do the same, but then it is more to use a virtualenv and pip. to be available on the system if a module needs those. In other words only pure python modules can be packaged. + +.airflowignore +'' + +A ``.airflowignore`` file specifies the directories or files in ``DAG_FOLDER`` +that Airflow should intentionally ignore. Each line in ``.airflowignore`` +specifies a regular expression pattern, and directories or files whose names +(not DAG id) match any of the patterns would be ignored (under the hood, +``re.findall()`` is used to match the pattern). Overall it works like a +``.gitignore`` file. + +``.airflowignore`` file should be put in your ``DAG_FOLDER``. +For example, you can prepare a ``.airflowignore`` file with contents + +.. code:: + +project_a +tenant_[\d] + + +Then files like "project_a_dag_1.py", "TESTING_project_a.py", "tenant_1.py", +"project_a/dag_1.py", and "tenant_1/dag_1.py" in your ``DAG_FOLDER`` would be ignored. Review comment: One small clarification here (that maybe doesn't matter) but if a directory matches a pattern then Airflow doesn't look at any files or sub-directories in it either - this is useful to exclude whole directories with a large number of files which can slow down airflow DAG finding. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb commented on issue #3873: [Airflow-2760] Decouple DAG parsing loop from scheduler loop
ashb commented on issue #3873: [Airflow-2760] Decouple DAG parsing loop from scheduler loop URL: https://github.com/apache/incubator-airflow/pull/3873#issuecomment-425024333 That's a good point about the audience of the docs. Cpaturing this info somewhere outside of a PR for future is def good, and in the code sound like a good idea for now. Or perhaps as doc-strings against relevant classes (DagFileProcessor)? They won't be rendered anywhere in to the docs at the moment, but could easily be in the future. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3118) DAGs not successful on new installation
[
https://issues.apache.org/jira/browse/AIRFLOW-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630051#comment-16630051
]
Kaxil Naik commented on AIRFLOW-3118:
-
[~brylie] Yes, I had identified that bug a few weeks back. Will fix this in the
upcoming days.
> DAGs not successful on new installation
> ---
>
> Key: AIRFLOW-3118
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3118
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG
>Affects Versions: 1.10.0
> Environment: Ubuntu 18.04
> Python 3.6
>Reporter: Brylie Christopher Oxley
>Priority: Blocker
> Attachments: Screenshot_20180926_161837.png,
> image-2018-09-26-12-39-03-094.png
>
>
> When trying out Airflow, on localhost, none of the DAG runs are getting to
> the 'success' state. They are getting stuck in 'running', or I manually label
> them as failed:
> !image-2018-09-26-12-39-03-094.png!
> h2. Steps to reproduce
> # create new conda environment
> ** conda create -n airflow
> ** source activate airflow
> # install airflow
> ** pip install apache-airflow
> # initialize Airflow db
> ** airflow initdb
> # disable default paused setting in airflow.cfg
> ** dags_are_paused_at_creation = False
> # {color:#6a8759}run airflow and airflow scheduler (in separate
> terminal){color}
> ** {color:#6a8759}airflow scheduler{color}
> ** {color:#6a8759}airflow webserver{color}
> # {color:#6a8759}unpause example_bash_operator{color}
> ** {color:#6a8759}airflow unpause example_bash_operator{color}
> # {color:#6a8759}log in to Airflow UI{color}
> # {color:#6a8759}turn on example_bash_operator{color}
> # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color}
> h2. {color:#6a8759}Observed result{color}
> {color:#6a8759}The `example_bash_operator` never leaves the "running"
> state.{color}
> h2. {color:#6a8759}Expected result{color}
> {color:#6a8759}The `example_bash_operator` would quickly enter the "success"
> state{color}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-3118) DAGs not successful on new installation
[
https://issues.apache.org/jira/browse/AIRFLOW-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik reassigned AIRFLOW-3118:
---
Assignee: Kaxil Naik
> DAGs not successful on new installation
> ---
>
> Key: AIRFLOW-3118
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3118
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG
>Affects Versions: 1.10.0
> Environment: Ubuntu 18.04
> Python 3.6
>Reporter: Brylie Christopher Oxley
>Assignee: Kaxil Naik
>Priority: Blocker
> Attachments: Screenshot_20180926_161837.png,
> image-2018-09-26-12-39-03-094.png
>
>
> When trying out Airflow, on localhost, none of the DAG runs are getting to
> the 'success' state. They are getting stuck in 'running', or I manually label
> them as failed:
> !image-2018-09-26-12-39-03-094.png!
> h2. Steps to reproduce
> # create new conda environment
> ** conda create -n airflow
> ** source activate airflow
> # install airflow
> ** pip install apache-airflow
> # initialize Airflow db
> ** airflow initdb
> # disable default paused setting in airflow.cfg
> ** dags_are_paused_at_creation = False
> # {color:#6a8759}run airflow and airflow scheduler (in separate
> terminal){color}
> ** {color:#6a8759}airflow scheduler{color}
> ** {color:#6a8759}airflow webserver{color}
> # {color:#6a8759}unpause example_bash_operator{color}
> ** {color:#6a8759}airflow unpause example_bash_operator{color}
> # {color:#6a8759}log in to Airflow UI{color}
> # {color:#6a8759}turn on example_bash_operator{color}
> # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color}
> h2. {color:#6a8759}Observed result{color}
> {color:#6a8759}The `example_bash_operator` never leaves the "running"
> state.{color}
> h2. {color:#6a8759}Expected result{color}
> {color:#6a8759}The `example_bash_operator` would quickly enter the "success"
> state{color}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] johnhofman commented on issue #3960: [AIRFLOW-2966] Catch ApiException in the Kubernetes Executor
johnhofman commented on issue #3960: [AIRFLOW-2966] Catch ApiException in the Kubernetes Executor URL: https://github.com/apache/incubator-airflow/pull/3960#issuecomment-425021729 This build will fail because the recent release of `pynacl` needs `make` to install. [Here](https://github.com/apache/incubator-airflow-ci/pull/3) is a PR to update the airflow-ci image to include `make`. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-2966) KubernetesExecutor + namespace quotas kills scheduler if the pod can't be launched
[
https://issues.apache.org/jira/browse/AIRFLOW-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630037#comment-16630037
]
ASF GitHub Bot commented on AIRFLOW-2966:
-
johnhofman opened a new pull request #3960: [AIRFLOW-2966] Catch ApiException
in the Kubernetes Executor
URL: https://github.com/apache/incubator-airflow/pull/3960
### Description
Creating a pod that exceeds a namespace's resource quota throws an
ApiException. This change catches the exception and the task is re-queued
inside the Executor instead of killing the scheduler.
`click 7.0` was recently released but `flask-appbuilder 1.11.1 has
requirement click==6.7`. I have pinned `click==6.7` to make the dependencies
resolve.
### Tests
This adds a single test `TestKubernetesExecutor. test_run_next_exception`
that covers this single scenario. Without the changes this test fails when the
ApiException is not caught.
This is the first test case for the `KubernetesExecutor`, so I needed to
add the `[kubernetes]` section to `default_test.cfg` so that the
`KubernetesExecutor` can be built without exceptions.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> KubernetesExecutor + namespace quotas kills scheduler if the pod can't be
> launched
> --
>
> Key: AIRFLOW-2966
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2966
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 2.0.0
> Environment: Kubernetes 1.9.8
>Reporter: John Hofman
>Priority: Major
>
> When running Airflow in Kubernetes with the KubernetesExecutor and resource
> quota's set on the namespace Airflow is deployed in. If the scheduler tries
> to launch a pod into the namespace that exceeds the namespace limits it gets
> an ApiException, and crashes the scheduler.
> This stack trace is an example of the ApiException from the kubernetes client:
> {code:java}
> [2018-08-27 09:51:08,516] {pod_launcher.py:58} ERROR - Exception when
> attempting to create Namespaced Pod.
> Traceback (most recent call last):
> File "/src/apache-airflow/airflow/contrib/kubernetes/pod_launcher.py", line
> 55, in run_pod_async
> resp = self._client.create_namespaced_pod(body=req, namespace=pod.namespace)
> File
> "/usr/local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py",
> line 6057, in create_namespaced_pod
> (data) = self.create_namespaced_pod_with_http_info(namespace, body, **kwargs)
> File
> "/usr/local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py",
> line 6142, in create_namespaced_pod_with_http_info
> collection_formats=collection_formats)
> File
> "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py",
> line 321, in call_api
> _return_http_data_only, collection_formats, _preload_content,
> _request_timeout)
> File
> "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py",
> line 155, in __call_api
> _request_timeout=_request_timeout)
> File
> "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py",
> line 364, in request
> body=body)
> File "/usr/local/lib/python3.6/site-packages/kubernetes/client/rest.py", line
> 266, in POST
> body=body)
> File "/usr/local/lib/python3.6/site-packages/kubernetes/client/rest.py", line
> 222, in request
> raise ApiException(http_resp=r)
> kubernetes.client.rest.ApiException: (403)
> Reason: Forbidden
> HTTP response headers: HTTPHeaderDict({'Audit-Id':
> 'b00e2cbb-bdb2-41f3-8090-824aee79448c', 'Content-Type': 'application/json',
> 'Date': 'Mon, 27 Aug 2018 09:51:08 GMT', 'Content-Length': '410'})
> HTTP response body:
> {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods
> \"podname-ec366e89ef934d91b2d3ffe96234a725\" is forbidden: exceeded quota:
> compute-resources, requested: limits.memory=4Gi, used: limits.memory=6508Mi,
> limited:
> limits.memory=10Gi","reason":"Forbidden","details":{"name":"podname-ec366e89ef934d91b2d3ffe96234a725","kind":"pods"},"code":403}{code}
>
> I would expect the scheduler to catch the Exception and at least mark the
> task as failed, or better yet retry the task later.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] johnhofman opened a new pull request #3960: [AIRFLOW-2966] Catch ApiException in the Kubernetes Executor
johnhofman opened a new pull request #3960: [AIRFLOW-2966] Catch ApiException in the Kubernetes Executor URL: https://github.com/apache/incubator-airflow/pull/3960 ### Description Creating a pod that exceeds a namespace's resource quota throws an ApiException. This change catches the exception and the task is re-queued inside the Executor instead of killing the scheduler. `click 7.0` was recently released but `flask-appbuilder 1.11.1 has requirement click==6.7`. I have pinned `click==6.7` to make the dependencies resolve. ### Tests This adds a single test `TestKubernetesExecutor. test_run_next_exception` that covers this single scenario. Without the changes this test fails when the ApiException is not caught. This is the first test case for the `KubernetesExecutor`, so I needed to add the `[kubernetes]` section to `default_test.cfg` so that the `KubernetesExecutor` can be built without exceptions. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-668) Configuration parsing doesn't work properly with python 3
[
https://issues.apache.org/jira/browse/AIRFLOW-668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor resolved AIRFLOW-668.
---
Resolution: Fixed
No longer an issue.
Tested with this config on Python3.5.2
{code}
sql_alchemy_conn_cmd = echo sqlite:Users/ash/airflow/airflow-2.db
{code}
> Configuration parsing doesn't work properly with python 3
> -
>
> Key: AIRFLOW-668
> URL: https://issues.apache.org/jira/browse/AIRFLOW-668
> Project: Apache Airflow
> Issue Type: Bug
> Environment: Airflow version: v1.7.1.3
> - Airflow components: webserver and scheduler with a postgres database and
> CeleryExecutor
> - Python Version: 3.4.5
>Reporter: Rafael Gomes Fernandes
>Assignee: Rafael Gomes Fernandes
>Priority: Major
>
> The problem is: if you use python3 and the '_cmd' on the config file airflow
> will not start due the error:
> {noformat}
> File
> "~/test/env/airflow3/lib/python3.4/site-packages/airflow/configuration.py",
> line 447, in _validate
> "sqlite" in self.get('core', 'sql_alchemy_conn')):
> TypeError: 'str' does not support the buffer interface
> {noformat}
> To reproduce the problem change the following line on airflow.cfg:
> {code:title=airflow.cfg|borderStyle=solid}
> sql_alchemy_conn_cmd = echo sqlite:~/airflow/airflow.db
> {code}
> The solution is change the following run_command method's line on
> airflow/configuration.py:
> {code:title=airflow/configuration.py|borderStyle=solid}
> command.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE,
> universal_newlines=True)
> {code}
> By setting the universal_newlines to true the file objects stdout and stderr
> are opened as text files and treated as string in python 2 and python 3
> avoiding the error.
> run_command with universal_newlines=True:
> When using python 3 output type: and no error.
> When using python 2 output type: and no error.
> run_command as it is :
> When using python 3 output type: and TypeError.
> When using python 2 output type: and no error.
> I have tested the change with Travis CI and it passed, but when I tried to
> run tox or unit test I found this problems:
> {code:title=run_unit_tests.sh|borderStyle=solid}
> ./run_unit_tests.sh -q -c airflow.cfg
> Initializing the DB
> Starting the unit tests with the following nose arguments: -q -c airflow.cfg
> ==
> ERROR: Failure: OperationalError ((sqlite3.OperationalError) no such table:
> task_instance [SQL: 'DELETE FROM task_instance WHERE task_instance.dag_id =
> ?'] [parameters: ('unit_tests',)])
> --
> Traceback (most recent call last):
> File
> "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/nose/failure.py",
> line 39, in runTest
> raise self.exc_val.with_traceback(self.tb)
> File
> "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/nose/loader.py",
> line 418, in loadTestsFromName
> addr.filename, addr.module)
> File
> "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/nose/importer.py",
> line 47, in importFromPath
> return self.importFromDir(dir_path, fqname)
> File
> "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/nose/importer.py",
> line 94, in importFromDir
> mod = load_module(part_fqname, fh, filename, desc)
> File "/home/varrun/test/env/airflow3/lib/python3.4/imp.py", line 245, in
> load_module
> return load_package(name, filename)
> File "/home/varrun/test/env/airflow3/lib/python3.4/imp.py", line 217, in
> load_package
> return methods.load()
> File "", line 1220, in load
> File "", line 1200, in _load_unlocked
> File "", line 1129, in _exec
> File "", line 1471, in exec_module
> File "", line 321, in _call_with_frames_removed
> File "/home/varrun/dev/incubator-airflow/tests/__init__.py", line 18, in
>
> from .contrib import *
> File "/home/varrun/dev/incubator-airflow/tests/contrib/__init__.py", line
> 16, in
> from .operators import *
> File
> "/home/varrun/dev/incubator-airflow/tests/contrib/operators/__init__.py",
> line 17, in
> from .ssh_execute_operator import *
> File
> "/home/varrun/dev/incubator-airflow/tests/contrib/operators/ssh_execute_operator.py",
> line 37, in
> reset()
> File
> "/home/varrun/dev/incubator-airflow/tests/contrib/operators/ssh_execute_operator.py",
> line 33, in reset
> tis.delete()
> File
> "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
> line 3155, in delete
> delete_op.exec_()
> File
> "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/orm/persistence.py",
> line 1168, in exec_
> self._do_exec()
> File
>
[GitHub] johnhofman commented on issue #3: Add make
johnhofman commented on issue #3: Add make URL: https://github.com/apache/incubator-airflow-ci/pull/3#issuecomment-425015925 I have pushed a build of this to dockerhub `jhofman/incubator-airflow-ci:latest` and am running the integration tests with the new image on a fork I am working on: https://travis-ci.org/johnhofman/incubator-airflow/builds/433982418. The master branch still fails due to other dependency problems relating to the latest `click` release being incompatible with `flask-appbuilder`. That needs to be pinned to `click==6.7` in the airflow `setup.py`. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb commented on issue #3958: [AIRFLOW-3124] Fix RBAC webserver debug mode
ashb commented on issue #3958: [AIRFLOW-3124] Fix RBAC webserver debug mode URL: https://github.com/apache/incubator-airflow/pull/3958#issuecomment-425013738 On the two create apps front: don't worry about it too much, the non-RBAC based UI is going to be removed This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-3123) Allow nested use of DAG as a context manager
[
https://issues.apache.org/jira/browse/AIRFLOW-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor updated AIRFLOW-3123:
---
Summary: Allow nested use of DAG as a context manager (was: DAG context
manager fails in nested cases)
> Allow nested use of DAG as a context manager
> -
>
> Key: AIRFLOW-3123
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3123
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG
>Reporter: Newton Le
>Assignee: Newton Le
>Priority: Major
>
> DAG context manager fails under some cases with nested contexts:
> {code:python}
> with DAG( ... ) as dag:
> op1 = Operator()
> with dag:
> op2 = Operator()
> op3 = Operator
> {code}
> op3 will not continue to be assigned the original DAG after exiting the
> nested context.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3118) DAGs not successful on new installation
[
https://issues.apache.org/jira/browse/AIRFLOW-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629963#comment-16629963
]
Brylie Christopher Oxley commented on AIRFLOW-3118:
---
[~kaxilnaik] I added a DAG from the tutorial, and things work as expected! :)
So, something is going wrong with the example DAGS. We have tested this on
three separate laptops (Ubuntu 18.04, Kubuntu 18.04, and Mac OS X).
> DAGs not successful on new installation
> ---
>
> Key: AIRFLOW-3118
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3118
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG
>Affects Versions: 1.10.0
> Environment: Ubuntu 18.04
> Python 3.6
>Reporter: Brylie Christopher Oxley
>Priority: Blocker
> Attachments: Screenshot_20180926_161837.png,
> image-2018-09-26-12-39-03-094.png
>
>
> When trying out Airflow, on localhost, none of the DAG runs are getting to
> the 'success' state. They are getting stuck in 'running', or I manually label
> them as failed:
> !image-2018-09-26-12-39-03-094.png!
> h2. Steps to reproduce
> # create new conda environment
> ** conda create -n airflow
> ** source activate airflow
> # install airflow
> ** pip install apache-airflow
> # initialize Airflow db
> ** airflow initdb
> # disable default paused setting in airflow.cfg
> ** dags_are_paused_at_creation = False
> # {color:#6a8759}run airflow and airflow scheduler (in separate
> terminal){color}
> ** {color:#6a8759}airflow scheduler{color}
> ** {color:#6a8759}airflow webserver{color}
> # {color:#6a8759}unpause example_bash_operator{color}
> ** {color:#6a8759}airflow unpause example_bash_operator{color}
> # {color:#6a8759}log in to Airflow UI{color}
> # {color:#6a8759}turn on example_bash_operator{color}
> # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color}
> h2. {color:#6a8759}Observed result{color}
> {color:#6a8759}The `example_bash_operator` never leaves the "running"
> state.{color}
> h2. {color:#6a8759}Expected result{color}
> {color:#6a8759}The `example_bash_operator` would quickly enter the "success"
> state{color}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] ashb commented on a change in pull request #3955: [AIRFLOW-3121] Define closed property on StreamLogWriter
ashb commented on a change in pull request #3955: [AIRFLOW-3121] Define closed property on StreamLogWriter URL: https://github.com/apache/incubator-airflow/pull/3955#discussion_r220840322 ## File path: airflow/utils/log/logging_mixin.py ## @@ -83,6 +84,16 @@ def __init__(self, logger, level): self.level = level self._buffer = str() +@property +def closed(self): +""" +Returns False to indicate that the stream is not closed (as it will be +open for the duration of Airflow's lifecycle). + +For compatibility. Review comment: For compatability with what please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3036) Upgrading to Airflow 1.10 not possible using GCP Cloud SQL for MYSQL
[
https://issues.apache.org/jira/browse/AIRFLOW-3036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629949#comment-16629949
]
Ash Berlin-Taylor commented on AIRFLOW-3036:
This is going to need someone who is familiar with MySQL handling of timestamp
and datetime columsn to experiment and check that Mysql does right things with
these columns, even if the TZ of the db server is set to something other than
UTC, and to make sure that it doesn't play silly games with default values
(which is what the setting we require turns off)
> Upgrading to Airflow 1.10 not possible using GCP Cloud SQL for MYSQL
>
>
> Key: AIRFLOW-3036
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3036
> Project: Apache Airflow
> Issue Type: Bug
> Components: core, db
>Affects Versions: 1.10.0
> Environment: Google Cloud Platform, Google Kubernetes Engine, Airflow
> 1.10 on Debian Stretch, Google Cloud SQL MySQL
>Reporter: Smith Mathieu
>Priority: Blocker
> Labels: 1.10, google, google-cloud-sql
>
> The upgrade path to airflow 1.10 seems impossible for users of MySQL in
> Google's Cloud SQL service given new mysql requirements for 1.10.
>
> When executing "airflow upgradedb"
> ```
> INFO [alembic.runtime.migration] Running upgrade d2ae31099d61 ->
> 0e2a74e0fc9f, Add time zone awareness
> Traceback (most recent call last):
> File "/usr/local/bin/airflow", line 32, in
> args.func(args)
> File "/usr/local/lib/python3.6/site-packages/airflow/bin/cli.py", line 1002,
> in initdb
> db_utils.initdb(settings.RBAC)
> File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 92,
> in initdb
> upgradedb()
> File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 346,
> in upgradedb
> command.upgrade(config, 'heads')
> File "/usr/local/lib/python3.6/site-packages/alembic/command.py", line 174,
> in upgrade
> script.run_env()
> File "/usr/local/lib/python3.6/site-packages/alembic/script/base.py", line
> 416, in run_env
> util.load_python_file(self.dir, 'env.py')
> File "/usr/local/lib/python3.6/site-packages/alembic/util/pyfiles.py", line
> 93, in load_python_file
> module = load_module_py(module_id, path)
> File "/usr/local/lib/python3.6/site-packages/alembic/util/compat.py", line
> 68, in load_module_py
> module_id, path).load_module(module_id)
> File "", line 399, in
> _check_name_wrapper
> File "", line 823, in load_module
> File "", line 682, in load_module
> File "", line 265, in _load_module_shim
> File "", line 684, in _load
> File "", line 665, in _load_unlocked
> File "", line 678, in exec_module
> File "", line 219, in _call_with_frames_removed
> File "/usr/local/lib/python3.6/site-packages/airflow/migrations/env.py",
> line 91, in
> run_migrations_online()
> File "/usr/local/lib/python3.6/site-packages/airflow/migrations/env.py",
> line 86, in run_migrations_online
> context.run_migrations()
> File "", line 8, in run_migrations
> File
> "/usr/local/lib/python3.6/site-packages/alembic/runtime/environment.py", line
> 807, in run_migrations
> self.get_context().run_migrations(**kw)
> File "/usr/local/lib/python3.6/site-packages/alembic/runtime/migration.py",
> line 321, in run_migrations
> step.migration_fn(**kw)
> File
> "/usr/local/lib/python3.6/site-packages/airflow/migrations/versions/0e2a74e0fc9f_add_time_zone_awareness.py",
> line 46, in upgrade
> raise Exception("Global variable explicit_defaults_for_timestamp needs to be
> on (1) for mysql")
> Exception: Global variable explicit_defaults_for_timestamp needs to be on
> (1) for mysql
> ```
>
> Reading documentation for upgrading to airflow 1.10, it seems the requirement
> for explicit_defaults_for_timestamp=1 was intentional.
>
> However, MySQL on Google Cloud SQL does not support configuring this
> variable and it is off by default. Users of MySQL and Cloud SQL do not have
> an upgrade path to 1.10. Alas, so close to the mythical Kubernetes Executor.
> In GCP, Cloud SQL is _the_ hosted MySQL solution.
> [https://cloud.google.com/sql/docs/mysql/flags]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Issue Comment Deleted] (AIRFLOW-3036) Upgrading to Airflow 1.10 not possible using GCP Cloud SQL for MYSQL
[
https://issues.apache.org/jira/browse/AIRFLOW-3036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor updated AIRFLOW-3036:
---
Comment: was deleted
(was: Fokko closed pull request #3908: [AIRFLOW-3036] Add relevant ECS options
to ECS operator.
URL: https://github.com/apache/incubator-airflow/pull/3908
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/contrib/operators/ecs_operator.py
b/airflow/contrib/operators/ecs_operator.py
index c85ae15b77..8bad285ffd 100644
--- a/airflow/contrib/operators/ecs_operator.py
+++ b/airflow/contrib/operators/ecs_operator.py
@@ -45,6 +45,15 @@ class ECSOperator(BaseOperator):
:type region_name: str
:param launch_type: the launch type on which to run your task ('EC2' or
'FARGATE')
:type launch_type: str
+:param group: the name of the task group associated with the task
+:type group: str
+:param placement_constraints: an array of placement constraint objects to
use for
+the task
+:type placement_constraints: list
+:param platform_version: the platform version on which your task is running
+:type platform_version: str
+:param network_configuration: the network configuration for the task
+:type network_configuration: dict
"""
ui_color = '#f0ede4'
@@ -54,7 +63,9 @@ class ECSOperator(BaseOperator):
@apply_defaults
def __init__(self, task_definition, cluster, overrides,
- aws_conn_id=None, region_name=None, launch_type='EC2',
**kwargs):
+ aws_conn_id=None, region_name=None, launch_type='EC2',
+ group=None, placement_constraints=None,
platform_version='LATEST',
+ network_configuration=None, **kwargs):
super(ECSOperator, self).__init__(**kwargs)
self.aws_conn_id = aws_conn_id
@@ -63,6 +74,10 @@ def __init__(self, task_definition, cluster, overrides,
self.cluster = cluster
self.overrides = overrides
self.launch_type = launch_type
+self.group = group
+self.placement_constraints = placement_constraints
+self.platform_version = platform_version
+self.network_configuration = network_configuration
self.hook = self.get_hook()
@@ -78,13 +93,21 @@ def execute(self, context):
region_name=self.region_name
)
-response = self.client.run_task(
-cluster=self.cluster,
-taskDefinition=self.task_definition,
-overrides=self.overrides,
-startedBy=self.owner,
-launchType=self.launch_type
-)
+run_opts = {
+'cluster': self.cluster,
+'taskDefinition': self.task_definition,
+'overrides': self.overrides,
+'startedBy': self.owner,
+'launchType': self.launch_type,
+'platformVersion': self.platform_version,
+}
+if self.group is not None:
+run_opts['group'] = self.group
+if self.placement_constraints is not None:
+run_opts['placementConstraints'] = self.placement_constraints
+if self.network_configuration is not None:
+run_opts['networkConfiguration'] = self.network_configuration
+response = self.client.run_task(**run_opts)
failures = response['failures']
if len(failures) > 0:
diff --git a/tests/contrib/operators/test_ecs_operator.py
b/tests/contrib/operators/test_ecs_operator.py
index 43a816da4a..842db1a44a 100644
--- a/tests/contrib/operators/test_ecs_operator.py
+++ b/tests/contrib/operators/test_ecs_operator.py
@@ -69,7 +69,20 @@ def setUp(self, aws_hook_mock):
cluster='c',
overrides={},
aws_conn_id=None,
-region_name='eu-west-1')
+region_name='eu-west-1',
+group='group',
+placement_constraints=[
+{
+'expression': 'attribute:ecs.instance-type =~ t2.*',
+'type': 'memberOf'
+}
+],
+network_configuration={
+'awsvpcConfiguration': {
+'securityGroups': ['sg-123abc']
+}
+}
+)
def test_init(self):
@@ -100,7 +113,20 @@ def test_execute_without_failures(self, check_mock,
wait_mock):
launchType='EC2',
overrides={},
startedBy=mock.ANY, # Can by 'airflow' or 'Airflow'
-taskDefinition='t'
+taskDefinition='t',
+group='group',
+placementConstraints=[
+{
+'expression': 'attribute:ecs.instance-type =~
[jira] [Updated] (AIRFLOW-3036) Upgrading to Airflow 1.10 not possible using GCP Cloud SQL for MYSQL
[
https://issues.apache.org/jira/browse/AIRFLOW-3036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor updated AIRFLOW-3036:
---
Fix Version/s: (was: 2.0.0)
> Upgrading to Airflow 1.10 not possible using GCP Cloud SQL for MYSQL
>
>
> Key: AIRFLOW-3036
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3036
> Project: Apache Airflow
> Issue Type: Bug
> Components: core, db
>Affects Versions: 1.10.0
> Environment: Google Cloud Platform, Google Kubernetes Engine, Airflow
> 1.10 on Debian Stretch, Google Cloud SQL MySQL
>Reporter: Smith Mathieu
>Priority: Blocker
> Labels: 1.10, google, google-cloud-sql
>
> The upgrade path to airflow 1.10 seems impossible for users of MySQL in
> Google's Cloud SQL service given new mysql requirements for 1.10.
>
> When executing "airflow upgradedb"
> ```
> INFO [alembic.runtime.migration] Running upgrade d2ae31099d61 ->
> 0e2a74e0fc9f, Add time zone awareness
> Traceback (most recent call last):
> File "/usr/local/bin/airflow", line 32, in
> args.func(args)
> File "/usr/local/lib/python3.6/site-packages/airflow/bin/cli.py", line 1002,
> in initdb
> db_utils.initdb(settings.RBAC)
> File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 92,
> in initdb
> upgradedb()
> File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 346,
> in upgradedb
> command.upgrade(config, 'heads')
> File "/usr/local/lib/python3.6/site-packages/alembic/command.py", line 174,
> in upgrade
> script.run_env()
> File "/usr/local/lib/python3.6/site-packages/alembic/script/base.py", line
> 416, in run_env
> util.load_python_file(self.dir, 'env.py')
> File "/usr/local/lib/python3.6/site-packages/alembic/util/pyfiles.py", line
> 93, in load_python_file
> module = load_module_py(module_id, path)
> File "/usr/local/lib/python3.6/site-packages/alembic/util/compat.py", line
> 68, in load_module_py
> module_id, path).load_module(module_id)
> File "", line 399, in
> _check_name_wrapper
> File "", line 823, in load_module
> File "", line 682, in load_module
> File "", line 265, in _load_module_shim
> File "", line 684, in _load
> File "", line 665, in _load_unlocked
> File "", line 678, in exec_module
> File "", line 219, in _call_with_frames_removed
> File "/usr/local/lib/python3.6/site-packages/airflow/migrations/env.py",
> line 91, in
> run_migrations_online()
> File "/usr/local/lib/python3.6/site-packages/airflow/migrations/env.py",
> line 86, in run_migrations_online
> context.run_migrations()
> File "", line 8, in run_migrations
> File
> "/usr/local/lib/python3.6/site-packages/alembic/runtime/environment.py", line
> 807, in run_migrations
> self.get_context().run_migrations(**kw)
> File "/usr/local/lib/python3.6/site-packages/alembic/runtime/migration.py",
> line 321, in run_migrations
> step.migration_fn(**kw)
> File
> "/usr/local/lib/python3.6/site-packages/airflow/migrations/versions/0e2a74e0fc9f_add_time_zone_awareness.py",
> line 46, in upgrade
> raise Exception("Global variable explicit_defaults_for_timestamp needs to be
> on (1) for mysql")
> Exception: Global variable explicit_defaults_for_timestamp needs to be on
> (1) for mysql
> ```
>
> Reading documentation for upgrading to airflow 1.10, it seems the requirement
> for explicit_defaults_for_timestamp=1 was intentional.
>
> However, MySQL on Google Cloud SQL does not support configuring this
> variable and it is off by default. Users of MySQL and Cloud SQL do not have
> an upgrade path to 1.10. Alas, so close to the mythical Kubernetes Executor.
> In GCP, Cloud SQL is _the_ hosted MySQL solution.
> [https://cloud.google.com/sql/docs/mysql/flags]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] johnhofman opened a new pull request #3: Add make
johnhofman opened a new pull request #3: Add make URL: https://github.com/apache/incubator-airflow-ci/pull/3 The latest release of pynacl 1.3.0 (a transitive dependency of Airflow) needs make to install. While building this locally I ran into [this issue](https://github.com/docker/hub-feedback/issues/727) unpacking the HADOOP tar, which the `--absolute-names` fixed for me locally. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work started] (AIRFLOW-2794) Add delete support for Azure blob
[ https://issues.apache.org/jira/browse/AIRFLOW-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-2794 started by Bart Eijk. -- > Add delete support for Azure blob > - > > Key: AIRFLOW-2794 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2794 > Project: Apache Airflow > Issue Type: Wish > Components: hooks, operators >Reporter: Bart Eijk >Assignee: Bart Eijk >Priority: Trivial > > As a developer, I would like to have the ability to create tasks that can > delete files in Azure blob storage. > Nice to have: the ability to delete a "folder", i.e. a prefix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] KevinYang21 commented on issue #3873: [Airflow-2760] Decouple DAG parsing loop from scheduler loop
KevinYang21 commented on issue #3873: [Airflow-2760] Decouple DAG parsing loop from scheduler loop URL: https://github.com/apache/incubator-airflow/pull/3873#issuecomment-424980424 @ashb Thanks for the feedback. I have a small updated the scheduler.rst to reflect the change but I didn't go into a lot details. The reason is that the document seems to me is facing Airflow users and they probly don't care too much about exactly what is happening, especially parts like how scheduler loop is divided into three logical pieces. From the existing doc, we stop at the details of DagFileProcessorManager( just mention that it would stay in sync with DAG folder but didn't go into the DagFileProcessor part), to me if we want to keep the same level of detail we should stop at briefly mention about DagFileProcessorManager and DagFileProcessorAgent level. Other the other hand, I can try add more comments in the code if you feel like it, as that would probly be the place where people care about implementation details are going to look at. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] msumit commented on issue #3946: [AIRFLOW-3062] Add Qubole in integration docs
msumit commented on issue #3946: [AIRFLOW-3062] Add Qubole in integration docs URL: https://github.com/apache/incubator-airflow/pull/3946#issuecomment-424968593 @r39132 can you merge now? Resolved all the review comments. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
