[jira] [Updated] (AIRFLOW-5538) Add a flag to make scheduling trigger on start_date instead of execution_date (make execution_date equal to start_date)
[
https://issues.apache.org/jira/browse/AIRFLOW-5538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kasim updated AIRFLOW-5538:
---
Description:
>From [https://airflow.apache.org/scheduler.html] :
> Note that if you run a DAG on a schedule_interval of one day, the run
> stamped 2016-01-01 will be trigger soon after 2016-01-01T23:59. In
> other words, the job instance is started once the period it covers has
> ended.
This feature is very hurt .
For example I have etl job which run every day, schedule_interval is `0 1 * *
*`, so it will trigger 2019-09-22 01:00:00 job on 2019-09-23 01:00:00 . But my
etl is processing all data before start_date , means data range is between
(history, 2019-09-23 00:00:00) , and we can't use `datetime.now()` because this
is unable to reproduce. This force me add 1 day to execution_date:
```python
etl_end_time = "\{{ (execution_date +
macros.timedelta(days=1)).strftime('%Y-%m-%d 00:00:00') }}"
```
However, when I need run a job with schedule_interval `45 2,3,4,5,6 * * *` ,
the `2019-09-22 06:45:00` job would run on `2019-09-23 02:45:00`, which is the
day after execution_date . Instead of adding a day, I had to changed
schedule_interval `45 2,3,4,5,6,7 * * *` and put a dummy operator on last run.
And in this situation , you don't need add one day to execution_date , this
means you have to define two `etl_end_time` to represent a same date in jobs
with different schedule_interval .
All these are very uncomfortable for me , adding a config or built-in method to
make execution_date equal to start_date. would be very nice . If this become
true, would only have one etl_end_time = "\{{
(execution_date.strftime('%Y-%m-%d 00:00:00') }}" .
was:
>From [https://airflow.apache.org/scheduler.html] :
> Note that if you run a DAG on a schedule_interval of one day, the run
> stamped 2016-01-01 will be trigger soon after 2016-01-01T23:59. In
> other words, the job instance is started once the period it covers has
> ended.
This feature is very hurt .
For example I have etl job which run every day, schedule_interval is `0 1 * *
*`, so it will trigger 2019-09-22 01:00:00 job on 2019-09-23 01:00:00 . But my
etl is processing all data before start_date , means data range is between
(history, 2019-09-23 00:00:00) , and we can't use `datetime.now()` because this
is unable to reproduce. This force me add 1 day to execution_date:
```python
etl_end_time = "\{{ (execution_date +
macros.timedelta(days=1)).strftime('%Y-%m-%d 00:00:00') }}"
```
However, when I need run a job with schedule_interval `45 2,3,4,5,6 * * *` ,
the `2019-09-22 06:45:00` job would run on `2019-09-23 02:45:00`, which is the
day after execution_date . Instead of adding a day, I had to changed
schedule_interval `45 2,3,4,5,6,7 * * *` and put a dummy operator on last run.
And in this situation , you don't need add one day to execution_date , this
means you have to define two `etl_end_time` to represent a same date in jobs
with different schedule_interval .
All these are very uncomfortable for me , adding a config or built-in method to
make execution_date equal to start_date. would be very nice .
> Add a flag to make scheduling trigger on start_date instead of execution_date
> (make execution_date equal to start_date)
> ---
>
> Key: AIRFLOW-5538
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5538
> Project: Apache Airflow
> Issue Type: Improvement
> Components: DagRun
>Affects Versions: 1.10.5
>Reporter: kasim
>Priority: Major
>
> From [https://airflow.apache.org/scheduler.html] :
> > Note that if you run a DAG on a schedule_interval of one day, the run
> > stamped 2016-01-01 will be trigger soon after 2016-01-01T23:59. In
> > other words, the job instance is started once the period it covers has
> > ended.
> This feature is very hurt .
> For example I have etl job which run every day, schedule_interval is `0 1 * *
> *`, so it will trigger 2019-09-22 01:00:00 job on 2019-09-23 01:00:00 . But
> my etl is processing all data before start_date , means data range is between
> (history, 2019-09-23 00:00:00) , and we can't use `datetime.now()` because
> this is unable to reproduce. This force me add 1 day to execution_date:
> ```python
> etl_end_time = "\{{ (execution_date +
> macros.timedelta(days=1)).strftime('%Y-%m-%d 00:00:00') }}"
> ```
> However, when I need run a job with schedule_interval `45 2,3,4,5,6 * * *` ,
> the `2019-09-22 06:45:00` job would run on `2019-09-23 02:45:00`, which is
> the day after execution_date . Instead of adding a day, I had to changed
> schedule_interval `45 2,3,4,5,6,7 * * *` and put a dummy operator on last run.
> And in this situation , you don't need add one day to
[jira] [Updated] (AIRFLOW-5538) Add a flag to make scheduling trigger on start_date instead of execution_date (make execution_date equal to start_date)
[
https://issues.apache.org/jira/browse/AIRFLOW-5538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kasim updated AIRFLOW-5538:
---
Description:
>From [https://airflow.apache.org/scheduler.html] :
> Note that if you run a DAG on a schedule_interval of one day, the run
> stamped 2016-01-01 will be trigger soon after 2016-01-01T23:59. In
> other words, the job instance is started once the period it covers has
> ended.
This feature is very hurt .
For example I have etl job which run every day, schedule_interval is `0 1 * *
*`, so it will trigger 2019-09-22 01:00:00 job on 2019-09-23 01:00:00 . But my
etl is processing all data before start_date , means data range is between
(history, 2019-09-23 00:00:00) , and we can't use `datetime.now()` because this
is unable to reproduce. This force me add 1 day to execution_date:
```python
etl_end_time = "\{{ (execution_date +
macros.timedelta(days=1)).strftime('%Y-%m-%d 00:00:00') }}"
```
However, when I need run a job with schedule_interval `45 2,3,4,5,6 * * *` ,
the `2019-09-22 06:45:00` job would run on `2019-09-23 02:45:00`, which is the
day after execution_date . Instead of adding a day, I had to changed
schedule_interval `45 2,3,4,5,6,7 * * *` and put a dummy operator on last run.
And in this situation , you don't need add one day to execution_date , this
means you have to define two `etl_end_time` to represent a same date in jobs
with different schedule_interval .
All these are very uncomfortable for me , adding a config or built-in method to
make execution_date equal to start_date. would be very nice .
was:
>From https://airflow.apache.org/scheduler.html :
> Note that if you run a DAG on a schedule_interval of one day, the run
> stamped 2016-01-01 will be trigger soon after 2016-01-01T23:59. In
> other words, the job instance is started once the period it covers has
> ended.
This feature is very hurt .
For example I have etl job which run every day, schedule_interval is `0 1 * *
*`, so it will trigger 2019-09-22 01:00:00 job on 2019-09-23 01:00:00 . But my
etl is processing all data before start_date , means data range is between
(history, 2019-09-23 00:00:00) , and we can't use `datetime.now()` because this
is unable to reproduce. This force me add 1 day to execution_date:
```python
etl_end_time = "\{{ (execution_date +
macros.timedelta(days=1)).strftime('%Y-%m-%d 00:00:00') }}"
```
However, when I need run a job with schedule_interval `45 2,3,4,5,6 * * *` ,
the `2019-09-22 06:45:00` job would run on `2019-09-23 02:45:00`, which is the
day after execution_date . Instead of adding a day, I had to changed
schedule_interval `45 2,3,4,5,6,7 * * *` and put a dummy operator on last run.
And in this situation , you don't need add one day to execution_date , this
means you have to define two `etl_end_time` to represent a same date in jobs
with different schedule_interval .
All these are very uncomfortable for me , is there any config or built-in
method to make execution_date equal to start_date ? Or I have to modify airflow
source code ...
> Add a flag to make scheduling trigger on start_date instead of execution_date
> (make execution_date equal to start_date)
> ---
>
> Key: AIRFLOW-5538
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5538
> Project: Apache Airflow
> Issue Type: Improvement
> Components: DagRun
>Affects Versions: 1.10.5
>Reporter: kasim
>Priority: Major
>
> From [https://airflow.apache.org/scheduler.html] :
> > Note that if you run a DAG on a schedule_interval of one day, the run
> > stamped 2016-01-01 will be trigger soon after 2016-01-01T23:59. In
> > other words, the job instance is started once the period it covers has
> > ended.
> This feature is very hurt .
> For example I have etl job which run every day, schedule_interval is `0 1 * *
> *`, so it will trigger 2019-09-22 01:00:00 job on 2019-09-23 01:00:00 . But
> my etl is processing all data before start_date , means data range is between
> (history, 2019-09-23 00:00:00) , and we can't use `datetime.now()` because
> this is unable to reproduce. This force me add 1 day to execution_date:
> ```python
> etl_end_time = "\{{ (execution_date +
> macros.timedelta(days=1)).strftime('%Y-%m-%d 00:00:00') }}"
> ```
> However, when I need run a job with schedule_interval `45 2,3,4,5,6 * * *` ,
> the `2019-09-22 06:45:00` job would run on `2019-09-23 02:45:00`, which is
> the day after execution_date . Instead of adding a day, I had to changed
> schedule_interval `45 2,3,4,5,6,7 * * *` and put a dummy operator on last run.
> And in this situation , you don't need add one day to execution_date , this
> means you have to define two `etl_end_time` to represent a same date in
[jira] [Created] (AIRFLOW-5538) Add a flag to make scheduling trigger on start_date instead of execution_date (make execution_date equal to start_date)
kasim created AIRFLOW-5538:
--
Summary: Add a flag to make scheduling trigger on start_date
instead of execution_date (make execution_date equal to start_date)
Key: AIRFLOW-5538
URL: https://issues.apache.org/jira/browse/AIRFLOW-5538
Project: Apache Airflow
Issue Type: Improvement
Components: DagRun
Affects Versions: 1.10.5
Reporter: kasim
>From https://airflow.apache.org/scheduler.html :
> Note that if you run a DAG on a schedule_interval of one day, the run
> stamped 2016-01-01 will be trigger soon after 2016-01-01T23:59. In
> other words, the job instance is started once the period it covers has
> ended.
This feature is very hurt .
For example I have etl job which run every day, schedule_interval is `0 1 * *
*`, so it will trigger 2019-09-22 01:00:00 job on 2019-09-23 01:00:00 . But my
etl is processing all data before start_date , means data range is between
(history, 2019-09-23 00:00:00) , and we can't use `datetime.now()` because this
is unable to reproduce. This force me add 1 day to execution_date:
```python
etl_end_time = "\{{ (execution_date +
macros.timedelta(days=1)).strftime('%Y-%m-%d 00:00:00') }}"
```
However, when I need run a job with schedule_interval `45 2,3,4,5,6 * * *` ,
the `2019-09-22 06:45:00` job would run on `2019-09-23 02:45:00`, which is the
day after execution_date . Instead of adding a day, I had to changed
schedule_interval `45 2,3,4,5,6,7 * * *` and put a dummy operator on last run.
And in this situation , you don't need add one day to execution_date , this
means you have to define two `etl_end_time` to represent a same date in jobs
with different schedule_interval .
All these are very uncomfortable for me , is there any config or built-in
method to make execution_date equal to start_date ? Or I have to modify airflow
source code ...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] codecov-io commented on issue #6130: [AIRFLOW-5508] Add a whitelist mechanism for the StatsD metrics
codecov-io commented on issue #6130: [AIRFLOW-5508] Add a whitelist mechanism for the StatsD metrics URL: https://github.com/apache/airflow/pull/6130#issuecomment-533939209 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6130?src=pr=h1) Report > Merging [#6130](https://codecov.io/gh/apache/airflow/pull/6130?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/30c442c9b8f4f98774841308a98f0e5ad1bce6a6?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `61.9%`. [](https://codecov.io/gh/apache/airflow/pull/6130?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6130 +/- ## == - Coverage 80.05% 80.05% -0.01% == Files 608 608 Lines 3503335048 +15 == + Hits2804728056 +9 - Misses 6986 6992 +6 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6130?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/stats.py](https://codecov.io/gh/apache/airflow/pull/6130/diff?src=pr=tree#diff-YWlyZmxvdy9zdGF0cy5weQ==) | `70% <61.9%> (+0.76%)` | :arrow_up: | | [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6130/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `85% <0%> (-5%)` | :arrow_down: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/6130/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `74.58% <0%> (+0.3%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6130?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6130?src=pr=footer). Last update [30c442c...609368d](https://codecov.io/gh/apache/airflow/pull/6130?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mikedias commented on issue #6130: [AIRFLOW-5508] Add a whitelist mechanism for the StatsD metrics
mikedias commented on issue #6130: [AIRFLOW-5508] Add a whitelist mechanism for the StatsD metrics URL: https://github.com/apache/airflow/pull/6130#issuecomment-533934474 Setting renamed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6167: [AIRFLOW-5533] Fixed failing CRON build
codecov-io edited a comment on issue #6167: [AIRFLOW-5533] Fixed failing CRON build URL: https://github.com/apache/airflow/pull/6167#issuecomment-533926688 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=h1) Report > Merging [#6167](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/4ed2c222c809e21663272c0c434bd7e69689522a?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6167 +/- ## == + Coverage 80.05% 80.06% +<.01% == Files 608 608 Lines 3503335033 == + Hits2804728048 +1 + Misses 6986 6985 -1 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6167/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `58.8% <0%> (-0.19%)` | :arrow_down: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/6167/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `74.58% <0%> (+0.3%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=footer). Last update [4ed2c22...b883e5a](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6167: [AIRFLOW-5533] Fixed failing CRON build
codecov-io edited a comment on issue #6167: [AIRFLOW-5533] Fixed failing CRON build URL: https://github.com/apache/airflow/pull/6167#issuecomment-533926688 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=h1) Report > Merging [#6167](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/4ed2c222c809e21663272c0c434bd7e69689522a?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6167 +/- ## == + Coverage 80.05% 80.06% +<.01% == Files 608 608 Lines 3503335033 == + Hits2804728048 +1 + Misses 6986 6985 -1 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6167/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `58.8% <0%> (-0.19%)` | :arrow_down: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/6167/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `74.58% <0%> (+0.3%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=footer). Last update [4ed2c22...b883e5a](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6167: [AIRFLOW-5533] Fixed failing CRON build
codecov-io edited a comment on issue #6167: [AIRFLOW-5533] Fixed failing CRON build URL: https://github.com/apache/airflow/pull/6167#issuecomment-533926688 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=h1) Report > Merging [#6167](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/4ed2c222c809e21663272c0c434bd7e69689522a?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6167 +/- ## == + Coverage 80.05% 80.06% +<.01% == Files 608 608 Lines 3503335033 == + Hits2804728048 +1 + Misses 6986 6985 -1 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6167/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `58.8% <0%> (-0.19%)` | :arrow_down: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/6167/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `74.58% <0%> (+0.3%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=footer). Last update [4ed2c22...b883e5a](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6167: [AIRFLOW-5533] Fixed failing CRON build
codecov-io edited a comment on issue #6167: [AIRFLOW-5533] Fixed failing CRON build URL: https://github.com/apache/airflow/pull/6167#issuecomment-533926688 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=h1) Report > Merging [#6167](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/4ed2c222c809e21663272c0c434bd7e69689522a?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6167 +/- ## == + Coverage 80.05% 80.06% +<.01% == Files 608 608 Lines 3503335033 == + Hits2804728048 +1 + Misses 6986 6985 -1 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6167/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `58.8% <0%> (-0.19%)` | :arrow_down: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/6167/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `74.58% <0%> (+0.3%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=footer). Last update [4ed2c22...b883e5a](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6167: [AIRFLOW-5533] Fixed failing CRON build
codecov-io commented on issue #6167: [AIRFLOW-5533] Fixed failing CRON build URL: https://github.com/apache/airflow/pull/6167#issuecomment-533926688 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=h1) Report > Merging [#6167](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/4ed2c222c809e21663272c0c434bd7e69689522a?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6167 +/- ## == + Coverage 80.05% 80.06% +<.01% == Files 608 608 Lines 3503335033 == + Hits2804728048 +1 + Misses 6986 6985 -1 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6167/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `58.8% <0%> (-0.19%)` | :arrow_down: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/6167/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `74.58% <0%> (+0.3%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=footer). Last update [4ed2c22...b883e5a](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6167: [AIRFLOW-5533] Fixed failing CRON build
codecov-io edited a comment on issue #6167: [AIRFLOW-5533] Fixed failing CRON build URL: https://github.com/apache/airflow/pull/6167#issuecomment-533926688 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=h1) Report > Merging [#6167](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/4ed2c222c809e21663272c0c434bd7e69689522a?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6167 +/- ## == + Coverage 80.05% 80.06% +<.01% == Files 608 608 Lines 3503335033 == + Hits2804728048 +1 + Misses 6986 6985 -1 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6167/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `58.8% <0%> (-0.19%)` | :arrow_down: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/6167/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `74.58% <0%> (+0.3%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=footer). Last update [4ed2c22...b883e5a](https://codecov.io/gh/apache/airflow/pull/6167?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-5536) Output log was removed on EXITING from sub-shell
[ https://issues.apache.org/jira/browse/AIRFLOW-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5536. --- Fix Version/s: 1.10.6 Resolution: Fixed > Output log was removed on EXITING from sub-shell > > > Key: AIRFLOW-5536 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5536 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0, 1.10.5 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 1.10.6 > > > Output log is now by default deleted at exit from the build scripts on CI > (via trap), This can be disabled but by default it is enabled and the way it > worked (handling EXIT signal) it was removed also when sub-shell > (hooks/build) exited (and there were still some writing to those files that > could happen afterwards). > Also in case of CRON build - when all the build cache was deleted, also the > temporary directory was deleted right after it was created (and trying to > write to the output log failed). Therefore the log is now simply created in a > temporary directory provided by the filesystem. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5535) Bad parameter used for ci verbosity
[ https://issues.apache.org/jira/browse/AIRFLOW-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5535. --- Fix Version/s: 1.10.6 Resolution: Fixed > Bad parameter used for ci verbosity > --- > > Key: AIRFLOW-5535 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5535 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0, 1.10.5 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 1.10.6 > > > Should be just VERBOSE in the host and AIRFLOW_CI_VERBOSE in container. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5533) CRON job fails when running static checks
[ https://issues.apache.org/jira/browse/AIRFLOW-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5533. --- Fix Version/s: 1.10.6 Resolution: Fixed > CRON job fails when running static checks > - > > Key: AIRFLOW-5533 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5533 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0, 1.10.5 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 1.10.6 > > > CRON job fails on static checks. > This is because the whole image is rebuilt from the scratch and it turned out > that while running static checks it was also rebuilding CI/CHECKLICENCE > images even though they were not needed (only CI_SLIM) images are needed for > static checks. > This was done in silent mode and took long time (because it is build from the > scratch) resulting in 10m inactivity. It also means that the builds on CI can > be still sped up because the images are also being built without a need in > regular jobs (but faster). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5534) Context Docker image running is verbose and leaves Docker container behind
[ https://issues.apache.org/jira/browse/AIRFLOW-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5534. --- Fix Version/s: 1.10.6 Resolution: Fixed > Context Docker image running is verbose and leaves Docker container behind > -- > > Key: AIRFLOW-5534 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5534 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0, 1.10.5 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 1.10.6 > > > The context docker image used to fix permissions in CI builds is a bit > verbose (output of docker build command) and leaves the container behind (no > --rm flag). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5533) CRON job fails when running static checks
[ https://issues.apache.org/jira/browse/AIRFLOW-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935396#comment-16935396 ] ASF subversion and git services commented on AIRFLOW-5533: -- Commit 8a53a90092d7232f0bd35cb7ce0cf7e95f74ed01 in airflow's branch refs/heads/v1-10-test from Jarek Potiuk [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=8a53a90 ] [AIRFLOW-5533] Fixed failing CRON build This change optimises further image building and removes unnecessary verbosity in building the images for CI builds. After this change is merged, only the necessary images are built for each type of check: * Tests -> only CI * Static checks (with/without pylint) -> Only CI_SLIM * Docs -> only CI_SLIM * Licence checks -> Only CHECKLICENCE Previously the right images only were built in ci_before_install.sh but then in case of static checks, the pre-commit build image step also rebuilt CHECKLICENCE and CI images - which was not necessary and very long in case of CRON job - this caused the CRON job to fail at 10m of inactivity. > CRON job fails when running static checks > - > > Key: AIRFLOW-5533 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5533 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0, 1.10.5 >Reporter: Jarek Potiuk >Priority: Major > > CRON job fails on static checks. > This is because the whole image is rebuilt from the scratch and it turned out > that while running static checks it was also rebuilding CI/CHECKLICENCE > images even though they were not needed (only CI_SLIM) images are needed for > static checks. > This was done in silent mode and took long time (because it is build from the > scratch) resulting in 10m inactivity. It also means that the builds on CI can > be still sped up because the images are also being built without a need in > regular jobs (but faster). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5537) Yamllint was installed in the host for CI and it is not needed
[ https://issues.apache.org/jira/browse/AIRFLOW-5537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935399#comment-16935399 ] ASF subversion and git services commented on AIRFLOW-5537: -- Commit 2dd2d0dab9b9a04f379431b73aaa74eadcfbc66d in airflow's branch refs/heads/v1-10-test from Jarek Potiuk [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=2dd2d0d ] [AIRFLOW-5537] Yamllint is not needed as dependency on host It used to be needed for pre-commits but is not needed any more as it is automatically installed as dependency in the virtualenv created by pre-commit > Yamllint was installed in the host for CI and it is not needed > -- > > Key: AIRFLOW-5537 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5537 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0, 1.10.5 >Reporter: Jarek Potiuk >Priority: Major > > Yamllint used to be required for pre-commits to work but it is not needed any > more (as it is installed by pre-commit as needed in the virtualenv created > for yamllint check) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5536) Output log was removed on EXITING from sub-shell
[ https://issues.apache.org/jira/browse/AIRFLOW-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935398#comment-16935398 ] ASF subversion and git services commented on AIRFLOW-5536: -- Commit e54e8271e7f45154caf6736e9fab7589b20eb287 in airflow's branch refs/heads/v1-10-test from Jarek Potiuk [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=e54e827 ] [AIRFLOW-5536] Better handling of temporary output files > Output log was removed on EXITING from sub-shell > > > Key: AIRFLOW-5536 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5536 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0, 1.10.5 >Reporter: Jarek Potiuk >Priority: Major > > Output log is now by default deleted at exit from the build scripts on CI > (via trap), This can be disabled but by default it is enabled and the way it > worked (handling EXIT signal) it was removed also when sub-shell > (hooks/build) exited (and there were still some writing to those files that > could happen afterwards). > Also in case of CRON build - when all the build cache was deleted, also the > temporary directory was deleted right after it was created (and trying to > write to the output log failed). Therefore the log is now simply created in a > temporary directory provided by the filesystem. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5537) Yamllint was installed in the host for CI and it is not needed
[ https://issues.apache.org/jira/browse/AIRFLOW-5537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5537. --- Fix Version/s: 1.10.6 Resolution: Fixed > Yamllint was installed in the host for CI and it is not needed > -- > > Key: AIRFLOW-5537 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5537 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0, 1.10.5 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 1.10.6 > > > Yamllint used to be required for pre-commits to work but it is not needed any > more (as it is installed by pre-commit as needed in the virtualenv created > for yamllint check) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5535) Bad parameter used for ci verbosity
[ https://issues.apache.org/jira/browse/AIRFLOW-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935397#comment-16935397 ] ASF subversion and git services commented on AIRFLOW-5535: -- Commit b5f12a75b906141ce019fa8f787c7efb6ef323f6 in airflow's branch refs/heads/v1-10-test from Jarek Potiuk [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=b5f12a7 ] [AIRFLOW-5535] Fix name of VERBOSE parameter > Bad parameter used for ci verbosity > --- > > Key: AIRFLOW-5535 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5535 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0, 1.10.5 >Reporter: Jarek Potiuk >Priority: Major > > Should be just VERBOSE in the host and AIRFLOW_CI_VERBOSE in container. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5533) CRON job fails when running static checks
[ https://issues.apache.org/jira/browse/AIRFLOW-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935391#comment-16935391 ] ASF subversion and git services commented on AIRFLOW-5533: -- Commit 1815ef32d86ed49d991faddb531d1b8408a11b0d in airflow's branch refs/heads/master from Jarek Potiuk [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=1815ef3 ] [AIRFLOW-5533] Fixed failing CRON build This change optimises further image building and removes unnecessary verbosity in building the images for CI builds. After this change is merged, only the necessary images are built for each type of check: * Tests -> only CI * Static checks (with/without pylint) -> Only CI_SLIM * Docs -> only CI_SLIM * Licence checks -> Only CHECKLICENCE Previously the right images only were built in ci_before_install.sh but then in case of static checks, the pre-commit build image step also rebuilt CHECKLICENCE and CI images - which was not necessary and very long in case of CRON job - this caused the CRON job to fail at 10m of inactivity. > CRON job fails when running static checks > - > > Key: AIRFLOW-5533 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5533 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0, 1.10.5 >Reporter: Jarek Potiuk >Priority: Major > > CRON job fails on static checks. > This is because the whole image is rebuilt from the scratch and it turned out > that while running static checks it was also rebuilding CI/CHECKLICENCE > images even though they were not needed (only CI_SLIM) images are needed for > static checks. > This was done in silent mode and took long time (because it is build from the > scratch) resulting in 10m inactivity. It also means that the builds on CI can > be still sped up because the images are also being built without a need in > regular jobs (but faster). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5535) Bad parameter used for ci verbosity
[ https://issues.apache.org/jira/browse/AIRFLOW-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935392#comment-16935392 ] ASF subversion and git services commented on AIRFLOW-5535: -- Commit 511615c884a09cd95d1e74a748cc10d6d9e9013d in airflow's branch refs/heads/master from Jarek Potiuk [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=511615c ] [AIRFLOW-5535] Fix name of VERBOSE parameter > Bad parameter used for ci verbosity > --- > > Key: AIRFLOW-5535 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5535 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0, 1.10.5 >Reporter: Jarek Potiuk >Priority: Major > > Should be just VERBOSE in the host and AIRFLOW_CI_VERBOSE in container. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5537) Yamllint was installed in the host for CI and it is not needed
[ https://issues.apache.org/jira/browse/AIRFLOW-5537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935394#comment-16935394 ] ASF subversion and git services commented on AIRFLOW-5537: -- Commit 30c442c9b8f4f98774841308a98f0e5ad1bce6a6 in airflow's branch refs/heads/master from Jarek Potiuk [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=30c442c ] [AIRFLOW-5537] Yamllint is not needed as dependency on host It used to be needed for pre-commits but is not needed any more as it is automatically installed as dependency in the virtualenv created by pre-commit > Yamllint was installed in the host for CI and it is not needed > -- > > Key: AIRFLOW-5537 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5537 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0, 1.10.5 >Reporter: Jarek Potiuk >Priority: Major > > Yamllint used to be required for pre-commits to work but it is not needed any > more (as it is installed by pre-commit as needed in the virtualenv created > for yamllint check) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5534) Context Docker image running is verbose and leaves Docker container behind
[ https://issues.apache.org/jira/browse/AIRFLOW-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935390#comment-16935390 ] ASF subversion and git services commented on AIRFLOW-5534: -- Commit 0ccb9256d7c34b13635f0e12ff19eb80019f69f5 in airflow's branch refs/heads/master from Jarek Potiuk [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=0ccb925 ] [AIRFLOW-5534] Less verobosity and removal of context container > Context Docker image running is verbose and leaves Docker container behind > -- > > Key: AIRFLOW-5534 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5534 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0, 1.10.5 >Reporter: Jarek Potiuk >Priority: Major > > The context docker image used to fix permissions in CI builds is a bit > verbose (output of docker build command) and leaves the container behind (no > --rm flag). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5533) CRON job fails when running static checks
[ https://issues.apache.org/jira/browse/AIRFLOW-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935389#comment-16935389 ] ASF GitHub Bot commented on AIRFLOW-5533: - potiuk commented on pull request #6167: [AIRFLOW-5533] Fixed failing CRON build URL: https://github.com/apache/airflow/pull/6167 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > CRON job fails when running static checks > - > > Key: AIRFLOW-5533 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5533 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0, 1.10.5 >Reporter: Jarek Potiuk >Priority: Major > > CRON job fails on static checks. > This is because the whole image is rebuilt from the scratch and it turned out > that while running static checks it was also rebuilding CI/CHECKLICENCE > images even though they were not needed (only CI_SLIM) images are needed for > static checks. > This was done in silent mode and took long time (because it is build from the > scratch) resulting in 10m inactivity. It also means that the builds on CI can > be still sped up because the images are also being built without a need in > regular jobs (but faster). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk merged pull request #6167: [AIRFLOW-5533] Fixed failing CRON build
potiuk merged pull request #6167: [AIRFLOW-5533] Fixed failing CRON build URL: https://github.com/apache/airflow/pull/6167 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #6167: [AIRFLOW-5533] Fixed failing CRON build
potiuk commented on a change in pull request #6167: [AIRFLOW-5533] Fixed
failing CRON build
URL: https://github.com/apache/airflow/pull/6167#discussion_r326912751
##
File path: scripts/ci/_utils.sh
##
@@ -885,3 +901,32 @@ function rebuild_all_images_if_needed_and_confirmed() {
fi
fi
}
+
+function build_image_on_ci() {
+if [[ "${CI:=}" != "true" ]]; then
+print_info
+print_info "Cleaning up docker installation!!"
+print_info
+"${AIRFLOW_SOURCES}/confirm" "Cleaning docker data and rebuilding"
+fi
+
+export AIRFLOW_CONTAINER_FORCE_PULL_IMAGES="true"
+export FORCE_BUILD="true"
+export VERBOSE="${VERBOSE:="false"}"
+
+# Cleanup docker installation. It should be empty in CI but let's not risk
+docker system prune --all --force
+rm -rf "${BUILD_CACHE_DIR}"
+
+if [[ ${TRAVIS_JOB_NAME:=""} == "Tests"* ]]; then
+rebuild_ci_image_if_needed
+elif [[ ${TRAVIS_JOB_NAME} == "Check lic"* ]]; then
+rebuild_checklicence_image_if_needed
+else
Review comment:
Fixed :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-5537) Yamllint was installed in the host for CI and it is not needed
Jarek Potiuk created AIRFLOW-5537: - Summary: Yamllint was installed in the host for CI and it is not needed Key: AIRFLOW-5537 URL: https://issues.apache.org/jira/browse/AIRFLOW-5537 Project: Apache Airflow Issue Type: Bug Components: ci Affects Versions: 1.10.5, 2.0.0 Reporter: Jarek Potiuk Yamllint used to be required for pre-commits to work but it is not needed any more (as it is installed by pre-commit as needed in the virtualenv created for yamllint check) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5536) Output log was removed on EXITING from sub-shell
Jarek Potiuk created AIRFLOW-5536: - Summary: Output log was removed on EXITING from sub-shell Key: AIRFLOW-5536 URL: https://issues.apache.org/jira/browse/AIRFLOW-5536 Project: Apache Airflow Issue Type: Bug Components: ci Affects Versions: 1.10.5, 2.0.0 Reporter: Jarek Potiuk Output log is now by default deleted at exit from the build scripts on CI (via trap), This can be disabled but by default it is enabled and the way it worked (handling EXIT signal) it was removed also when sub-shell (hooks/build) exited (and there were still some writing to those files that could happen afterwards). Also in case of CRON build - when all the build cache was deleted, also the temporary directory was deleted right after it was created (and trying to write to the output log failed). Therefore the log is now simply created in a temporary directory provided by the filesystem. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5535) Bad parameter used for ci verbosity
Jarek Potiuk created AIRFLOW-5535: - Summary: Bad parameter used for ci verbosity Key: AIRFLOW-5535 URL: https://issues.apache.org/jira/browse/AIRFLOW-5535 Project: Apache Airflow Issue Type: Bug Components: ci Affects Versions: 1.10.5, 2.0.0 Reporter: Jarek Potiuk Should be just VERBOSE in the host and AIRFLOW_CI_VERBOSE in container. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] mik-laj commented on a change in pull request #6167: [AIRFLOW-5533] Fixed failing CRON build
mik-laj commented on a change in pull request #6167: [AIRFLOW-5533] Fixed
failing CRON build
URL: https://github.com/apache/airflow/pull/6167#discussion_r326912244
##
File path: scripts/ci/_utils.sh
##
@@ -885,3 +901,32 @@ function rebuild_all_images_if_needed_and_confirmed() {
fi
fi
}
+
+function build_image_on_ci() {
+if [[ "${CI:=}" != "true" ]]; then
+print_info
+print_info "Cleaning up docker installation!!"
+print_info
+"${AIRFLOW_SOURCES}/confirm" "Cleaning docker data and rebuilding"
+fi
+
+export AIRFLOW_CONTAINER_FORCE_PULL_IMAGES="true"
+export FORCE_BUILD="true"
+export VERBOSE="${VERBOSE:="false"}"
+
+# Cleanup docker installation. It should be empty in CI but let's not risk
+docker system prune --all --force
+rm -rf "${BUILD_CACHE_DIR}"
+
+if [[ ${TRAVIS_JOB_NAME:=""} == "Tests"* ]]; then
+rebuild_ci_image_if_needed
+elif [[ ${TRAVIS_JOB_NAME} == "Check lic"* ]]; then
+rebuild_checklicence_image_if_needed
+else
Review comment:
Can you specify the cases when this image is build and the other case
returns an error? This may allow easier script debugging.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-5534) Context Docker image running is verbose and leaves Docker container behind
Jarek Potiuk created AIRFLOW-5534: - Summary: Context Docker image running is verbose and leaves Docker container behind Key: AIRFLOW-5534 URL: https://issues.apache.org/jira/browse/AIRFLOW-5534 Project: Apache Airflow Issue Type: Bug Components: ci Affects Versions: 1.10.5, 2.0.0 Reporter: Jarek Potiuk The context docker image used to fix permissions in CI builds is a bit verbose (output of docker build command) and leaves the container behind (no --rm flag). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5533) CRON job fails when running static checks
[
https://issues.apache.org/jira/browse/AIRFLOW-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935373#comment-16935373
]
ASF GitHub Bot commented on AIRFLOW-5533:
-
potiuk commented on pull request #6167: [AIRFLOW-5533] Fixed failing CRON build
URL: https://github.com/apache/airflow/pull/6167
This change optimises further image building and removes unnecessary
verbosity in building the images for CI builds.
After this change is merged, only the necessary images are built for
each type of check:
* Tests -> only CI
* Static checks (with/without pylint) -> Only CI_SLIM
* Docs -> only CI_SLIM
* Licence checks -> Only CHECKLICENCE
Previously the right images only were built in ci_before_install.sh
but then in case of static checks, the pre-commit build image step
also rebuilt CHECKLICENCE and CI images - which was not necessary
and very long in case of CRON job - this caused the CRON job to
fail at 10m of inactivity.
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> CRON job fails when running static checks
> -
>
> Key: AIRFLOW-5533
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5533
> Project: Apache Airflow
> Issue Type: Bug
> Components: ci
>Affects Versions: 2.0.0, 1.10.5
>Reporter: Jarek Potiuk
>Priority: Major
>
> CRON job fails on static checks.
> This is because the whole image is rebuilt from the scratch and it turned out
> that while running static checks it was also rebuilding CI/CHECKLICENCE
> images even though they were not needed (only CI_SLIM) images are needed for
> static checks.
> This was done in silent mode and took long time (because it is build from the
> scratch) resulting in 10m inactivity. It also means that the builds on CI can
> be still sped up because the images are also being built without a need in
> regular jobs (but faster).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] potiuk opened a new pull request #6167: [AIRFLOW-5533] Fixed failing CRON build
potiuk opened a new pull request #6167: [AIRFLOW-5533] Fixed failing CRON build
URL: https://github.com/apache/airflow/pull/6167
This change optimises further image building and removes unnecessary
verbosity in building the images for CI builds.
After this change is merged, only the necessary images are built for
each type of check:
* Tests -> only CI
* Static checks (with/without pylint) -> Only CI_SLIM
* Docs -> only CI_SLIM
* Licence checks -> Only CHECKLICENCE
Previously the right images only were built in ci_before_install.sh
but then in case of static checks, the pre-commit build image step
also rebuilt CHECKLICENCE and CI images - which was not necessary
and very long in case of CRON job - this caused the CRON job to
fail at 10m of inactivity.
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-5533) CRON job fails when running static checks
Jarek Potiuk created AIRFLOW-5533: - Summary: CRON job fails when running static checks Key: AIRFLOW-5533 URL: https://issues.apache.org/jira/browse/AIRFLOW-5533 Project: Apache Airflow Issue Type: Bug Components: ci Affects Versions: 1.10.5, 2.0.0 Reporter: Jarek Potiuk CRON job fails on static checks. This is because the whole image is rebuilt from the scratch and it turned out that while running static checks it was also rebuilding CI/CHECKLICENCE images even though they were not needed (only CI_SLIM) images are needed for static checks. This was done in silent mode and took long time (because it is build from the scratch) resulting in 10m inactivity. It also means that the builds on CI can be still sped up because the images are also being built without a need in regular jobs (but faster). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] codecov-io edited a comment on issue #6166: [AIRFLOW-5532] Fix imagePullSecrets in pod created from k8s executor
codecov-io edited a comment on issue #6166: [AIRFLOW-5532] Fix imagePullSecrets in pod created from k8s executor URL: https://github.com/apache/airflow/pull/6166#issuecomment-533891119 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6166?src=pr=h1) Report > Merging [#6166](https://codecov.io/gh/apache/airflow/pull/6166?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/2d19a41208220994d6001833b2830026ee5a357d?src=pr=desc) will **increase** coverage by `0.13%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6166?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6166 +/- ## == + Coverage 79.93% 80.06% +0.13% == Files 608 608 Lines 3503035028 -2 == + Hits2800028046 +46 + Misses 7030 6982 -48 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6166?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/kubernetes/worker\_configuration.py](https://codecov.io/gh/apache/airflow/pull/6166/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3dvcmtlcl9jb25maWd1cmF0aW9uLnB5) | `99.25% <ø> (+2.85%)` | :arrow_up: | | [airflow/gcp/operators/kubernetes\_engine.py](https://codecov.io/gh/apache/airflow/pull/6166/diff?src=pr=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2t1YmVybmV0ZXNfZW5naW5lLnB5) | `97.26% <0%> (-1.32%)` | :arrow_down: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6166/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.72% <0%> (+0.5%)` | :arrow_up: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/6166/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `74.28% <0%> (+0.9%)` | :arrow_up: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6166/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `58.98% <0%> (+2.54%)` | :arrow_up: | | [airflow/executors/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/6166/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvX19pbml0X18ucHk=) | `67.34% <0%> (+4.08%)` | :arrow_up: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6166/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `93.22% <0%> (+6.77%)` | :arrow_up: | | [airflow/utils/log/colored\_log.py](https://codecov.io/gh/apache/airflow/pull/6166/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvY29sb3JlZF9sb2cucHk=) | `93.18% <0%> (+11.36%)` | :arrow_up: | | [airflow/executors/sequential\_executor.py](https://codecov.io/gh/apache/airflow/pull/6166/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvc2VxdWVudGlhbF9leGVjdXRvci5weQ==) | `100% <0%> (+52.38%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6166?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6166?src=pr=footer). Last update [2d19a41...e592485](https://codecov.io/gh/apache/airflow/pull/6166?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #6166: [AIRFLOW-5532] Fix imagePullSecrets in pod created from k8s executor
potiuk commented on issue #6166: [AIRFLOW-5532] Fix imagePullSecrets in pod created from k8s executor URL: https://github.com/apache/airflow/pull/6166#issuecomment-533900256 Restarted. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] shawnzhu commented on issue #6166: [AIRFLOW-5532] Fix imagePullSecrets in pod created from k8s executor
shawnzhu commented on issue #6166: [AIRFLOW-5532] Fix imagePullSecrets in pod created from k8s executor URL: https://github.com/apache/airflow/pull/6166#issuecomment-533894173 FYI: All PR build jobs passes except [Job #25643.8](https://travis-ci.org/apache/airflow/jobs/588119636#L5072) failed b/c of: > ```SHELL No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] sjmiller609 commented on issue #6079: [AIRFLOW-5448] Handle istio-proxy for Kubernetes Pods
sjmiller609 commented on issue #6079: [AIRFLOW-5448] Handle istio-proxy for Kubernetes Pods URL: https://github.com/apache/airflow/pull/6079#issuecomment-533893894 Yes I will, I have been meaning to get to it. Thank you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6166: [AIRFLOW-5532] Fix imagePullSecrets in pod created from k8s executor
codecov-io commented on issue #6166: [AIRFLOW-5532] Fix imagePullSecrets in pod created from k8s executor URL: https://github.com/apache/airflow/pull/6166#issuecomment-533891119 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6166?src=pr=h1) Report > Merging [#6166](https://codecov.io/gh/apache/airflow/pull/6166?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/2d19a41208220994d6001833b2830026ee5a357d?src=pr=desc) will **increase** coverage by `0.13%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6166?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6166 +/- ## == + Coverage 79.93% 80.06% +0.13% == Files 608 608 Lines 3503035031 +1 == + Hits2800028048 +48 + Misses 7030 6983 -47 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6166?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/kubernetes/worker\_configuration.py](https://codecov.io/gh/apache/airflow/pull/6166/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3dvcmtlcl9jb25maWd1cmF0aW9uLnB5) | `99.28% <100%> (+2.88%)` | :arrow_up: | | [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6166/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `85% <0%> (-5%)` | :arrow_down: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6166/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.72% <0%> (+0.5%)` | :arrow_up: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/6166/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `74.58% <0%> (+1.2%)` | :arrow_up: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6166/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `58.98% <0%> (+2.54%)` | :arrow_up: | | [airflow/executors/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/6166/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvX19pbml0X18ucHk=) | `67.34% <0%> (+4.08%)` | :arrow_up: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6166/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `93.22% <0%> (+6.77%)` | :arrow_up: | | [airflow/utils/log/colored\_log.py](https://codecov.io/gh/apache/airflow/pull/6166/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvY29sb3JlZF9sb2cucHk=) | `93.18% <0%> (+11.36%)` | :arrow_up: | | [airflow/executors/sequential\_executor.py](https://codecov.io/gh/apache/airflow/pull/6166/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvc2VxdWVudGlhbF9leGVjdXRvci5weQ==) | `100% <0%> (+52.38%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6166?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6166?src=pr=footer). Last update [2d19a41...e592485](https://codecov.io/gh/apache/airflow/pull/6166?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5435) Add fallback for connection's project id in GKEPodOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935337#comment-16935337 ] ASF GitHub Bot commented on AIRFLOW-5435: - mik-laj commented on pull request #6051: [AIRFLOW-5435] Add fallback for connection's project id in GKEPodOperator URL: https://github.com/apache/airflow/pull/6051 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add fallback for connection's project id in GKEPodOperator > -- > > Key: AIRFLOW-5435 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5435 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5435) Add fallback for connection's project id in GKEPodOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935339#comment-16935339 ] ASF subversion and git services commented on AIRFLOW-5435: -- Commit 4ed2c222c809e21663272c0c434bd7e69689522a in airflow's branch refs/heads/master from Kamil Breguła [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=4ed2c22 ] [AIRFLOW-5435] Add fallback for connection's project id in GKEPodOperator (#6051) * [AIRFLOW-5435] Add fallback for connection's project id in GKEPodOperator * fixup! [AIRFLOW-5435] Add fallback for connection's project id in GKEPodOperator > Add fallback for connection's project id in GKEPodOperator > -- > > Key: AIRFLOW-5435 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5435 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5435) Add fallback for connection's project id in GKEPodOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935341#comment-16935341 ] ASF subversion and git services commented on AIRFLOW-5435: -- Commit 4ed2c222c809e21663272c0c434bd7e69689522a in airflow's branch refs/heads/master from Kamil Breguła [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=4ed2c22 ] [AIRFLOW-5435] Add fallback for connection's project id in GKEPodOperator (#6051) * [AIRFLOW-5435] Add fallback for connection's project id in GKEPodOperator * fixup! [AIRFLOW-5435] Add fallback for connection's project id in GKEPodOperator > Add fallback for connection's project id in GKEPodOperator > -- > > Key: AIRFLOW-5435 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5435 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5435) Add fallback for connection's project id in GKEPodOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935340#comment-16935340 ] ASF subversion and git services commented on AIRFLOW-5435: -- Commit 4ed2c222c809e21663272c0c434bd7e69689522a in airflow's branch refs/heads/master from Kamil Breguła [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=4ed2c22 ] [AIRFLOW-5435] Add fallback for connection's project id in GKEPodOperator (#6051) * [AIRFLOW-5435] Add fallback for connection's project id in GKEPodOperator * fixup! [AIRFLOW-5435] Add fallback for connection's project id in GKEPodOperator > Add fallback for connection's project id in GKEPodOperator > -- > > Key: AIRFLOW-5435 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5435 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] mik-laj merged pull request #6051: [AIRFLOW-5435] Add fallback for connection's project id in GKEPodOperator
mik-laj merged pull request #6051: [AIRFLOW-5435] Add fallback for connection's project id in GKEPodOperator URL: https://github.com/apache/airflow/pull/6051 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] Fokko edited a comment on issue #6074: [AIRFLOW-5390] Remove provide context
Fokko edited a comment on issue #6074: [AIRFLOW-5390] Remove provide context URL: https://github.com/apache/airflow/pull/6074#issuecomment-533875001 I've rebased the branch, let's see if we can make Travis happy again. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] Fokko commented on issue #6074: [AIRFLOW-5390] Remove provide context
Fokko commented on issue #6074: [AIRFLOW-5390] Remove provide context URL: https://github.com/apache/airflow/pull/6074#issuecomment-533875001 I've rebased the branch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5531) Replace deprecated log.warn() with log.warning()
[ https://issues.apache.org/jira/browse/AIRFLOW-5531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935292#comment-16935292 ] ASF subversion and git services commented on AIRFLOW-5531: -- Commit b683e2334e15a766a9792422b55b209f7c0f19c2 in airflow's branch refs/heads/v1-10-test from Kaxil Naik [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=b683e23 ] [AIRFLOW-5531] Replace deprecated log.warn() with log.warning() (#6165) - Add pre-commit hook to detect this (cherry picked from commit ccf44086c55b27325bfdd9e76ad1698f70d35cde) > Replace deprecated log.warn() with log.warning() > > > Key: AIRFLOW-5531 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5531 > Project: Apache Airflow > Issue Type: Improvement > Components: logging >Affects Versions: 2.0.0, 1.10.5 >Reporter: Kaxil Naik >Assignee: Kaxil Naik >Priority: Minor > Fix For: 2.0.0, 1.10.6 > > > We should replace *log.warn* which is deprecated with *log.warning* -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] benjamingrenier commented on issue #5565: [AIRFLOW-4899] Fix get_dataset_list from bigquery hook to return next…
benjamingrenier commented on issue #5565: [AIRFLOW-4899] Fix get_dataset_list from bigquery hook to return next… URL: https://github.com/apache/airflow/pull/5565#issuecomment-533874400 > Will approve after rebase. Rebase done, but I don't understand why there are errors ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #5566: [AIRFLOW-4935] Add method in the bigquery hook to list tables in a dataset
codecov-io commented on issue #5566: [AIRFLOW-4935] Add method in the bigquery hook to list tables in a dataset URL: https://github.com/apache/airflow/pull/5566#issuecomment-533872947 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5566?src=pr=h1) Report > Merging [#5566](https://codecov.io/gh/apache/airflow/pull/5566?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/2d19a41208220994d6001833b2830026ee5a357d?src=pr=desc) will **increase** coverage by `0.13%`. > The diff coverage is `94.44%`. [](https://codecov.io/gh/apache/airflow/pull/5566?src=pr=tree) ```diff @@Coverage Diff @@ ## master#5566 +/- ## == + Coverage 79.93% 80.06% +0.13% == Files 608 608 Lines 3503035048 +18 == + Hits2800028062 +62 + Misses 7030 6986 -44 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5566?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/5566/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=) | `70.85% <94.44%> (+0.64%)` | :arrow_up: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/5566/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.72% <0%> (+0.5%)` | :arrow_up: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/5566/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `74.28% <0%> (+0.9%)` | :arrow_up: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/5566/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `58.98% <0%> (+2.54%)` | :arrow_up: | | [airflow/executors/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/5566/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvX19pbml0X18ucHk=) | `67.34% <0%> (+4.08%)` | :arrow_up: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/5566/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `93.22% <0%> (+6.77%)` | :arrow_up: | | [airflow/utils/log/colored\_log.py](https://codecov.io/gh/apache/airflow/pull/5566/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvY29sb3JlZF9sb2cucHk=) | `93.18% <0%> (+11.36%)` | :arrow_up: | | [airflow/executors/sequential\_executor.py](https://codecov.io/gh/apache/airflow/pull/5566/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvc2VxdWVudGlhbF9leGVjdXRvci5weQ==) | `100% <0%> (+52.38%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5566?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5566?src=pr=footer). Last update [2d19a41...e1d8ecd](https://codecov.io/gh/apache/airflow/pull/5566?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] jsurloppe commented on issue #5680: [AIRFLOW-5066] allow k8s fieldref substitution
jsurloppe commented on issue #5680: [AIRFLOW-5066] allow k8s fieldref substitution URL: https://github.com/apache/airflow/pull/5680#issuecomment-533871887 correct me if i'm wrong, but it's seems that the errors are related to the current state of the `v1-10-stable` branch, the code and tests introduced in that PR passes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5531) Replace deprecated log.warn() with log.warning()
[ https://issues.apache.org/jira/browse/AIRFLOW-5531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935279#comment-16935279 ] ASF subversion and git services commented on AIRFLOW-5531: -- Commit f4beb40d6489c6467a769a5db8cb88b324650e05 in airflow's branch refs/heads/v1-10-test from Kaxil Naik [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=f4beb40 ] [AIRFLOW-5531] Replace deprecated log.warn() with log.warning() (#6165) - Add pre-commit hook to detect this (cherry picked from commit ccf44086c55b27325bfdd9e76ad1698f70d35cde) > Replace deprecated log.warn() with log.warning() > > > Key: AIRFLOW-5531 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5531 > Project: Apache Airflow > Issue Type: Improvement > Components: logging >Affects Versions: 2.0.0, 1.10.5 >Reporter: Kaxil Naik >Assignee: Kaxil Naik >Priority: Minor > Fix For: 2.0.0, 1.10.6 > > > We should replace *log.warn* which is deprecated with *log.warning* -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-3388) AWS Batch Operator, add support to Array Jobs
[ https://issues.apache.org/jira/browse/AIRFLOW-3388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula closed AIRFLOW-3388. -- Fix Version/s: 1.10.6 Resolution: Fixed > AWS Batch Operator, add support to Array Jobs > - > > Key: AIRFLOW-3388 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3388 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Hugo Prudente >Assignee: Hugo Prudente >Priority: Major > Fix For: 1.10.6 > > > Add support to Array Jobs -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-3388) AWS Batch Operator, add support to Array Jobs
[ https://issues.apache.org/jira/browse/AIRFLOW-3388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935267#comment-16935267 ] ASF subversion and git services commented on AIRFLOW-3388: -- Commit 2d19a41208220994d6001833b2830026ee5a357d in airflow's branch refs/heads/master from Peng Hou [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=2d19a41 ] [AIRFLOW-3388] Add support to Array Jobs for AWS Batch Operator (#6153) > AWS Batch Operator, add support to Array Jobs > - > > Key: AIRFLOW-3388 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3388 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Hugo Prudente >Assignee: Hugo Prudente >Priority: Major > > Add support to Array Jobs -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-3388) AWS Batch Operator, add support to Array Jobs
[ https://issues.apache.org/jira/browse/AIRFLOW-3388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935266#comment-16935266 ] ASF GitHub Bot commented on AIRFLOW-3388: - mik-laj commented on pull request #6153: [AIRFLOW-3388] Add support to Array Jobs for AWS Batch Operator URL: https://github.com/apache/airflow/pull/6153 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > AWS Batch Operator, add support to Array Jobs > - > > Key: AIRFLOW-3388 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3388 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Hugo Prudente >Assignee: Hugo Prudente >Priority: Major > > Add support to Array Jobs -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] mik-laj merged pull request #6153: [AIRFLOW-3388] Add support to Array Jobs for AWS Batch Operator
mik-laj merged pull request #6153: [AIRFLOW-3388] Add support to Array Jobs for AWS Batch Operator URL: https://github.com/apache/airflow/pull/6153 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] bharathpalaksha commented on issue #6026: [AIRFLOW-5419] - Use `sudo` to kill cleared tasks when running with impersonation
bharathpalaksha commented on issue #6026: [AIRFLOW-5419] - Use `sudo` to kill cleared tasks when running with impersonation URL: https://github.com/apache/airflow/pull/6026#issuecomment-533863633 @ashb please look into this This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-5531) Replace deprecated log.warn() with log.warning()
[ https://issues.apache.org/jira/browse/AIRFLOW-5531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5531. --- Resolution: Fixed > Replace deprecated log.warn() with log.warning() > > > Key: AIRFLOW-5531 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5531 > Project: Apache Airflow > Issue Type: Improvement > Components: logging >Affects Versions: 2.0.0, 1.10.5 >Reporter: Kaxil Naik >Assignee: Kaxil Naik >Priority: Minor > Fix For: 2.0.0, 1.10.6 > > > We should replace *log.warn* which is deprecated with *log.warning* -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5531) Replace deprecated log.warn() with log.warning()
[ https://issues.apache.org/jira/browse/AIRFLOW-5531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935253#comment-16935253 ] ASF subversion and git services commented on AIRFLOW-5531: -- Commit 2669343f276a2c5213bcc967ef0e2ad0742d3fcf in airflow's branch refs/heads/v1-10-test from Kaxil Naik [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=2669343 ] [AIRFLOW-5531] Replace deprecated log.warn() with log.warning() (#6165) - Add pre-commit hook to detect this (cherry picked from commit ccf44086c55b27325bfdd9e76ad1698f70d35cde) > Replace deprecated log.warn() with log.warning() > > > Key: AIRFLOW-5531 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5531 > Project: Apache Airflow > Issue Type: Improvement > Components: logging >Affects Versions: 2.0.0, 1.10.5 >Reporter: Kaxil Naik >Assignee: Kaxil Naik >Priority: Minor > Fix For: 2.0.0, 1.10.6 > > > We should replace *log.warn* which is deprecated with *log.warning* -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] codecov-io edited a comment on issue #5998: [AIRFLOW-5398] Update contrib example DAGs to context manager
codecov-io edited a comment on issue #5998: [AIRFLOW-5398] Update contrib example DAGs to context manager URL: https://github.com/apache/airflow/pull/5998#issuecomment-531900151 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5998?src=pr=h1) Report > Merging [#5998](https://codecov.io/gh/apache/airflow/pull/5998?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/25c53b0a10f078417a82be7acbc6485e7a38c9a6?src=pr=desc) will **decrease** coverage by `0.03%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/airflow/pull/5998?src=pr=tree) ```diff @@Coverage Diff @@ ## master#5998 +/- ## == - Coverage 80.09% 80.06% -0.04% == Files 596 609 +13 Lines 3487235024 +152 == + Hits2793228042 +110 - Misses 6940 6982 +42 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5998?src=pr=tree) | Coverage Δ | | |---|---|---| | [...ontrib/example\_dags/example\_databricks\_operator.py](https://codecov.io/gh/apache/airflow/pull/5998/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX2RhdGFicmlja3Nfb3BlcmF0b3IucHk=) | `0% <0%> (ø)` | :arrow_up: | | [...dags/example\_azure\_container\_instances\_operator.py](https://codecov.io/gh/apache/airflow/pull/5998/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX2F6dXJlX2NvbnRhaW5lcl9pbnN0YW5jZXNfb3BlcmF0b3IucHk=) | `0% <0%> (ø)` | :arrow_up: | | [...example\_dags/example\_kubernetes\_executor\_config.py](https://codecov.io/gh/apache/airflow/pull/5998/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX2t1YmVybmV0ZXNfZXhlY3V0b3JfY29uZmlnLnB5) | `0% <0%> (ø)` | :arrow_up: | | [...ow/contrib/example\_dags/example\_qubole\_operator.py](https://codecov.io/gh/apache/airflow/pull/5998/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3F1Ym9sZV9vcGVyYXRvci5weQ==) | `0% <0%> (ø)` | :arrow_up: | | [...ample\_dags/example\_emr\_job\_flow\_automatic\_steps.py](https://codecov.io/gh/apache/airflow/pull/5998/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX2Vtcl9qb2JfZmxvd19hdXRvbWF0aWNfc3RlcHMucHk=) | `0% <0%> (ø)` | :arrow_up: | | [...trib/example\_dags/example\_azure\_cosmosdb\_sensor.py](https://codecov.io/gh/apache/airflow/pull/5998/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX2F6dXJlX2Nvc21vc2RiX3NlbnNvci5weQ==) | `0% <0%> (ø)` | :arrow_up: | | [...flow/contrib/example\_dags/example\_qubole\_sensor.py](https://codecov.io/gh/apache/airflow/pull/5998/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3F1Ym9sZV9zZW5zb3IucHk=) | `0% <0%> (ø)` | :arrow_up: | | [.../contrib/example\_dags/example\_dingding\_operator.py](https://codecov.io/gh/apache/airflow/pull/5998/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX2RpbmdkaW5nX29wZXJhdG9yLnB5) | `0% <0%> (ø)` | :arrow_up: | | [...contrib/example\_dags/example\_papermill\_operator.py](https://codecov.io/gh/apache/airflow/pull/5998/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3BhcGVybWlsbF9vcGVyYXRvci5weQ==) | `0% <0%> (ø)` | | | [...low/contrib/example\_dags/example\_winrm\_operator.py](https://codecov.io/gh/apache/airflow/pull/5998/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3dpbnJtX29wZXJhdG9yLnB5) | `0% <0%> (ø)` | :arrow_up: | | ... and [109 more](https://codecov.io/gh/apache/airflow/pull/5998/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5998?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5998?src=pr=footer). Last update [25c53b0...a9b8393](https://codecov.io/gh/apache/airflow/pull/5998?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #5998: [AIRFLOW-5398] Update contrib example DAGs to context manager
codecov-io edited a comment on issue #5998: [AIRFLOW-5398] Update contrib example DAGs to context manager URL: https://github.com/apache/airflow/pull/5998#issuecomment-531900151 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5998?src=pr=h1) Report > Merging [#5998](https://codecov.io/gh/apache/airflow/pull/5998?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/25c53b0a10f078417a82be7acbc6485e7a38c9a6?src=pr=desc) will **decrease** coverage by `0.03%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/airflow/pull/5998?src=pr=tree) ```diff @@Coverage Diff @@ ## master#5998 +/- ## == - Coverage 80.09% 80.06% -0.04% == Files 596 609 +13 Lines 3487235024 +152 == + Hits2793228042 +110 - Misses 6940 6982 +42 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5998?src=pr=tree) | Coverage Δ | | |---|---|---| | [...ontrib/example\_dags/example\_databricks\_operator.py](https://codecov.io/gh/apache/airflow/pull/5998/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX2RhdGFicmlja3Nfb3BlcmF0b3IucHk=) | `0% <0%> (ø)` | :arrow_up: | | [...dags/example\_azure\_container\_instances\_operator.py](https://codecov.io/gh/apache/airflow/pull/5998/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX2F6dXJlX2NvbnRhaW5lcl9pbnN0YW5jZXNfb3BlcmF0b3IucHk=) | `0% <0%> (ø)` | :arrow_up: | | [...example\_dags/example\_kubernetes\_executor\_config.py](https://codecov.io/gh/apache/airflow/pull/5998/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX2t1YmVybmV0ZXNfZXhlY3V0b3JfY29uZmlnLnB5) | `0% <0%> (ø)` | :arrow_up: | | [...ow/contrib/example\_dags/example\_qubole\_operator.py](https://codecov.io/gh/apache/airflow/pull/5998/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3F1Ym9sZV9vcGVyYXRvci5weQ==) | `0% <0%> (ø)` | :arrow_up: | | [...ample\_dags/example\_emr\_job\_flow\_automatic\_steps.py](https://codecov.io/gh/apache/airflow/pull/5998/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX2Vtcl9qb2JfZmxvd19hdXRvbWF0aWNfc3RlcHMucHk=) | `0% <0%> (ø)` | :arrow_up: | | [...trib/example\_dags/example\_azure\_cosmosdb\_sensor.py](https://codecov.io/gh/apache/airflow/pull/5998/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX2F6dXJlX2Nvc21vc2RiX3NlbnNvci5weQ==) | `0% <0%> (ø)` | :arrow_up: | | [...flow/contrib/example\_dags/example\_qubole\_sensor.py](https://codecov.io/gh/apache/airflow/pull/5998/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3F1Ym9sZV9zZW5zb3IucHk=) | `0% <0%> (ø)` | :arrow_up: | | [.../contrib/example\_dags/example\_dingding\_operator.py](https://codecov.io/gh/apache/airflow/pull/5998/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX2RpbmdkaW5nX29wZXJhdG9yLnB5) | `0% <0%> (ø)` | :arrow_up: | | [...contrib/example\_dags/example\_papermill\_operator.py](https://codecov.io/gh/apache/airflow/pull/5998/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3BhcGVybWlsbF9vcGVyYXRvci5weQ==) | `0% <0%> (ø)` | | | [...low/contrib/example\_dags/example\_winrm\_operator.py](https://codecov.io/gh/apache/airflow/pull/5998/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3dpbnJtX29wZXJhdG9yLnB5) | `0% <0%> (ø)` | :arrow_up: | | ... and [109 more](https://codecov.io/gh/apache/airflow/pull/5998/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5998?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5998?src=pr=footer). Last update [25c53b0...a9b8393](https://codecov.io/gh/apache/airflow/pull/5998?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5531) Replace deprecated log.warn() with log.warning()
[ https://issues.apache.org/jira/browse/AIRFLOW-5531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935246#comment-16935246 ] ASF subversion and git services commented on AIRFLOW-5531: -- Commit ccf44086c55b27325bfdd9e76ad1698f70d35cde in airflow's branch refs/heads/master from Kaxil Naik [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=ccf4408 ] [AIRFLOW-5531] Replace deprecated log.warn() with log.warning() (#6165) - Add pre-commit hook to detect this > Replace deprecated log.warn() with log.warning() > > > Key: AIRFLOW-5531 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5531 > Project: Apache Airflow > Issue Type: Improvement > Components: logging >Affects Versions: 2.0.0, 1.10.5 >Reporter: Kaxil Naik >Assignee: Kaxil Naik >Priority: Minor > Fix For: 2.0.0, 1.10.6 > > > We should replace *log.warn* which is deprecated with *log.warning* -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5531) Replace deprecated log.warn() with log.warning()
[ https://issues.apache.org/jira/browse/AIRFLOW-5531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935245#comment-16935245 ] ASF GitHub Bot commented on AIRFLOW-5531: - potiuk commented on pull request #6165: [AIRFLOW-5531] Replace deprecated log.warn() with log.warning() URL: https://github.com/apache/airflow/pull/6165 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Replace deprecated log.warn() with log.warning() > > > Key: AIRFLOW-5531 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5531 > Project: Apache Airflow > Issue Type: Improvement > Components: logging >Affects Versions: 2.0.0, 1.10.5 >Reporter: Kaxil Naik >Assignee: Kaxil Naik >Priority: Minor > Fix For: 2.0.0, 1.10.6 > > > We should replace *log.warn* which is deprecated with *log.warning* -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] codecov-io edited a comment on issue #5998: [AIRFLOW-5398] Update contrib example DAGs to context manager
codecov-io edited a comment on issue #5998: [AIRFLOW-5398] Update contrib example DAGs to context manager URL: https://github.com/apache/airflow/pull/5998#issuecomment-531900151 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5998?src=pr=h1) Report > Merging [#5998](https://codecov.io/gh/apache/airflow/pull/5998?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/25c53b0a10f078417a82be7acbc6485e7a38c9a6?src=pr=desc) will **decrease** coverage by `0.02%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/airflow/pull/5998?src=pr=tree) ```diff @@Coverage Diff @@ ## master#5998 +/- ## == - Coverage 80.09% 80.07% -0.03% == Files 596 609 +13 Lines 3487235024 +152 == + Hits2793228046 +114 - Misses 6940 6978 +38 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5998?src=pr=tree) | Coverage Δ | | |---|---|---| | [...ontrib/example\_dags/example\_databricks\_operator.py](https://codecov.io/gh/apache/airflow/pull/5998/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX2RhdGFicmlja3Nfb3BlcmF0b3IucHk=) | `0% <0%> (ø)` | :arrow_up: | | [...dags/example\_azure\_container\_instances\_operator.py](https://codecov.io/gh/apache/airflow/pull/5998/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX2F6dXJlX2NvbnRhaW5lcl9pbnN0YW5jZXNfb3BlcmF0b3IucHk=) | `0% <0%> (ø)` | :arrow_up: | | [...example\_dags/example\_kubernetes\_executor\_config.py](https://codecov.io/gh/apache/airflow/pull/5998/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX2t1YmVybmV0ZXNfZXhlY3V0b3JfY29uZmlnLnB5) | `0% <0%> (ø)` | :arrow_up: | | [...ow/contrib/example\_dags/example\_qubole\_operator.py](https://codecov.io/gh/apache/airflow/pull/5998/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3F1Ym9sZV9vcGVyYXRvci5weQ==) | `0% <0%> (ø)` | :arrow_up: | | [...ample\_dags/example\_emr\_job\_flow\_automatic\_steps.py](https://codecov.io/gh/apache/airflow/pull/5998/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX2Vtcl9qb2JfZmxvd19hdXRvbWF0aWNfc3RlcHMucHk=) | `0% <0%> (ø)` | :arrow_up: | | [...trib/example\_dags/example\_azure\_cosmosdb\_sensor.py](https://codecov.io/gh/apache/airflow/pull/5998/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX2F6dXJlX2Nvc21vc2RiX3NlbnNvci5weQ==) | `0% <0%> (ø)` | :arrow_up: | | [...flow/contrib/example\_dags/example\_qubole\_sensor.py](https://codecov.io/gh/apache/airflow/pull/5998/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3F1Ym9sZV9zZW5zb3IucHk=) | `0% <0%> (ø)` | :arrow_up: | | [.../contrib/example\_dags/example\_dingding\_operator.py](https://codecov.io/gh/apache/airflow/pull/5998/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX2RpbmdkaW5nX29wZXJhdG9yLnB5) | `0% <0%> (ø)` | :arrow_up: | | [...contrib/example\_dags/example\_papermill\_operator.py](https://codecov.io/gh/apache/airflow/pull/5998/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3BhcGVybWlsbF9vcGVyYXRvci5weQ==) | `0% <0%> (ø)` | | | [...low/contrib/example\_dags/example\_winrm\_operator.py](https://codecov.io/gh/apache/airflow/pull/5998/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3dpbnJtX29wZXJhdG9yLnB5) | `0% <0%> (ø)` | :arrow_up: | | ... and [108 more](https://codecov.io/gh/apache/airflow/pull/5998/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5998?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5998?src=pr=footer). Last update [25c53b0...a9b8393](https://codecov.io/gh/apache/airflow/pull/5998?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
