[GitHub] chang commented on a change in pull request #4028: [AIRFLOW-3099] Don't ever warn about missing sections of config
chang commented on a change in pull request #4028: [AIRFLOW-3099] Don't ever
warn about missing sections of config
URL: https://github.com/apache/incubator-airflow/pull/4028#discussion_r223923848
##
File path: airflow/bin/cli.py
##
@@ -519,25 +500,7 @@ def run(args, dag=None):
if os.path.exists(args.cfg_path):
os.remove(args.cfg_path)
-# Do not log these properties since some may contain passwords.
-# This may also set default values for database properties like
-# core.sql_alchemy_pool_size
-# core.sql_alchemy_pool_recycle
-for section, config in conf_dict.items():
-for option, value in config.items():
-try:
-conf.set(section, option, value)
-except NoSectionError:
-no_section_msg = (
-'Section {section} Option {option} '
-'does not exist in the config!'
-).format(section=section, option=option)
-
-if section in OPTIONAL_AIRFLOW_CFG_SECTIONS:
-log.debug(no_section_msg)
-else:
-log.error(no_section_msg)
-
+conf.conf.read_dict(conf_dict, source=conf_file)
Review comment:
Also, is this `source=conf_file` intentional? If so I believe it needs to be
in the `with as conf_file` context manager above.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] dima-asana commented on a change in pull request #4010: [AIRFLOW-1837] Respect task start_date when different from dag's
dima-asana commented on a change in pull request #4010: [AIRFLOW-1837] Respect
task start_date when different from dag's
URL: https://github.com/apache/incubator-airflow/pull/4010#discussion_r223922901
##
File path: tests/www_rbac/test_views.py
##
@@ -267,8 +267,8 @@ def test_mount(self):
class TestAirflowBaseViews(TestBase):
-default_date = timezone.datetime(2018, 3, 1)
-run_id = "test_{}".format(models.DagRun.id_for_date(default_date))
+EXAMPLE_DAG_DEFAULT_DATE = dates.days_ago(2)
Review comment:
Yes, it's needed.
This date gets called in e.g.
https://github.com/apache/incubator-airflow/pull/4010/files/49ca7ad528cec88503bd011ca0c71218b168a05c#diff-8919d07682d67296863b9c62a96af963L298
That line creates a DagRun for previously 2018-03-01.
The example DAG that this test is testing has a start date of 2 day ago,
e.g. 2018-10-07.
Before this change, task instances would get created regardless of the DAG
start date. After this change, only task instances after the DAG start date
get created. So if I kept the DagRun for 2018-03-01, the instances being
tested wouldn't get created.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] chang commented on a change in pull request #4028: [AIRFLOW-3099] Don't ever warn about missing sections of config
chang commented on a change in pull request #4028: [AIRFLOW-3099] Don't ever
warn about missing sections of config
URL: https://github.com/apache/incubator-airflow/pull/4028#discussion_r223922633
##
File path: airflow/bin/cli.py
##
@@ -519,25 +500,7 @@ def run(args, dag=None):
if os.path.exists(args.cfg_path):
os.remove(args.cfg_path)
-# Do not log these properties since some may contain passwords.
-# This may also set default values for database properties like
-# core.sql_alchemy_pool_size
-# core.sql_alchemy_pool_recycle
-for section, config in conf_dict.items():
-for option, value in config.items():
-try:
-conf.set(section, option, value)
-except NoSectionError:
-no_section_msg = (
-'Section {section} Option {option} '
-'does not exist in the config!'
-).format(section=section, option=option)
-
-if section in OPTIONAL_AIRFLOW_CFG_SECTIONS:
-log.debug(no_section_msg)
-else:
-log.error(no_section_msg)
-
+conf.conf.read_dict(conf_dict, source=conf_file)
Review comment:
Should `conf._validate()` be called here in case an option is overwritten
that invalidates the config?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (AIRFLOW-3180) Chinese characters all become gibberish when using BashOperator beeline command insert data into hive table
[
https://issues.apache.org/jira/browse/AIRFLOW-3180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liwei li updated AIRFLOW-3180:
--
Description:
with BashOperator ,i use beeline to insert data into hive ,hql with chinese
characters ,after dag run success,hive data contain unreadable code.
python :
{code:java}
# -*- coding: utf-8 -*-
import airflow
from airflow.models import DAG
from airflow.operators.bash_operator import BashOperator
from airflow.operators.python_operator import BranchPythonOperator
from datetime import datetime
import time
from datetime import timedelta
import sys
import pendulum
local_tz = pendulum.timezone("Asia/Shanghai")
reload(sys)
sys.setdefaultencoding('utf-8')
default_args = {
'owner': 'airflow',
'depends_on_past':False,
'start_date':datetime(2018,10,9,19,22,20,tzinfo=local_tz),
'retries':0
}
dag = DAG(
'inserthiveutf8',
default_args=default_args,
description='null',
catchup=False,
schedule_interval=None
)
adf37 = r"""
beeline -u "jdbc:hive2://10.138.***.***:30010/di_zz" -n "*" -p "*" -e
"insert into di_zz.tt_wms_inout_detail_new(fac_id) values ('中')"
"""
abcd8491539084126613 =BashOperator(
task_id='abcd8491539084126613',
bash_command=adf37,
dag=dag){code}
i have tried this:
{code:java}
abcd8491539084126613 =BashOperator( task_id='abcd8491539084126613',
bash_command="sh ~/insert.sh ", dag=dag){code}
this:
{code:java}
export LANG=en_US.UTF-8
beeline -u "jdbc:hive2://10.138.***.***:30010/di_zz" -n "*" -p "*" -e
'insert into di_zz.tt_wms_inout_detail_new(fac_id) values ("中")'{code}
this:
{code:java}
beeline -u "jdbc:hive2://10.138.***.***:30010/di_zz" -n "*" -p "*" -f
~/hql.sql{code}
log:
{code:java}
[2018-10-09 21:00:58,485] {bash_operator.py:110} INFO - INFO : Compiling
command(queryId=hive_2018100921_89390a92-c4de-413f-9958-4d7da1065ef9):
insert into di_zz.tt_wms_inout_detail_new(fac_id) values ("???")
[2018-10-09 21:00:58,485] {bash_operator.py:110} INFO - INFO : Semantic
Analysis Completed
[2018-10-09 21:00:58,486] {bash_operator.py:110} INFO - INFO : Returning Hive
schema: Schema(fieldSchemas:[FieldSchema(name:_col0, type:timestamp,
comment:null), FieldSchema(name:_col1, type:string, comment:null),
FieldSchema(name:_col2, type:void, comment:null), FieldSchema(name:_col3,
type:void, comment:null), FieldSchema(name:_col4, type:void, comment:null),
FieldSchema(name:_col5, type:void, comment:null), FieldSchema(name:_col6,
type:void, comment:null), FieldSchema(name:_col7, type:bigint, comment:null),
FieldSchema(name:_col8, type:bigint, comment:null), FieldSchema(name:_col9,
type:bigint, comment:null), FieldSchema(name:_col10, type:void, comment:null),
FieldSchema(name:_col11, type:timestamp, comment:null)], properties:null)
[2018-10-09 21:00:58,486] {bash_operator.py:110} INFO - INFO : Completed
compiling
command(queryId=hive_2018100921_89390a92-c4de-413f-9958-4d7da1065ef9); Time
taken: 0.291 seconds
[2018-10-09 21:00:58,486] {bash_operator.py:110} INFO - INFO : Executing
command(queryId=hive_2018100921_89390a92-c4de-413f-9958-4d7da1065ef9):
insert into di_zz.tt_wms_inout_detail_new(fac_id) values ("???")
[2018-10-09 21:00:58,486] {bash_operator.py:110} INFO - INFO : Query ID =
hive_2018100921_89390a92-c4de-413f-9958-4d7da1065ef9
{code}
data:
{code:java}
++-+---+--+++---+---+--+---+---+--+--+
| tt_wms_inout_detail_new.stat_date | tt_wms_inout_detail_new.fac_id |
tt_wms_inout_detail_new.fac_name | tt_wms_inout_detail_new.ware_id |
tt_wms_inout_detail_new.ware_name | tt_wms_inout_detail_new.ware_type |
tt_wms_inout_detail_new.product_code | tt_wms_inout_detail_new.ware_cnt |
tt_wms_inout_detail_new.ware_in | tt_wms_inout_detail_new.ware_out |
tt_wms_inout_detail_new.sap_factory_name | tt_wms_inout_detail_new.di_etl_date
|
++-+---+--+++---+---+--+---+---+--+--+
| NULL | ��� | NULL
[jira] [Created] (AIRFLOW-3180) Chinese characters all become gibberish when using BashOperator beeline command insert data into hive table
liwei li created AIRFLOW-3180:
-
Summary: Chinese characters all become gibberish when using
BashOperator beeline command insert data into hive table
Key: AIRFLOW-3180
URL: https://issues.apache.org/jira/browse/AIRFLOW-3180
Project: Apache Airflow
Issue Type: Bug
Affects Versions: 1.10.0
Reporter: liwei li
with BashOperator ,i use beeline to insert data into hive ,hql with chinese
characters ,after dag run success,hive data contain unreadable code.
python :
{code:java}
# -*- coding: utf-8 -*-
import airflow
from airflow.models import DAG
from airflow.operators.bash_operator import BashOperator
from airflow.operators.python_operator import BranchPythonOperator
from datetime import datetime
import time
from datetime import timedelta
import sys
import pendulum
local_tz = pendulum.timezone("Asia/Shanghai")
reload(sys)
sys.setdefaultencoding('utf-8')
default_args = {
'owner': 'airflow',
'depends_on_past':False,
'start_date':datetime(2018,10,9,19,22,20,tzinfo=local_tz),
'retries':0
}
dag = DAG(
'inserthiveutf8',
default_args=default_args,
description='null',
catchup=False,
schedule_interval=None
)
adf37 = r"""
beeline -u "jdbc:hive2://10.138.***.***:30010/di_zz" -n "*" -p "*" -e
"insert into di_zz.tt_wms_inout_detail_new(fac_id) values ('中')"
"""
abcd8491539084126613 =BashOperator(
task_id='abcd8491539084126613',
bash_command=adf37,
dag=dag){code}
i have tried this:
{code:java}
abcd8491539084126613 =BashOperator( task_id='abcd8491539084126613',
bash_command="sh ~/insert.sh ", dag=dag){code}
this:
{code:java}
export LANG=en_US.UTF-8
beeline -u "jdbc:hive2://10.138.***.***:30010/di_zz" -n "*" -p "*" -e
'insert into di_zz.tt_wms_inout_detail_new(fac_id) values ("中")'{code}
this:
{code:java}
beeline -u "jdbc:hive2://10.138.***.***:30010/di_zz" -n "*" -p "*" -f
~/hql.sql{code}
log:
{code:java}
[2018-10-09 21:00:58,485] {bash_operator.py:110} INFO - INFO : Compiling
command(queryId=hive_2018100921_89390a92-c4de-413f-9958-4d7da1065ef9):
insert into di_zz.tt_wms_inout_detail_new(fac_id) values ("???")
[2018-10-09 21:00:58,485] {bash_operator.py:110} INFO - INFO : Semantic
Analysis Completed
[2018-10-09 21:00:58,486] {bash_operator.py:110} INFO - INFO : Returning Hive
schema: Schema(fieldSchemas:[FieldSchema(name:_col0, type:timestamp,
comment:null), FieldSchema(name:_col1, type:string, comment:null),
FieldSchema(name:_col2, type:void, comment:null), FieldSchema(name:_col3,
type:void, comment:null), FieldSchema(name:_col4, type:void, comment:null),
FieldSchema(name:_col5, type:void, comment:null), FieldSchema(name:_col6,
type:void, comment:null), FieldSchema(name:_col7, type:bigint, comment:null),
FieldSchema(name:_col8, type:bigint, comment:null), FieldSchema(name:_col9,
type:bigint, comment:null), FieldSchema(name:_col10, type:void, comment:null),
FieldSchema(name:_col11, type:timestamp, comment:null)], properties:null)
[2018-10-09 21:00:58,486] {bash_operator.py:110} INFO - INFO : Completed
compiling
command(queryId=hive_2018100921_89390a92-c4de-413f-9958-4d7da1065ef9); Time
taken: 0.291 seconds
[2018-10-09 21:00:58,486] {bash_operator.py:110} INFO - INFO : Executing
command(queryId=hive_2018100921_89390a92-c4de-413f-9958-4d7da1065ef9):
insert into di_zz.tt_wms_inout_detail_new(fac_id) values ("???")
[2018-10-09 21:00:58,486] {bash_operator.py:110} INFO - INFO : Query ID =
hive_2018100921_89390a92-c4de-413f-9958-4d7da1065ef9
{code}
data:
{code:java}
++-+---+--+++---+---+--+---+---+--+--+
| tt_wms_inout_detail_new.stat_date | tt_wms_inout_detail_new.fac_id |
tt_wms_inout_detail_new.fac_name | tt_wms_inout_detail_new.ware_id |
tt_wms_inout_detail_new.ware_name | tt_wms_inout_detail_new.ware_type |
tt_wms_inout_detail_new.product_code | tt_wms_inout_detail_new.ware_cnt |
tt_wms_inout_detail_new.ware_in | tt_wms_inout_detail_new.ware_out |
tt_wms_inout_detail_new.sap_factory_name | tt_wms_inout_detail_new.di_etl_date
|
[GitHub] DanSedov edited a comment on issue #4015: [AIRFLOW-2789] Create single node DataProc cluster
DanSedov edited a comment on issue #4015: [AIRFLOW-2789] Create single node DataProc cluster URL: https://github.com/apache/incubator-airflow/pull/4015#issuecomment-428397984 @fenglu-g LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] DanSedov commented on issue #4015: [AIRFLOW-2789] Create single node DataProc cluster
DanSedov commented on issue #4015: [AIRFLOW-2789] Create single node DataProc cluster URL: https://github.com/apache/incubator-airflow/pull/4015#issuecomment-428397984 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] chang commented on issue #4028: [AIRFLOW-3099] Don't ever warn about missing sections of config
chang commented on issue #4028: [AIRFLOW-3099] Don't ever warn about missing sections of config URL: https://github.com/apache/incubator-airflow/pull/4028#issuecomment-428397197 Sure thing! I'll take a look at this later today. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] schnie commented on issue #4027: [AIRFLOW-3177] Change scheduler_heartbeat from gauge to counter
schnie commented on issue #4027: [AIRFLOW-3177] Change scheduler_heartbeat from gauge to counter URL: https://github.com/apache/incubator-airflow/pull/4027#issuecomment-428385234 Absolutely, just updated `UPDATING.md` with the change notice. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] fenglu-g commented on issue #4015: [AIRFLOW-2789] Create single node DataProc cluster
fenglu-g commented on issue #4015: [AIRFLOW-2789] Create single node DataProc cluster URL: https://github.com/apache/incubator-airflow/pull/4015#issuecomment-428381565 @DanSedov mind taking a quick look on this dataproc change? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] fenglu-g commented on issue #4022: [AIRFLOW-3078] Basic operators for Google Compute Engine
fenglu-g commented on issue #4022: [AIRFLOW-3078] Basic operators for Google Compute Engine URL: https://github.com/apache/incubator-airflow/pull/4022#issuecomment-428381362 @kaxil @Fokko Could either of you help to review this PR? Thank you! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #4024: [AIRFLOW-3173] Add _cmd options for password config options
codecov-io commented on issue #4024: [AIRFLOW-3173] Add _cmd options for password config options URL: https://github.com/apache/incubator-airflow/pull/4024#issuecomment-428374426 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4024?src=pr=h1) Report > Merging [#4024](https://codecov.io/gh/apache/incubator-airflow/pull/4024?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/4e94997bc9b09df9ccd6d1a73abe0c5b091f480c?src=pr=desc) will **increase** coverage by `0.49%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4024?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4024 +/- ## == + Coverage 75.77% 76.27% +0.49% == Files 199 199 Lines 1594616462 +516 == + Hits1208312556 +473 - Misses 3863 3906 +43 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4024?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/configuration.py](https://codecov.io/gh/apache/incubator-airflow/pull/4024/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.25% <ø> (ø)` | :arrow_up: | | [airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/4024/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5) | `85.1% <0%> (+2.97%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4024?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4024?src=pr=footer). Last update [4e94997...ae9405d](https://codecov.io/gh/apache/incubator-airflow/pull/4024?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #4028: [AIRFLOW-3099] Don't ever warn about missing sections of config
codecov-io commented on issue #4028: [AIRFLOW-3099] Don't ever warn about missing sections of config URL: https://github.com/apache/incubator-airflow/pull/4028#issuecomment-428371809 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4028?src=pr=h1) Report > Merging [#4028](https://codecov.io/gh/apache/incubator-airflow/pull/4028?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/fbf04e97eb39fb0bce17f8232253811574c63dad?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4028?src=pr=tree) ```diff @@Coverage Diff@@ ## master #4028 +/- ## = + Coverage 75.79% 75.8% +<.01% = Files 199 199 Lines 15946 15936 -10 = - Hits12087 12080 -7 + Misses 38593856 -3 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4028?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/bin/cli.py](https://codecov.io/gh/apache/incubator-airflow/pull/4028/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `65.01% <0%> (+0.53%)` | :arrow_up: | | [airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/4028/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5) | `82.13% <0%> (-0.36%)` | :arrow_down: | | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4028/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `91.67% <0%> (-0.05%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4028?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4028?src=pr=footer). Last update [fbf04e9...502555d](https://codecov.io/gh/apache/incubator-airflow/pull/4028?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #3586: [AIRFLOW-2733] Reconcile psutil and subprocess in webserver cli
codecov-io edited a comment on issue #3586: [AIRFLOW-2733] Reconcile psutil and subprocess in webserver cli URL: https://github.com/apache/incubator-airflow/pull/3586#issuecomment-403631506 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3586?src=pr=h1) Report > Merging [#3586](https://codecov.io/gh/apache/incubator-airflow/pull/3586?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/b8be322d3badfeadfa8f08e0bf92a12a6cd26418?src=pr=desc) will **increase** coverage by `1.76%`. > The diff coverage is `50%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3586?src=pr=tree) ```diff @@Coverage Diff @@ ## master#3586 +/- ## == + Coverage 75.79% 77.56% +1.76% == Files 199 204 +5 Lines 1594615767 -179 == + Hits1208612229 +143 + Misses 3860 3538 -322 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3586?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/bin/cli.py](https://codecov.io/gh/apache/incubator-airflow/pull/3586/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `64.43% <50%> (-0.05%)` | :arrow_down: | | [airflow/sensors/s3\_key\_sensor.py](https://codecov.io/gh/apache/incubator-airflow/pull/3586/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL3MzX2tleV9zZW5zb3IucHk=) | `31.03% <0%> (-68.97%)` | :arrow_down: | | [airflow/sensors/s3\_prefix\_sensor.py](https://codecov.io/gh/apache/incubator-airflow/pull/3586/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL3MzX3ByZWZpeF9zZW5zb3IucHk=) | `41.17% <0%> (-58.83%)` | :arrow_down: | | [airflow/utils/helpers.py](https://codecov.io/gh/apache/incubator-airflow/pull/3586/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9oZWxwZXJzLnB5) | `71.34% <0%> (-13.04%)` | :arrow_down: | | [airflow/hooks/mysql\_hook.py](https://codecov.io/gh/apache/incubator-airflow/pull/3586/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9teXNxbF9ob29rLnB5) | `78% <0%> (-12%)` | :arrow_down: | | [airflow/sensors/sql\_sensor.py](https://codecov.io/gh/apache/incubator-airflow/pull/3586/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL3NxbF9zZW5zb3IucHk=) | `90.47% <0%> (-9.53%)` | :arrow_down: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/incubator-airflow/pull/3586/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `73.91% <0%> (-7.52%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/incubator-airflow/pull/3586/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `83.95% <0%> (-5.31%)` | :arrow_down: | | [airflow/utils/state.py](https://codecov.io/gh/apache/incubator-airflow/pull/3586/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zdGF0ZS5weQ==) | `93.33% <0%> (-3.34%)` | :arrow_down: | | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/3586/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `88.54% <0%> (-3.18%)` | :arrow_down: | | ... and [74 more](https://codecov.io/gh/apache/incubator-airflow/pull/3586/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3586?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3586?src=pr=footer). Last update [b8be322...e7e5a68](https://codecov.io/gh/apache/incubator-airflow/pull/3586?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-3064) Tutorial says to expect output, but no output due to default logging config
[
https://issues.apache.org/jira/browse/AIRFLOW-3064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor resolved AIRFLOW-3064.
Resolution: Duplicate
> Tutorial says to expect output, but no output due to default logging config
> ---
>
> Key: AIRFLOW-3064
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3064
> Project: Apache Airflow
> Issue Type: Bug
> Components: Documentation, logging
>Affects Versions: 1.10.0
> Environment: CentOS release 6.5, Python 2.7.5
>Reporter: Brian King
>Priority: Minor
>
> On [https://airflow.incubator.apache.org/tutorial.html#id1] , it says about
> running the test commands:
> {quote}This should result in displaying a verbose log of events and
> ultimately running your bash command and printing the result.
> Note that the {{airflow test}} command runs task instances locally, outputs
> their log to stdout (on screen), ...
> {quote}
> In fact, there is some logging output, but no output of the tasks:
> {code:java}
> $ airflow test tutorial print_date 2015-06-01
> [2018-09-14 14:43:58,380] {__init__.py:51} INFO - Using executor
> SequentialExecutor
> [2018-09-14 14:43:58,493] {models.py:258} INFO - Filling up the DagBag from
> /vagrant/airflow/dags
> [2018-09-14 14:43:58,571] {example_kubernetes_operator.py:54} WARNING - Could
> not import KubernetesPodOperator: No module named kubernetes
> [2018-09-14 14:43:58,572] {example_kubernetes_operator.py:55} WARNING -
> Install kubernetes dependencies with: pip install
> airflow['kubernetes']{code}
>
> I looked at the logging config, and thought that perhaps the task output
> would be logged to a file (since the default logging config's task handler
> logs to files), but I didn't find anything (relevant) in the log directory.
> To see the task output, I had to use a custom logging config, based on the
> DEFAULT_LOGGING_CONFIG, that used the console handler instead of the task
> handler for the 'airflow.task' logger:
> {code:java}
> 'loggers': {
> 'airflow.task': {
> # 'handlers': ['task'],
> 'handlers': ['console'],
> 'level': 'INFO',
> 'propagate': False,
> },{code}
> This results in the task output showing up:
> {code:java}
> $ airflow test tutorial print_date 2015-06-01
> [2018-09-14 14:49:16,897] {__init__.py:51} INFO - Using executor
> SequentialExecutor
> [2018-09-14 14:49:17,017] {models.py:258} INFO - Filling up the DagBag from
> /vagrant/airflow/dags
> [2018-09-14 14:49:17,093] {example_kubernetes_operator.py:54} WARNING - Could
> not import KubernetesPodOperator: No module named kubernetes
> [2018-09-14 14:49:17,093] {example_kubernetes_operator.py:55} WARNING -
> Install kubernetes dependencies with: pip install airflow['kubernetes']
> [2018-09-14 14:49:17,176] {models.py:1335} INFO - Dependencies all met for
>
> [2018-09-14 14:49:17,179] {models.py:1335} INFO - Dependencies all met for
>
> [2018-09-14 14:49:17,179] {models.py:1547} INFO -
>
> Starting attempt 1 of 2
>
> [2018-09-14 14:49:17,180] {models.py:1569} INFO - Executing
> on 2015-06-01T00:00:00+00:00
> [2018-09-14 14:49:17,236] {bash_operator.py:74} INFO - Tmp dir root location:
> /tmp
> [2018-09-14 14:49:17,237] {bash_operator.py:87} INFO - Temporary script
> location: /tmp/airflowtmp6ieJDv/print_dateZV3cw8
> [2018-09-14 14:49:17,237] {bash_operator.py:97} INFO - Running command: date
> [2018-09-14 14:49:17,241] {bash_operator.py:106} INFO - Output:
> [2018-09-14 14:49:17,250] {bash_operator.py:110} INFO - Fri Sep 14 14:49:17
> UTC 2018
> [2018-09-14 14:49:17,252] {bash_operator.py:114} INFO - Command exited with
> return code 0{code}
>
> That change to the logging config is probably not the appropriate change to
> make for real life usage, but for someone working through the tutorial, it's
> nice to see the output.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (AIRFLOW-3064) Tutorial says to expect output, but no output due to default logging config
[
https://issues.apache.org/jira/browse/AIRFLOW-3064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor reopened AIRFLOW-3064:
Not fixed.
> Tutorial says to expect output, but no output due to default logging config
> ---
>
> Key: AIRFLOW-3064
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3064
> Project: Apache Airflow
> Issue Type: Bug
> Components: Documentation, logging
>Affects Versions: 1.10.0
> Environment: CentOS release 6.5, Python 2.7.5
>Reporter: Brian King
>Priority: Minor
>
> On [https://airflow.incubator.apache.org/tutorial.html#id1] , it says about
> running the test commands:
> {quote}This should result in displaying a verbose log of events and
> ultimately running your bash command and printing the result.
> Note that the {{airflow test}} command runs task instances locally, outputs
> their log to stdout (on screen), ...
> {quote}
> In fact, there is some logging output, but no output of the tasks:
> {code:java}
> $ airflow test tutorial print_date 2015-06-01
> [2018-09-14 14:43:58,380] {__init__.py:51} INFO - Using executor
> SequentialExecutor
> [2018-09-14 14:43:58,493] {models.py:258} INFO - Filling up the DagBag from
> /vagrant/airflow/dags
> [2018-09-14 14:43:58,571] {example_kubernetes_operator.py:54} WARNING - Could
> not import KubernetesPodOperator: No module named kubernetes
> [2018-09-14 14:43:58,572] {example_kubernetes_operator.py:55} WARNING -
> Install kubernetes dependencies with: pip install
> airflow['kubernetes']{code}
>
> I looked at the logging config, and thought that perhaps the task output
> would be logged to a file (since the default logging config's task handler
> logs to files), but I didn't find anything (relevant) in the log directory.
> To see the task output, I had to use a custom logging config, based on the
> DEFAULT_LOGGING_CONFIG, that used the console handler instead of the task
> handler for the 'airflow.task' logger:
> {code:java}
> 'loggers': {
> 'airflow.task': {
> # 'handlers': ['task'],
> 'handlers': ['console'],
> 'level': 'INFO',
> 'propagate': False,
> },{code}
> This results in the task output showing up:
> {code:java}
> $ airflow test tutorial print_date 2015-06-01
> [2018-09-14 14:49:16,897] {__init__.py:51} INFO - Using executor
> SequentialExecutor
> [2018-09-14 14:49:17,017] {models.py:258} INFO - Filling up the DagBag from
> /vagrant/airflow/dags
> [2018-09-14 14:49:17,093] {example_kubernetes_operator.py:54} WARNING - Could
> not import KubernetesPodOperator: No module named kubernetes
> [2018-09-14 14:49:17,093] {example_kubernetes_operator.py:55} WARNING -
> Install kubernetes dependencies with: pip install airflow['kubernetes']
> [2018-09-14 14:49:17,176] {models.py:1335} INFO - Dependencies all met for
>
> [2018-09-14 14:49:17,179] {models.py:1335} INFO - Dependencies all met for
>
> [2018-09-14 14:49:17,179] {models.py:1547} INFO -
>
> Starting attempt 1 of 2
>
> [2018-09-14 14:49:17,180] {models.py:1569} INFO - Executing
> on 2015-06-01T00:00:00+00:00
> [2018-09-14 14:49:17,236] {bash_operator.py:74} INFO - Tmp dir root location:
> /tmp
> [2018-09-14 14:49:17,237] {bash_operator.py:87} INFO - Temporary script
> location: /tmp/airflowtmp6ieJDv/print_dateZV3cw8
> [2018-09-14 14:49:17,237] {bash_operator.py:97} INFO - Running command: date
> [2018-09-14 14:49:17,241] {bash_operator.py:106} INFO - Output:
> [2018-09-14 14:49:17,250] {bash_operator.py:110} INFO - Fri Sep 14 14:49:17
> UTC 2018
> [2018-09-14 14:49:17,252] {bash_operator.py:114} INFO - Command exited with
> return code 0{code}
>
> That change to the logging config is probably not the appropriate change to
> make for real life usage, but for someone working through the tutorial, it's
> nice to see the output.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (AIRFLOW-3064) Tutorial says to expect output, but no output due to default logging config
[
https://issues.apache.org/jira/browse/AIRFLOW-3064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor closed AIRFLOW-3064.
--
Resolution: Fixed
> Tutorial says to expect output, but no output due to default logging config
> ---
>
> Key: AIRFLOW-3064
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3064
> Project: Apache Airflow
> Issue Type: Bug
> Components: Documentation, logging
>Affects Versions: 1.10.0
> Environment: CentOS release 6.5, Python 2.7.5
>Reporter: Brian King
>Priority: Minor
>
> On [https://airflow.incubator.apache.org/tutorial.html#id1] , it says about
> running the test commands:
> {quote}This should result in displaying a verbose log of events and
> ultimately running your bash command and printing the result.
> Note that the {{airflow test}} command runs task instances locally, outputs
> their log to stdout (on screen), ...
> {quote}
> In fact, there is some logging output, but no output of the tasks:
> {code:java}
> $ airflow test tutorial print_date 2015-06-01
> [2018-09-14 14:43:58,380] {__init__.py:51} INFO - Using executor
> SequentialExecutor
> [2018-09-14 14:43:58,493] {models.py:258} INFO - Filling up the DagBag from
> /vagrant/airflow/dags
> [2018-09-14 14:43:58,571] {example_kubernetes_operator.py:54} WARNING - Could
> not import KubernetesPodOperator: No module named kubernetes
> [2018-09-14 14:43:58,572] {example_kubernetes_operator.py:55} WARNING -
> Install kubernetes dependencies with: pip install
> airflow['kubernetes']{code}
>
> I looked at the logging config, and thought that perhaps the task output
> would be logged to a file (since the default logging config's task handler
> logs to files), but I didn't find anything (relevant) in the log directory.
> To see the task output, I had to use a custom logging config, based on the
> DEFAULT_LOGGING_CONFIG, that used the console handler instead of the task
> handler for the 'airflow.task' logger:
> {code:java}
> 'loggers': {
> 'airflow.task': {
> # 'handlers': ['task'],
> 'handlers': ['console'],
> 'level': 'INFO',
> 'propagate': False,
> },{code}
> This results in the task output showing up:
> {code:java}
> $ airflow test tutorial print_date 2015-06-01
> [2018-09-14 14:49:16,897] {__init__.py:51} INFO - Using executor
> SequentialExecutor
> [2018-09-14 14:49:17,017] {models.py:258} INFO - Filling up the DagBag from
> /vagrant/airflow/dags
> [2018-09-14 14:49:17,093] {example_kubernetes_operator.py:54} WARNING - Could
> not import KubernetesPodOperator: No module named kubernetes
> [2018-09-14 14:49:17,093] {example_kubernetes_operator.py:55} WARNING -
> Install kubernetes dependencies with: pip install airflow['kubernetes']
> [2018-09-14 14:49:17,176] {models.py:1335} INFO - Dependencies all met for
>
> [2018-09-14 14:49:17,179] {models.py:1335} INFO - Dependencies all met for
>
> [2018-09-14 14:49:17,179] {models.py:1547} INFO -
>
> Starting attempt 1 of 2
>
> [2018-09-14 14:49:17,180] {models.py:1569} INFO - Executing
> on 2015-06-01T00:00:00+00:00
> [2018-09-14 14:49:17,236] {bash_operator.py:74} INFO - Tmp dir root location:
> /tmp
> [2018-09-14 14:49:17,237] {bash_operator.py:87} INFO - Temporary script
> location: /tmp/airflowtmp6ieJDv/print_dateZV3cw8
> [2018-09-14 14:49:17,237] {bash_operator.py:97} INFO - Running command: date
> [2018-09-14 14:49:17,241] {bash_operator.py:106} INFO - Output:
> [2018-09-14 14:49:17,250] {bash_operator.py:110} INFO - Fri Sep 14 14:49:17
> UTC 2018
> [2018-09-14 14:49:17,252] {bash_operator.py:114} INFO - Command exited with
> return code 0{code}
>
> That change to the logging config is probably not the appropriate change to
> make for real life usage, but for someone working through the tutorial, it's
> nice to see the output.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2869) Remove smart quote from default config
[ https://issues.apache.org/jira/browse/AIRFLOW-2869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor updated AIRFLOW-2869: --- Fix Version/s: (was: 1.10.1) (was: 2.0.0) 1.10.0 > Remove smart quote from default config > -- > > Key: AIRFLOW-2869 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2869 > Project: Apache Airflow > Issue Type: Bug >Reporter: Siddharth Anand >Assignee: Siddharth Anand >Priority: Trivial > Fix For: 1.10.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ashb commented on issue #4028: [AIRFLOW-3099] Don't ever warn about missing sections of config
ashb commented on issue #4028: [AIRFLOW-3099] Don't ever warn about missing sections of config URL: https://github.com/apache/incubator-airflow/pull/4028#issuecomment-428365750 @chang you made a similar PR (#4002) - mind taking a look at this one too? It takes a completely different approach, but should mean there's no list to udpate. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3178) `airflow run` config doens't cope with % in config values.
[
https://issues.apache.org/jira/browse/AIRFLOW-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644148#comment-16644148
]
ASF GitHub Bot commented on AIRFLOW-3178:
-
ashb opened a new pull request #4029: [AIRFLOW-3178] Handle percents signs in
configs for airflow run
URL: https://github.com/apache/incubator-airflow/pull/4029
Make sure you have checked _all_ steps below.
### Jira
- [x] https://issues.apache.org/jira/browse/AIRFLOW-3178
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
To have a `%` appear in the config file, it needs to appear as `%%`. However
the way we were generating the temporary config for `airflow run` it would
loose that escaping and cause a config error.
This fixes the way we generate that temp config by using methods built-in to
ConfigParser to get the raw values (rather than grubbing around with
`deepcopy()`). I have left this PR as three individual commits as I think it
will be easier to review in it's current form.
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> `airflow run` config doens't cope with % in config values.
> --
>
> Key: AIRFLOW-3178
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3178
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Ash Berlin-Taylor
>Priority: Major
> Fix For: 1.10.1
>
>
> ConfigParser uses %-signs for interpolation, so if you need a literal % in
> airflow.cfg it needs to be escaped in by specifying them as {{%%}}), but by
> the time they are written out to the temporary config that is passed to
> `airflow run` they escaping is lost.
> This is also true for configuration via environment variables, where {{%}}
> doesn't need to be escaped at all, but is included (sometimes needlessly) in
> the config un-escaped.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] ashb opened a new pull request #4029: [AIRFLOW-3178] Handle percents signs in configs for airflow run
ashb opened a new pull request #4029: [AIRFLOW-3178] Handle percents signs in
configs for airflow run
URL: https://github.com/apache/incubator-airflow/pull/4029
Make sure you have checked _all_ steps below.
### Jira
- [x] https://issues.apache.org/jira/browse/AIRFLOW-3178
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
To have a `%` appear in the config file, it needs to appear as `%%`. However
the way we were generating the temporary config for `airflow run` it would
loose that escaping and cause a config error.
This fixes the way we generate that temp config by using methods built-in to
ConfigParser to get the raw values (rather than grubbing around with
`deepcopy()`). I have left this PR as three individual commits as I think it
will be easier to review in it's current form.
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-3099) Errors raised when some blocs are missing in airflow.cfg
[
https://issues.apache.org/jira/browse/AIRFLOW-3099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644119#comment-16644119
]
ASF GitHub Bot commented on AIRFLOW-3099:
-
ashb opened a new pull request #4028: [AIRFLOW-3099] Don't ever warn about
missing sections of config
URL: https://github.com/apache/incubator-airflow/pull/4028
Make sure you have checked _all_ steps below.
### Jira
- [X] https://issues.apache.org/jira/browse/AIRFLOW-3099
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
Rather than looping through and setting each config variable
individually, and having to know which sections are optional and which
aren't, instead we can just call a single function on ConfigParser and
it will read the config from the dict, and more importantly here, never
error about missing sections - it will just create them as needed.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason: already covered by existing tests
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Errors raised when some blocs are missing in airflow.cfg
>
>
> Key: AIRFLOW-3099
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3099
> Project: Apache Airflow
> Issue Type: Bug
> Components: configuration
>Affects Versions: 1.10.0
>Reporter: Christophe
>Assignee: Kaxil Naik
>Priority: Minor
> Fix For: 1.10.1
>
>
> When we upgrade from a version of airflow to another one and new config bloc
> are available or if we delete useless blocs, lots of errors are raised if we
> don't have some blocs in airflow.cfg file.
> We need to avoid these errors for non-required blocs.
>
> For the record logs (not exhaustive):
>
> {noformat}
> [2018-09-21 10:49:37,727] {cli.py:464} ERROR - Section atlas Option
> sasl_enabled does not exist in the config!
> [2018-09-21 10:49:37,727] {cli.py:464} ERROR - Section atlas Option host does
> not exist in the config!
> [2018-09-21 10:49:37,727] {cli.py:464} ERROR - Section atlas Option port does
> not exist in the config!
> [2018-09-21 10:49:37,727] {cli.py:464} ERROR - Section atlas Option username
> does not exist in the config!
> [2018-09-21 10:49:37,727] {cli.py:464} ERROR - Section atlas Option password
> does not exist in the config!
> [2018-09-21 10:49:37,728] {cli.py:464} ERROR - Section hive Option
> default_hive_mapred_queue does not exist in the config!
> [2018-09-21 10:49:37,728] {cli.py:464} ERROR - Section mesos Option master
> does not exist in the config!
> [2018-09-21 10:49:37,728] {cli.py:464} ERROR - Section mesos Option
> framework_name does not exist in the config!
> [2018-09-21 10:49:37,728] {cli.py:464} ERROR - Section mesos Option task_cpu
> does not exist in the config!
> [2018-09-21 10:49:37,728] {cli.py:464} ERROR - Section mesos Option
> task_memory does not exist in the config!
> [2018-09-21 10:49:37,728] {cli.py:464} ERROR - Section mesos Option
> checkpoint does not exist in the config!
> [2018-09-21 10:49:37,728] {cli.py:464} ERROR - Section mesos Option
> authenticate does not exist in the config!
> [2018-09-21 10:49:37,729] {cli.py:464} ERROR - Section elasticsearch Option
> elasticsearch_host does not exist in the config!
> [2018-09-21 10:49:37,729] {cli.py:464} ERROR - Section elasticsearch Option
> elasticsearch_log_id_template does not exist in the config!
> [2018-09-21 10:49:37,729] {cli.py:464} ERROR - Section elasticsearch Option
>
[GitHub] ashb opened a new pull request #4028: [AIRFLOW-3099] Don't ever warn about missing sections of config
ashb opened a new pull request #4028: [AIRFLOW-3099] Don't ever warn about
missing sections of config
URL: https://github.com/apache/incubator-airflow/pull/4028
Make sure you have checked _all_ steps below.
### Jira
- [X] https://issues.apache.org/jira/browse/AIRFLOW-3099
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
Rather than looping through and setting each config variable
individually, and having to know which sections are optional and which
aren't, instead we can just call a single function on ConfigParser and
it will read the config from the dict, and more importantly here, never
error about missing sections - it will just create them as needed.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason: already covered by existing tests
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-3179) Remove airflow_travis.cfg
Ash Berlin-Taylor created AIRFLOW-3179: -- Summary: Remove airflow_travis.cfg Key: AIRFLOW-3179 URL: https://issues.apache.org/jira/browse/AIRFLOW-3179 Project: Apache Airflow Issue Type: Task Components: configuration Reporter: Ash Berlin-Taylor We appear to have a different config for running tests on Travis to running tests locally/in docker. This means that for me locally the tests/configuration.py tests fail. We should remove this extra config file *or* remove the default_test.cfg file and only make use of this one config file. Either way having two is bad. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3178) `airflow run` config doens't cope with % in config values.
[
https://issues.apache.org/jira/browse/AIRFLOW-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor updated AIRFLOW-3178:
---
Description:
ConfigParser uses %-signs for interpolation, so if you need a literal % in
airflow.cfg it needs to be escaped in by specifying them as {{%%}}), but by
the time they are written out to the temporary config that is passed to
`airflow run` they escaping is lost.
This is also true for configuration via environment variables, where {{%}}
doesn't need to be escaped at all, but is included (sometimes needlessly) in
the config un-escaped.
was:
Even if percent sings are correctly escaped in the airflow.cfg (by specifying
them as {{%%}}) by the time they are written out to the temporary config that
is passed to `airflow run` they escaping is lost.
This is also true for configuration via environment variables, where {{%}}
doesn't need to be escaped at all, but is included (sometimes needlessly) in
the config un-escaped.
> `airflow run` config doens't cope with % in config values.
> --
>
> Key: AIRFLOW-3178
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3178
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Ash Berlin-Taylor
>Priority: Major
> Fix For: 1.10.1
>
>
> ConfigParser uses %-signs for interpolation, so if you need a literal % in
> airflow.cfg it needs to be escaped in by specifying them as {{%%}}), but by
> the time they are written out to the temporary config that is passed to
> `airflow run` they escaping is lost.
> This is also true for configuration via environment variables, where {{%}}
> doesn't need to be escaped at all, but is included (sometimes needlessly) in
> the config un-escaped.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-3178) `airflow run` config doens't cope with % in config values.
Ash Berlin-Taylor created AIRFLOW-3178:
--
Summary: `airflow run` config doens't cope with % in config values.
Key: AIRFLOW-3178
URL: https://issues.apache.org/jira/browse/AIRFLOW-3178
Project: Apache Airflow
Issue Type: Bug
Reporter: Ash Berlin-Taylor
Fix For: 1.10.1
Even if percent sings are correctly escaped in the airflow.cfg (by specifying
them as {{%%}}) by the time they are written out to the temporary config that
is passed to `airflow run` they escaping is lost.
This is also true for configuration via environment variables, where {{%}}
doesn't need to be escaped at all, but is included (sometimes needlessly) in
the config un-escaped.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-479) Config: Add LDAP Bind Password captured by stdout
[ https://issues.apache.org/jira/browse/AIRFLOW-479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor resolved AIRFLOW-479. --- Resolution: Fixed > Config: Add LDAP Bind Password captured by stdout > - > > Key: AIRFLOW-479 > URL: https://issues.apache.org/jira/browse/AIRFLOW-479 > Project: Apache Airflow > Issue Type: Improvement > Components: security >Affects Versions: 1.7.1 >Reporter: Jason DeCorte >Assignee: Yongjun Park >Priority: Minor > > Currently Airflow configuration only allows certain properties to be assigned > by the standard output of a command shell script. For security reasons, I > would like the bind_password property for ldap configuration to be added to > the list. This will enhance security by allowing the encryption of the > password while stored and the decryption of it at run time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2869) Remove smart quote from default config
[ https://issues.apache.org/jira/browse/AIRFLOW-2869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor updated AIRFLOW-2869: --- Fix Version/s: 1.10.1 Issue Type: Bug (was: Improvement) > Remove smart quote from default config > -- > > Key: AIRFLOW-2869 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2869 > Project: Apache Airflow > Issue Type: Bug >Reporter: Siddharth Anand >Assignee: Siddharth Anand >Priority: Trivial > Fix For: 2.0.0, 1.10.1 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3173) Add _cmd options for password config options
[
https://issues.apache.org/jira/browse/AIRFLOW-3173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor updated AIRFLOW-3173:
---
Issue Type: Improvement (was: New Feature)
> Add _cmd options for password config options
>
>
> Key: AIRFLOW-3173
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3173
> Project: Apache Airflow
> Issue Type: Improvement
>Reporter: Ash Berlin-Taylor
>Priority: Major
> Fix For: 1.10.1
>
>
> Right now we support specifying a command to get a small number of config
> options ({{sql_alchemy_conn_cmd}} for instance) but there are some new config
> options that contain a password that don't have a {{_cmd }} option.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] ashb commented on issue #4027: [AIRFLOW-3177] Change scheduler_heartbeat from gauge to counter
ashb commented on issue #4027: [AIRFLOW-3177] Change scheduler_heartbeat from gauge to counter URL: https://github.com/apache/incubator-airflow/pull/4027#issuecomment-428332416 The logic here sounds sensible to me. I think @feng-tao makes heavy use of the metrics from Airflow for monitoring. At the very least we will need a note in UPDATING.md about this change. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3177) Change scheduler_heartbeat metric from gauge to counter
[
https://issues.apache.org/jira/browse/AIRFLOW-3177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644015#comment-16644015
]
ASF GitHub Bot commented on AIRFLOW-3177:
-
schnie opened a new pull request #4027: [AIRFLOW-3177] Change
scheduler_heartbeat from gauge to counter
URL: https://github.com/apache/incubator-airflow/pull/4027
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW-3177) issues and references
them in the PR title.
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
Currently, the `scheduler_heartbeat` metric exposed with the statsd
integration is a gauge. I'm proposing to change the gauge to a counter for a
better integration with Prometheus via the
[statsd_exporter](https://github.com/prometheus/statsd_exporter.)
Rather than pointing Airflow at an actual statsd server, you can point it at
this exporter, which will accumulate the metrics and expose them to be scraped
by Prometheus at /metrics. The problem is that once this value is set when the
scheduler runs its first loop, it will always be exposed to Prometheus as 1.
The scheduler can crash, or be turned off and the statsd exporter will report a
1 until it is restarted and rebuilds its internal state.
By turning this metric into a counter, we can detect an issue with the
scheduler by graphing and alerting using a rate. If the rate of change of the
counter drops below what it should be at (determined by the
scheduler_heartbeat_secs setting), we can fire an alert.
This should be helpful for adoption in Kubernetes environments where
Prometheus is pretty much the standard.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason: No existing tests around statsd
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Change scheduler_heartbeat metric from gauge to counter
> ---
>
> Key: AIRFLOW-3177
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3177
> Project: Apache Airflow
> Issue Type: Improvement
> Components: scheduler
>Reporter: Greg Neiheisel
>Assignee: Greg Neiheisel
>Priority: Minor
>
> Currently, the scheduler_heartbeat metric exposed with the statsd integration
> is a gauge. I'm proposing to change the gauge to a counter for a better
> integration with Prometheus via the
> [statsd_exporter|[https://github.com/prometheus/statsd_exporter].]
> Rather than pointing Airflow at an actual statsd server, you can point it at
> this exporter, which will accumulate the metrics and expose them to be
> scraped by Prometheus at /metrics. The problem is that once this value is set
> when the scheduler runs its first loop, it will always be exposed to
> Prometheus as 1. The scheduler can crash, or be turned off and the statsd
> exporter will report a 1 until it is restarted and rebuilds its internal
> state.
> By turning this metric into a counter, we can detect an issue with the
> scheduler by graphing and alerting using a rate. If the rate of change of the
> counter drops below what it should be at (determined by the
> scheduler_heartbeat_secs setting), we can fire an alert.
> This should be helpful for adoption in Kubernetes environments where
> Prometheus is pretty much the standard.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] schnie opened a new pull request #4027: [AIRFLOW-3177] Change scheduler_heartbeat from gauge to counter
schnie opened a new pull request #4027: [AIRFLOW-3177] Change
scheduler_heartbeat from gauge to counter
URL: https://github.com/apache/incubator-airflow/pull/4027
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW-3177) issues and references
them in the PR title.
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
Currently, the `scheduler_heartbeat` metric exposed with the statsd
integration is a gauge. I'm proposing to change the gauge to a counter for a
better integration with Prometheus via the
[statsd_exporter](https://github.com/prometheus/statsd_exporter.)
Rather than pointing Airflow at an actual statsd server, you can point it at
this exporter, which will accumulate the metrics and expose them to be scraped
by Prometheus at /metrics. The problem is that once this value is set when the
scheduler runs its first loop, it will always be exposed to Prometheus as 1.
The scheduler can crash, or be turned off and the statsd exporter will report a
1 until it is restarted and rebuilds its internal state.
By turning this metric into a counter, we can detect an issue with the
scheduler by graphing and alerting using a rate. If the rate of change of the
counter drops below what it should be at (determined by the
scheduler_heartbeat_secs setting), we can fire an alert.
This should be helpful for adoption in Kubernetes environments where
Prometheus is pretty much the standard.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason: No existing tests around statsd
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] r39132 closed pull request #4026: Include Pagar.me in list of users of Airflow
r39132 closed pull request #4026: Include Pagar.me in list of users of Airflow URL: https://github.com/apache/incubator-airflow/pull/4026 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/README.md b/README.md index d41ba25698..64740ca27a 100644 --- a/README.md +++ b/README.md @@ -229,6 +229,7 @@ Currently **officially** using Airflow: 1. [OneFineStay](https://www.onefinestay.com) [[@slangwald](https://github.com/slangwald)] 1. [Open Knowledge International](https://okfn.org) [@vitorbaptista](https://github.com/vitorbaptista) 1. [Overstock](https://www.github.com/overstock) [[@mhousley](https://github.com/mhousley) & [@mct0006](https://github.com/mct0006)] +1. [Pagar.me](https://pagar.me/) [[@pagarme](https://github.com/pagarme) 1. [Pandora Media](https://www.pandora.com/) [[@Acehaidrey](https://github.com/Acehaidrey) & [@wolfier](https://github.com/wolfier)] 1. [PAYMILL](https://www.paymill.com/) [[@paymill](https://github.com/paymill) & [@matthiashuschle](https://github.com/matthiashuschle)] 1. [PayPal](https://www.paypal.com/) [[@r39132](https://github.com/r39132) & [@jhsenjaliya](https://github.com/jhsenjaliya)] This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] r39132 commented on issue #4026: Include Pagar.me in list of users of Airflow
r39132 commented on issue #4026: Include Pagar.me in list of users of Airflow URL: https://github.com/apache/incubator-airflow/pull/4026#issuecomment-428323823 @thalesmello Welcome aboard! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-3177) Change scheduler_heartbeat metric from gauge to counter
Greg Neiheisel created AIRFLOW-3177: --- Summary: Change scheduler_heartbeat metric from gauge to counter Key: AIRFLOW-3177 URL: https://issues.apache.org/jira/browse/AIRFLOW-3177 Project: Apache Airflow Issue Type: Improvement Components: scheduler Reporter: Greg Neiheisel Assignee: Greg Neiheisel Currently, the scheduler_heartbeat metric exposed with the statsd integration is a gauge. I'm proposing to change the gauge to a counter for a better integration with Prometheus via the [statsd_exporter|[https://github.com/prometheus/statsd_exporter].] Rather than pointing Airflow at an actual statsd server, you can point it at this exporter, which will accumulate the metrics and expose them to be scraped by Prometheus at /metrics. The problem is that once this value is set when the scheduler runs its first loop, it will always be exposed to Prometheus as 1. The scheduler can crash, or be turned off and the statsd exporter will report a 1 until it is restarted and rebuilds its internal state. By turning this metric into a counter, we can detect an issue with the scheduler by graphing and alerting using a rate. If the rate of change of the counter drops below what it should be at (determined by the scheduler_heartbeat_secs setting), we can fire an alert. This should be helpful for adoption in Kubernetes environments where Prometheus is pretty much the standard. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] michaelDovgal commented on issue #3356: [AIRFLOW-2460] Users can acces PersistentVolumeClaims and Configmaps using KubernetesPodOperator
michaelDovgal commented on issue #3356: [AIRFLOW-2460] Users can acces PersistentVolumeClaims and Configmaps using KubernetesPodOperator URL: https://github.com/apache/incubator-airflow/pull/3356#issuecomment-428269287 @dimberman maybe I misunderstand something, but where is in this change something related to confimaps usage? is it going to be added? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] thalesmello opened a new pull request #4026: Include Pagar.me in list of users of Airflow
thalesmello opened a new pull request #4026: Include Pagar.me in list of users
of Airflow
URL: https://github.com/apache/incubator-airflow/pull/4026
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-3176) Duration tooltip on Tree View of Tasks
Nicolás Kittsteiner created AIRFLOW-3176: Summary: Duration tooltip on Tree View of Tasks Key: AIRFLOW-3176 URL: https://issues.apache.org/jira/browse/AIRFLOW-3176 Project: Apache Airflow Issue Type: Wish Components: ui Affects Versions: 1.10.0 Reporter: Nicolás Kittsteiner Attachments: Screen Shot 2018-10-09 at 13.27.42.png On the Tree View of the UI over de squares of tasks, are a tooltip that show details of tasks. The field duration could be in a more friendly format like HH:MM:SS. Thanks :) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] rcorre commented on issue #3546: AIRFLOW-2664: Support filtering dag runs by id prefix in API.
rcorre commented on issue #3546: AIRFLOW-2664: Support filtering dag runs by id
prefix in API.
URL:
https://github.com/apache/incubator-airflow/pull/3546#issuecomment-428252232
Anyone able to help me figure out why the test I added is failing?
On my initial commit, I was getting `No row was found for one()` on the
second `trigger_dag` call in my test.
After [trying to fix
that](https://github.com/apache/incubator-airflow/pull/3546/commits/0b4ce4da97e879ab3159450d0ff723d1e8ddc07f)
I'm getting a variety of errors such as:
```
ERROR - Add Permission to Role Error: Can't flush None value found in
collection Role.permissions
```
and
```
OperationalError: (sqlite3.OperationalError) no such table: task_instance
[SQL: u'DELETE FROM task_instance WHERE task_instance.dag_id = ?'] [parameters:
('unit_tests',)]
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] codecov-io edited a comment on issue #3992: [AIRFLOW-620] Feature to tail custom number of logs instead of rendering whole log
codecov-io edited a comment on issue #3992: [AIRFLOW-620] Feature to tail custom number of logs instead of rendering whole log URL: https://github.com/apache/incubator-airflow/pull/3992#issuecomment-426519197 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3992?src=pr=h1) Report > Merging [#3992](https://codecov.io/gh/apache/incubator-airflow/pull/3992?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/b8be322d3badfeadfa8f08e0bf92a12a6cd26418?src=pr=desc) will **decrease** coverage by `0.15%`. > The diff coverage is `52.74%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3992?src=pr=tree) ```diff @@Coverage Diff @@ ## master#3992 +/- ## == - Coverage 75.79% 75.63% -0.16% == Files 199 199 Lines 1594616024 +78 == + Hits1208612120 +34 - Misses 3860 3904 +44 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3992?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/log/file\_task\_handler.py](https://codecov.io/gh/apache/incubator-airflow/pull/3992/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZmlsZV90YXNrX2hhbmRsZXIucHk=) | `76.41% <32.14%> (-13%)` | :arrow_down: | | [airflow/bin/cli.py](https://codecov.io/gh/apache/incubator-airflow/pull/3992/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `63.05% <4.76%> (-1.43%)` | :arrow_down: | | [airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3992/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `69.03% <90.47%> (+0.17%)` | :arrow_up: | | [airflow/www\_rbac/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3992/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==) | `72.21% <90.47%> (+0.16%)` | :arrow_up: | | [airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/3992/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5) | `82.13% <0%> (-0.27%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3992?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3992?src=pr=footer). Last update [b8be322...85360ab](https://codecov.io/gh/apache/incubator-airflow/pull/3992?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #3992: [AIRFLOW-620] Feature to tail custom number of logs instead of rendering whole log
codecov-io edited a comment on issue #3992: [AIRFLOW-620] Feature to tail custom number of logs instead of rendering whole log URL: https://github.com/apache/incubator-airflow/pull/3992#issuecomment-426519197 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3992?src=pr=h1) Report > Merging [#3992](https://codecov.io/gh/apache/incubator-airflow/pull/3992?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/b8be322d3badfeadfa8f08e0bf92a12a6cd26418?src=pr=desc) will **decrease** coverage by `0.15%`. > The diff coverage is `52.74%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3992?src=pr=tree) ```diff @@Coverage Diff @@ ## master#3992 +/- ## == - Coverage 75.79% 75.63% -0.16% == Files 199 199 Lines 1594616024 +78 == + Hits1208612120 +34 - Misses 3860 3904 +44 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3992?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/log/file\_task\_handler.py](https://codecov.io/gh/apache/incubator-airflow/pull/3992/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZmlsZV90YXNrX2hhbmRsZXIucHk=) | `76.41% <32.14%> (-13%)` | :arrow_down: | | [airflow/bin/cli.py](https://codecov.io/gh/apache/incubator-airflow/pull/3992/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `63.05% <4.76%> (-1.43%)` | :arrow_down: | | [airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3992/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `69.03% <90.47%> (+0.17%)` | :arrow_up: | | [airflow/www\_rbac/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3992/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==) | `72.21% <90.47%> (+0.16%)` | :arrow_up: | | [airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/3992/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5) | `82.13% <0%> (-0.27%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3992?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3992?src=pr=footer). Last update [b8be322...85360ab](https://codecov.io/gh/apache/incubator-airflow/pull/3992?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] phani8996 commented on a change in pull request #3989: [AIRFLOW-1945] Autoscale celery workers for airflow added
phani8996 commented on a change in pull request #3989: [AIRFLOW-1945] Autoscale celery workers for airflow added URL: https://github.com/apache/incubator-airflow/pull/3989#discussion_r223754538 ## File path: airflow/config_templates/default_airflow.cfg ## @@ -349,6 +349,14 @@ celery_app_name = airflow.executors.celery_executor # your worker box and the nature of your tasks worker_concurrency = 16 +# The minimum and maximum concurrency that will be used when starting workers with the +# "airflow worker" command. This defines the number of task instances that +# a worker will take, minimum numbers are created in beginning and scaled up and down +# based on number of queued tasks. pick these numbers based on resources on +# worker box and the nature of the task. If autoscale option is available worker_concurrency +# will be ignored +worker_autoscale = 12,16 Review comment: @ashb this value has been commented out. Please review This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] phani8996 commented on a change in pull request #3992: [AIRFLOW-620] Feature to tail custom number of logs instead of rendering whole log
phani8996 commented on a change in pull request #3992: [AIRFLOW-620] Feature to
tail custom number of logs instead of rendering whole log
URL: https://github.com/apache/incubator-airflow/pull/3992#discussion_r223753602
##
File path: airflow/www/views.py
##
@@ -836,6 +839,21 @@ def log(self, session=None):
dttm = pendulum.parse(execution_date)
form = DateTimeForm(data={'execution_date': dttm})
dag = dagbag.get_dag(dag_id)
+metadata = {}
+tailing_required = False
+tail_lines_list = [100, 200, 500, 1000]
+num_lines = 100
+if conf.has_option('webserver', 'tail_lines_list'):
+tail_lines_list = conf.get('webserver', 'tail_lines_list')
+tail_lines_list = list(map(int, tail_lines_list.split(',')))
+if conf.has_option('webserver', 'tail_logs'):
+tailing_required = conf.getboolean('webserver', 'tail_logs')
+if tailing_required and conf.has_option('webserver', 'num_lines'):
+num_lines = conf.getint('webserver', 'num_lines')
+metadata["num_lines"] = num_lines
Review comment:
Relevant comments have been added. @ashb please review.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] phani8996 commented on a change in pull request #3992: [AIRFLOW-620] Feature to tail custom number of logs instead of rendering whole log
phani8996 commented on a change in pull request #3992: [AIRFLOW-620] Feature to
tail custom number of logs instead of rendering whole log
URL: https://github.com/apache/incubator-airflow/pull/3992#discussion_r223753060
##
File path: airflow/www/templates/airflow/ti_log.html
##
@@ -29,6 +29,28 @@ {{ title }}
{% endfor %}
+ {% if tailing_required %}
+
+
+
+
+Tail Logs
+
+
+{% for tail_line in tail_lines_list %}
Review comment:
Changes have been made to stop showing dropdown for unsupported platforms.
Please review.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] XD-DENG commented on issue #4025: [AIRFLOW-3175] Fix docstring format in airflow/jobs.py
XD-DENG commented on issue #4025: [AIRFLOW-3175] Fix docstring format in airflow/jobs.py URL: https://github.com/apache/incubator-airflow/pull/4025#issuecomment-428239110 @ashb sure, understand. BTW, forgot to apply the trick `[ci skip]` you shared earlier. Costed ASF some money for this ;-) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb commented on issue #4025: [AIRFLOW-3175] Fix docstring format in airflow/jobs.py
ashb commented on issue #4025: [AIRFLOW-3175] Fix docstring format in airflow/jobs.py URL: https://github.com/apache/incubator-airflow/pull/4025#issuecomment-428237482 That sounds like a good idea. There's no hard-or-fast rule, and each person makes their own judgment, but for typo only or formatting only fixes (such as the new lines) it's okay by my book. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-3086) Add extras group in setup.py for google oauth
[ https://issues.apache.org/jira/browse/AIRFLOW-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor resolved AIRFLOW-3086. Resolution: Fixed Fix Version/s: 2.0.0 > Add extras group in setup.py for google oauth > - > > Key: AIRFLOW-3086 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3086 > Project: Apache Airflow > Issue Type: Bug >Reporter: Josh Carp >Priority: Major > Fix For: 2.0.0 > > > Since the google auth backend requires Flask-OAuthlib, it would be helpful to > add an extras group to setup.py for google auth that installs this dependency. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3086) Add extras group in setup.py for google oauth
[ https://issues.apache.org/jira/browse/AIRFLOW-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16643603#comment-16643603 ] ASF GitHub Bot commented on AIRFLOW-3086: - ashb closed pull request #3917: [AIRFLOW-3086] Add extras group for google auth to setup.py. URL: https://github.com/apache/incubator-airflow/pull/3917 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/docs/installation.rst b/docs/installation.rst index 3db48e45dd..bb1365ff00 100644 --- a/docs/installation.rst +++ b/docs/installation.rst @@ -41,71 +41,75 @@ these extra dependencies. Here's the list of the subpackages and what they enable: -+---+--+-+ -| subpackage| install command | enables | -+===+==+=+ -| all | ``pip install apache-airflow[all]`` | All Airflow features known to man | -+---+--+-+ -| all_dbs | ``pip install apache-airflow[all_dbs]`` | All databases integrations | -+---+--+-+ -| async | ``pip install apache-airflow[async]``| Async worker classes for Gunicorn | -+---+--+-+ -| celery| ``pip install apache-airflow[celery]`` | CeleryExecutor | -+---+--+-+ -| cloudant | ``pip install apache-airflow[cloudant]`` | Cloudant hook | -+---+--+-+ -| crypto| ``pip install apache-airflow[crypto]`` | Encrypt connection passwords in metadata db | -+---+--+-+ -| devel | ``pip install apache-airflow[devel]``| Minimum dev tools requirements | -+---+--+-+ -| devel_hadoop | ``pip install apache-airflow[devel_hadoop]`` | Airflow + dependencies on the Hadoop stack | -+---+--+-+ -| druid | ``pip install apache-airflow[druid]``| Druid related operators & hooks | -+---+--+-+ -| gcp_api | ``pip install apache-airflow[gcp_api]`` | Google Cloud Platform hooks and operators | -| | | (using ``google-api-python-client``)| -+---+--+-+ -| hdfs | ``pip install apache-airflow[hdfs]`` | HDFS hooks and operators| -+---+--+-+ -| hive | ``pip install apache-airflow[hive]`` | All Hive related operators | -+---+--+-+ -| jdbc | ``pip install apache-airflow[jdbc]`` | JDBC hooks and operators| -+---+--+-+ -| kerberos | ``pip install apache-airflow[kerberos]`` | Kerberos integration for Kerberized Hadoop | -+---+--+-+ -| ldap | ``pip install apache-airflow[ldap]`` | LDAP authentication for users | -+---+--+-+ -| mssql | ``pip install apache-airflow[mssql]``| Microsoft SQL Server operators and
[GitHub] ashb closed pull request #3917: [AIRFLOW-3086] Add extras group for google auth to setup.py.
ashb closed pull request #3917: [AIRFLOW-3086] Add extras group for google auth to setup.py. URL: https://github.com/apache/incubator-airflow/pull/3917 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/docs/installation.rst b/docs/installation.rst index 3db48e45dd..bb1365ff00 100644 --- a/docs/installation.rst +++ b/docs/installation.rst @@ -41,71 +41,75 @@ these extra dependencies. Here's the list of the subpackages and what they enable: -+---+--+-+ -| subpackage| install command | enables | -+===+==+=+ -| all | ``pip install apache-airflow[all]`` | All Airflow features known to man | -+---+--+-+ -| all_dbs | ``pip install apache-airflow[all_dbs]`` | All databases integrations | -+---+--+-+ -| async | ``pip install apache-airflow[async]``| Async worker classes for Gunicorn | -+---+--+-+ -| celery| ``pip install apache-airflow[celery]`` | CeleryExecutor | -+---+--+-+ -| cloudant | ``pip install apache-airflow[cloudant]`` | Cloudant hook | -+---+--+-+ -| crypto| ``pip install apache-airflow[crypto]`` | Encrypt connection passwords in metadata db | -+---+--+-+ -| devel | ``pip install apache-airflow[devel]``| Minimum dev tools requirements | -+---+--+-+ -| devel_hadoop | ``pip install apache-airflow[devel_hadoop]`` | Airflow + dependencies on the Hadoop stack | -+---+--+-+ -| druid | ``pip install apache-airflow[druid]``| Druid related operators & hooks | -+---+--+-+ -| gcp_api | ``pip install apache-airflow[gcp_api]`` | Google Cloud Platform hooks and operators | -| | | (using ``google-api-python-client``)| -+---+--+-+ -| hdfs | ``pip install apache-airflow[hdfs]`` | HDFS hooks and operators| -+---+--+-+ -| hive | ``pip install apache-airflow[hive]`` | All Hive related operators | -+---+--+-+ -| jdbc | ``pip install apache-airflow[jdbc]`` | JDBC hooks and operators| -+---+--+-+ -| kerberos | ``pip install apache-airflow[kerberos]`` | Kerberos integration for Kerberized Hadoop | -+---+--+-+ -| ldap | ``pip install apache-airflow[ldap]`` | LDAP authentication for users | -+---+--+-+ -| mssql | ``pip install apache-airflow[mssql]``| Microsoft SQL Server operators and hook,| -| | | support as an Airflow backend |
[jira] [Closed] (AIRFLOW-2001) Make sensors relinquish their execution slots
[ https://issues.apache.org/jira/browse/AIRFLOW-2001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor closed AIRFLOW-2001. -- Resolution: Fixed > Make sensors relinquish their execution slots > - > > Key: AIRFLOW-2001 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2001 > Project: Apache Airflow > Issue Type: Bug > Components: db, scheduler >Reporter: Yati >Assignee: Yati >Priority: Major > > A sensor task instance should not take up an execution slot for the entirety > of its lifetime (as is currently the case). Indeed, for reasons outlined > below, it would be better if sensor execution was preempted by the scheduler > by parking it away from the slot till the next poll. > Some sensors sense for a condition to be true which is affected only by an > external party (e.g., materialization by external means of certain rows in a > table). By external, I mean external to the Airflow installation in question, > such that the producing entity itself does not need an execution slot in an > Airflow pool. If all sensors and their dependencies were of this nature, > there would be no issue. Unfortunately, a lot of real world DAGs have sensor > dependencies on results produced by another task, typically in some other > DAG, but scheduled by the same Airflow scheduler. > Consider a simple example (arrow direction represents "must happen before", > just like in Airflow): DAG1(a >> b) and DAG2(c:sensor(DAG1.b) >> d). In other > words, The opening task c of the second dag has a sensor dependency on the > ending task b of the first dag. Imagine we have a single pool with 10 > execution slots, and somehow task instances for c fill up the pool, while the > corresponding task instances of DAG1.b have not had a chance to execute (in > the real world this happens because of, say, back-fills or reprocesses by > clearing those sensors instances and their upstream). This is a deadlock > situation, since no progress can be made here – the sensors have filled up > the pool waiting on tasks that themselves will never get a chance to run. > This problem has been [acknowledged > here|https://cwiki.apache.org/confluence/display/AIRFLOW/Common+Pitfalls] > One way (suggested by Fokko) to solve this is to always run sensors on their > pool, and to be careful with the concurrency settings of sensor tasks. This > is what a lot of users do now, but there are better solutions to this. Since > all the sensor interface allows for is a poll, we can, after each poll, > "park" the sensor's execution slot and yield it to other tasks. In the above > scenario, there would be no "filling up" of the pool by sensors tasks, as > they will be polled, determined to be still unfulfilled, and then parked > away, thereby giving a chance to other tasks. > This would likely have some changes to the DB, and of course to the scheduler. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-2639) Dagrun of subdags is set to RUNNING immediately
[ https://issues.apache.org/jira/browse/AIRFLOW-2639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor closed AIRFLOW-2639. -- Resolution: Invalid > Dagrun of subdags is set to RUNNING immediately > --- > > Key: AIRFLOW-2639 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2639 > Project: Apache Airflow > Issue Type: Bug >Reporter: Chao-Han Tsai >Assignee: Chao-Han Tsai >Priority: Major > > This change has a side-effect. The subdag run and it's task instances are > eagerly created, the subdag is immediately set to "RUNNING" state. This means > it is immediately visible in the UI (tree view and dagrun view). > In our case we skip the SubDagOperator base on some conditions. However the > subdag run is then still visible in th UI and in "RUNNING" state which looks > scary, see attached screenshot. Before there was no subdag run visible at all > for skipped subdags. > One option I see is to not set subdags to "RUNNING" state but "NONE". Then it > will still be visible in the UI but not as running. Another idea is to try to > pass the conf directly in the SubDagOperator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] codecov-io edited a comment on issue #4023: [AIRFLOW-3174] Refine Docstring for SQL Operators
codecov-io edited a comment on issue #4023: [AIRFLOW-3174] Refine Docstring for SQL Operators URL: https://github.com/apache/incubator-airflow/pull/4023#issuecomment-428154092 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4023?src=pr=h1) Report > Merging [#4023](https://codecov.io/gh/apache/incubator-airflow/pull/4023?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/4e94997bc9b09df9ccd6d1a73abe0c5b091f480c?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4023?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#4023 +/- ## === Coverage 75.77% 75.77% === Files 199 199 Lines 1594615946 === Hits1208312083 Misses 3863 3863 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4023?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/operators/mssql\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4023/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXNzcWxfb3BlcmF0b3IucHk=) | `0% <ø> (ø)` | :arrow_up: | | [airflow/operators/sqlite\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4023/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvc3FsaXRlX29wZXJhdG9yLnB5) | `100% <ø> (ø)` | :arrow_up: | | [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4023/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `0% <ø> (ø)` | :arrow_up: | | [airflow/operators/oracle\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4023/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvb3JhY2xlX29wZXJhdG9yLnB5) | `0% <ø> (ø)` | :arrow_up: | | [airflow/operators/jdbc\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4023/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvamRiY19vcGVyYXRvci5weQ==) | `0% <ø> (ø)` | :arrow_up: | | [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4023/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `0% <ø> (ø)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4023?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4023?src=pr=footer). Last update [4e94997...4600b3a](https://codecov.io/gh/apache/incubator-airflow/pull/4023?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4023: [AIRFLOW-3174] Refine Docstring for SQL Operators
codecov-io edited a comment on issue #4023: [AIRFLOW-3174] Refine Docstring for SQL Operators URL: https://github.com/apache/incubator-airflow/pull/4023#issuecomment-428154092 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4023?src=pr=h1) Report > Merging [#4023](https://codecov.io/gh/apache/incubator-airflow/pull/4023?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/4e94997bc9b09df9ccd6d1a73abe0c5b091f480c?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4023?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#4023 +/- ## === Coverage 75.77% 75.77% === Files 199 199 Lines 1594615946 === Hits1208312083 Misses 3863 3863 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4023?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/operators/mssql\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4023/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXNzcWxfb3BlcmF0b3IucHk=) | `0% <ø> (ø)` | :arrow_up: | | [airflow/operators/sqlite\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4023/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvc3FsaXRlX29wZXJhdG9yLnB5) | `100% <ø> (ø)` | :arrow_up: | | [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4023/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `0% <ø> (ø)` | :arrow_up: | | [airflow/operators/oracle\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4023/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvb3JhY2xlX29wZXJhdG9yLnB5) | `0% <ø> (ø)` | :arrow_up: | | [airflow/operators/jdbc\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4023/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvamRiY19vcGVyYXRvci5weQ==) | `0% <ø> (ø)` | :arrow_up: | | [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4023/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `0% <ø> (ø)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4023?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4023?src=pr=footer). Last update [4e94997...4600b3a](https://codecov.io/gh/apache/incubator-airflow/pull/4023?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] XD-DENG commented on issue #4025: [AIRFLOW-3175] Fix docstring format in airflow/jobs.py
XD-DENG commented on issue #4025: [AIRFLOW-3175] Fix docstring format in airflow/jobs.py URL: https://github.com/apache/incubator-airflow/pull/4025#issuecomment-428229732 Hi @ashb, thanks for merging this PR. I was confused by this since sometimes I saw other committers fixed docstrings with a JIRA ticket. So also added Jira issue for this. May we make the information in this PR template more "explicit"? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3175) Fix docstring format in airflow/jobs.py
[
https://issues.apache.org/jira/browse/AIRFLOW-3175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16643578#comment-16643578
]
ASF GitHub Bot commented on AIRFLOW-3175:
-
ashb closed pull request #4025: [AIRFLOW-3175] Fix docstring format in
airflow/jobs.py
URL: https://github.com/apache/incubator-airflow/pull/4025
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/jobs.py b/airflow/jobs.py
index da1089d690..b224f75545 100644
--- a/airflow/jobs.py
+++ b/airflow/jobs.py
@@ -349,10 +349,10 @@ def _launch_process(result_queue,
:param file_path: the file to process
:type file_path: unicode
:param pickle_dags: whether to pickle the DAGs found in the file and
-save them to the DB
+save them to the DB
:type pickle_dags: bool
:param dag_id_white_list: if specified, only examine DAG ID's that are
-in this list
+in this list
:type dag_id_white_list: list[unicode]
:param thread_name: the name to use for the process that is launched
:type thread_name: unicode
@@ -424,6 +424,7 @@ def start(self):
def terminate(self, sigkill=False):
"""
Terminate (and then kill) the process launched to process the file.
+
:param sigkill: whether to issue a SIGKILL if SIGTERM doesn't work.
:type sigkill: bool
"""
@@ -452,6 +453,7 @@ def pid(self):
def exit_code(self):
"""
After the process is finished, this can be called to get the return
code
+
:return: the exit code of the process
:rtype: int
"""
@@ -463,6 +465,7 @@ def exit_code(self):
def done(self):
"""
Check if the process launched to process this file is done.
+
:return: whether the process is finished running
:rtype: bool
"""
@@ -544,16 +547,18 @@ def __init__(
:param dag_ids: if specified, only schedule tasks with these DAG IDs
:type dag_ids: list[unicode]
:param subdir: directory containing Python files with Airflow DAG
-definitions, or a specific path to a file
+definitions, or a specific path to a file
:type subdir: unicode
:param num_runs: The number of times to try to schedule each DAG file.
--1 for unlimited within the run_duration.
+-1 for unlimited within the run_duration.
+:type num_runs: int
:param processor_poll_interval: The number of seconds to wait between
-polls of running processors
+polls of running processors
+:type processor_poll_interval: int
:param run_duration: how long to run (in seconds) before exiting
:type run_duration: int
:param do_pickle: once a DAG object is obtained by executing the Python
-file, whether to serialize the DAG object to the DB
+file, whether to serialize the DAG object to the DB
:type do_pickle: bool
"""
# for BaseJob compatibility
@@ -782,7 +787,7 @@ def update_import_errors(session, dagbag):
def create_dag_run(self, dag, session=None):
"""
This method checks whether a new DagRun needs to be created
-for a DAG based on scheduling interval
+for a DAG based on scheduling interval.
Returns DagRun if one is scheduled. Otherwise returns None.
"""
if dag.schedule_interval and conf.getboolean('scheduler',
'USE_JOB_SCHEDULE'):
@@ -990,7 +995,7 @@ def _change_state_for_tis_without_dagrun(self,
:param new_state: set TaskInstances to this state
:type new_state: State
:param simple_dag_bag: TaskInstances associated with DAGs in the
-simple_dag_bag and with states in the old_state will be examined
+simple_dag_bag and with states in the old_state will be examined
:type simple_dag_bag: SimpleDagBag
"""
tis_changed = 0
@@ -1061,7 +1066,7 @@ def _find_executable_task_instances(self, simple_dag_bag,
states, session=None):
dag concurrency, executor state, and priority.
:param simple_dag_bag: TaskInstances associated with DAGs in the
-simple_dag_bag will be fetched from the DB and executed
+simple_dag_bag will be fetched from the DB and executed
:type simple_dag_bag: SimpleDagBag
:param executor: the executor that runs task instances
:type executor: BaseExecutor
@@ -1374,7 +1379,7 @@ def _execute_task_instances(self,
3. Enqueue the TIs in the executor.
:param simple_dag_bag: TaskInstances associated with DAGs in the
-
[GitHub] ashb closed pull request #4025: [AIRFLOW-3175] Fix docstring format in airflow/jobs.py
ashb closed pull request #4025: [AIRFLOW-3175] Fix docstring format in
airflow/jobs.py
URL: https://github.com/apache/incubator-airflow/pull/4025
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/jobs.py b/airflow/jobs.py
index da1089d690..b224f75545 100644
--- a/airflow/jobs.py
+++ b/airflow/jobs.py
@@ -349,10 +349,10 @@ def _launch_process(result_queue,
:param file_path: the file to process
:type file_path: unicode
:param pickle_dags: whether to pickle the DAGs found in the file and
-save them to the DB
+save them to the DB
:type pickle_dags: bool
:param dag_id_white_list: if specified, only examine DAG ID's that are
-in this list
+in this list
:type dag_id_white_list: list[unicode]
:param thread_name: the name to use for the process that is launched
:type thread_name: unicode
@@ -424,6 +424,7 @@ def start(self):
def terminate(self, sigkill=False):
"""
Terminate (and then kill) the process launched to process the file.
+
:param sigkill: whether to issue a SIGKILL if SIGTERM doesn't work.
:type sigkill: bool
"""
@@ -452,6 +453,7 @@ def pid(self):
def exit_code(self):
"""
After the process is finished, this can be called to get the return
code
+
:return: the exit code of the process
:rtype: int
"""
@@ -463,6 +465,7 @@ def exit_code(self):
def done(self):
"""
Check if the process launched to process this file is done.
+
:return: whether the process is finished running
:rtype: bool
"""
@@ -544,16 +547,18 @@ def __init__(
:param dag_ids: if specified, only schedule tasks with these DAG IDs
:type dag_ids: list[unicode]
:param subdir: directory containing Python files with Airflow DAG
-definitions, or a specific path to a file
+definitions, or a specific path to a file
:type subdir: unicode
:param num_runs: The number of times to try to schedule each DAG file.
--1 for unlimited within the run_duration.
+-1 for unlimited within the run_duration.
+:type num_runs: int
:param processor_poll_interval: The number of seconds to wait between
-polls of running processors
+polls of running processors
+:type processor_poll_interval: int
:param run_duration: how long to run (in seconds) before exiting
:type run_duration: int
:param do_pickle: once a DAG object is obtained by executing the Python
-file, whether to serialize the DAG object to the DB
+file, whether to serialize the DAG object to the DB
:type do_pickle: bool
"""
# for BaseJob compatibility
@@ -782,7 +787,7 @@ def update_import_errors(session, dagbag):
def create_dag_run(self, dag, session=None):
"""
This method checks whether a new DagRun needs to be created
-for a DAG based on scheduling interval
+for a DAG based on scheduling interval.
Returns DagRun if one is scheduled. Otherwise returns None.
"""
if dag.schedule_interval and conf.getboolean('scheduler',
'USE_JOB_SCHEDULE'):
@@ -990,7 +995,7 @@ def _change_state_for_tis_without_dagrun(self,
:param new_state: set TaskInstances to this state
:type new_state: State
:param simple_dag_bag: TaskInstances associated with DAGs in the
-simple_dag_bag and with states in the old_state will be examined
+simple_dag_bag and with states in the old_state will be examined
:type simple_dag_bag: SimpleDagBag
"""
tis_changed = 0
@@ -1061,7 +1066,7 @@ def _find_executable_task_instances(self, simple_dag_bag,
states, session=None):
dag concurrency, executor state, and priority.
:param simple_dag_bag: TaskInstances associated with DAGs in the
-simple_dag_bag will be fetched from the DB and executed
+simple_dag_bag will be fetched from the DB and executed
:type simple_dag_bag: SimpleDagBag
:param executor: the executor that runs task instances
:type executor: BaseExecutor
@@ -1374,7 +1379,7 @@ def _execute_task_instances(self,
3. Enqueue the TIs in the executor.
:param simple_dag_bag: TaskInstances associated with DAGs in the
-simple_dag_bag will be fetched from the DB and executed
+simple_dag_bag will be fetched from the DB and executed
:type simple_dag_bag: SimpleDagBag
:param states: Execute TaskInstances in these states
:type
[GitHub] ashb commented on issue #4025: [AIRFLOW-3175] Fix docstring format in airflow/jobs.py
ashb commented on issue #4025: [AIRFLOW-3175] Fix docstring format in airflow/jobs.py URL: https://github.com/apache/incubator-airflow/pull/4025#issuecomment-428227683 BTW this is a prime example of "In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-XXX], code changes always need a Jira issue." This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3175) Fix docstring format in airflow/jobs.py
[ https://issues.apache.org/jira/browse/AIRFLOW-3175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16643576#comment-16643576 ] ASF GitHub Bot commented on AIRFLOW-3175: - XD-DENG opened a new pull request #4025: [AIRFLOW-3175] Fix docstring format in airflow/jobs.py URL: https://github.com/apache/incubator-airflow/pull/4025 Make sure you have checked _all_ steps below. ### Jira - [x] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-3175 - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. ### Description - [x] Here are some details about my PR, including screenshots of any UI changes: Fix docstrings in `ariflow/jobs.py` which could not be parsed by Sphinx syntax properly. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Fix docstring format in airflow/jobs.py > --- > > Key: AIRFLOW-3175 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3175 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Trivial > > Some minor issues in docstring format in airflow/jobs.py -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ashb commented on a change in pull request #4010: [AIRFLOW-1837] Respect task start_date when different from dag's
ashb commented on a change in pull request #4010: [AIRFLOW-1837] Respect task
start_date when different from dag's
URL: https://github.com/apache/incubator-airflow/pull/4010#discussion_r223739630
##
File path: tests/www_rbac/test_views.py
##
@@ -267,8 +267,8 @@ def test_mount(self):
class TestAirflowBaseViews(TestBase):
-default_date = timezone.datetime(2018, 3, 1)
-run_id = "test_{}".format(models.DagRun.id_for_date(default_date))
+EXAMPLE_DAG_DEFAULT_DATE = dates.days_ago(2)
Review comment:
Is this sort of change needed? It looks like it's an unrelated change from
the diff.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] XD-DENG opened a new pull request #4025: [AIRFLOW-3175] Fix docstring format in airflow/jobs.py
XD-DENG opened a new pull request #4025: [AIRFLOW-3175] Fix docstring format in airflow/jobs.py URL: https://github.com/apache/incubator-airflow/pull/4025 Make sure you have checked _all_ steps below. ### Jira - [x] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-3175 - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. ### Description - [x] Here are some details about my PR, including screenshots of any UI changes: Fix docstrings in `ariflow/jobs.py` which could not be parsed by Sphinx syntax properly. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb commented on issue #4015: [AIRFLOW-2789] Create single node cluster
ashb commented on issue #4015: [AIRFLOW-2789] Create single node cluster URL: https://github.com/apache/incubator-airflow/pull/4015#issuecomment-428226072 Code-cove is sometimes a bit noisy :( This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] phani8996 commented on issue #3992: [AIRFLOW-620] Feature to tail custom number of logs instead of rendering whole log
phani8996 commented on issue #3992: [AIRFLOW-620] Feature to tail custom number of logs instead of rendering whole log URL: https://github.com/apache/incubator-airflow/pull/3992#issuecomment-428226076 > @msumit This isn't the equivalent of `tail -f` -- it doesn't stream the logs in as they happen, it's just `tail -n 300` etc. That's true sumit. This feature doesn't tail logs in fly mode. I am planning to add that in a separate PR as this brings a lot of changes to current log viewing mechanism and this maybe unstable and experimental. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] phani8996 commented on a change in pull request #3992: [AIRFLOW-620] Feature to tail custom number of logs instead of rendering whole log
phani8996 commented on a change in pull request #3992: [AIRFLOW-620] Feature to
tail custom number of logs instead of rendering whole log
URL: https://github.com/apache/incubator-airflow/pull/3992#discussion_r223736747
##
File path: airflow/www/views.py
##
@@ -836,6 +839,21 @@ def log(self, session=None):
dttm = pendulum.parse(execution_date)
form = DateTimeForm(data={'execution_date': dttm})
dag = dagbag.get_dag(dag_id)
+metadata = {}
+tailing_required = False
+tail_lines_list = [100, 200, 500, 1000]
Review comment:
I am using 3 values from config for this feature. A flag(`tail_logs`),
default value(`num_lines`) and comma separated string(`tail_lines_list`). Above
hard coded value is added to avoid app crashes in the cases where flag is true
and `tail_lines_list` variable is not defined.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-3175) Fix docstring format in airflow/jobs.py
Xiaodong DENG created AIRFLOW-3175: -- Summary: Fix docstring format in airflow/jobs.py Key: AIRFLOW-3175 URL: https://issues.apache.org/jira/browse/AIRFLOW-3175 Project: Apache Airflow Issue Type: Improvement Reporter: Xiaodong DENG Assignee: Xiaodong DENG Some minor issues in docstring format in airflow/jobs.py -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] phani8996 commented on a change in pull request #3992: [AIRFLOW-620] Feature to tail custom number of logs instead of rendering whole log
phani8996 commented on a change in pull request #3992: [AIRFLOW-620] Feature to
tail custom number of logs instead of rendering whole log
URL: https://github.com/apache/incubator-airflow/pull/3992#discussion_r223734185
##
File path: airflow/www/views.py
##
@@ -836,6 +839,21 @@ def log(self, session=None):
dttm = pendulum.parse(execution_date)
form = DateTimeForm(data={'execution_date': dttm})
dag = dagbag.get_dag(dag_id)
+metadata = {}
+tailing_required = False
+tail_lines_list = [100, 200, 500, 1000]
+num_lines = 100
+if conf.has_option('webserver', 'tail_lines_list'):
+tail_lines_list = conf.get('webserver', 'tail_lines_list')
+tail_lines_list = list(map(int, tail_lines_list.split(',')))
+if conf.has_option('webserver', 'tail_logs'):
+tailing_required = conf.getboolean('webserver', 'tail_logs')
+if tailing_required and conf.has_option('webserver', 'num_lines'):
+num_lines = conf.getint('webserver', 'num_lines')
+metadata["num_lines"] = num_lines
Review comment:
I am essentially using three fields for this feature. Boolean flag
`tail_logs` in `webserver` to turn on and off this feature. `tail_lines_list`
in `webserver` to show default entries in dropdown and finally `num_lines` in
`webserver` to pick default number of lines to tail when user lands on
`get_logs` page.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] phani8996 commented on a change in pull request #3992: [AIRFLOW-620] Feature to tail custom number of logs instead of rendering whole log
phani8996 commented on a change in pull request #3992: [AIRFLOW-620] Feature to
tail custom number of logs instead of rendering whole log
URL: https://github.com/apache/incubator-airflow/pull/3992#discussion_r223732458
##
File path: airflow/www/views.py
##
@@ -836,6 +839,21 @@ def log(self, session=None):
dttm = pendulum.parse(execution_date)
form = DateTimeForm(data={'execution_date': dttm})
dag = dagbag.get_dag(dag_id)
+metadata = {}
+tailing_required = False
+tail_lines_list = [100, 200, 500, 1000]
+num_lines = 100
+if conf.has_option('webserver', 'tail_lines_list'):
+tail_lines_list = conf.get('webserver', 'tail_lines_list')
+tail_lines_list = list(map(int, tail_lines_list.split(',')))
+if conf.has_option('webserver', 'tail_logs'):
+tailing_required = conf.getboolean('webserver', 'tail_logs')
+if tailing_required and conf.has_option('webserver', 'num_lines'):
+num_lines = conf.getint('webserver', 'num_lines')
+metadata["num_lines"] = num_lines
+if num_lines in tail_lines_list:
+tail_lines_list.remove(num_lines)
Review comment:
That's to avoid duplicates. lets say user have added in config that dropdown
should contain entries 100,200,500,1000 and user picked 100 as default number
of lines to tail when user lands on get_logs page. Without this condition
airflow ending showing 2 entries for 100 in dropdown. This will ensure no
duplicates in such case
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-3174) Refine Docstring for SQL Operators
Xiaodong DENG created AIRFLOW-3174:
--
Summary: Refine Docstring for SQL Operators
Key: AIRFLOW-3174
URL: https://issues.apache.org/jira/browse/AIRFLOW-3174
Project: Apache Airflow
Issue Type: Improvement
Components: operators
Reporter: Xiaodong DENG
Assignee: Xiaodong DENG
SQL operators (based on {{DbApiHook}}) receive {{sql}} and {{parameters}}
(optional) as arguments. But {{parameters}} were not documented.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3173) Add _cmd options for password config options
[
https://issues.apache.org/jira/browse/AIRFLOW-3173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16643476#comment-16643476
]
ASF GitHub Bot commented on AIRFLOW-3173:
-
ashb opened a new pull request #4024: [AIRFLOW-3173] Add _cmd options for
password config options
URL: https://github.com/apache/incubator-airflow/pull/4024
Make sure you have checked _all_ steps below.
### Jira
- [x] https://issues.apache.org/jira/browse/AIRFLOW-3173
### Description
There were a few more "password" config options added over the last few
months that didn't have _cmd options. Any config option that is a
password should be able to be provided via a _cmd version.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason: None needed really?
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- Updated docs to mention all config options that are supported by _cmd
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add _cmd options for password config options
>
>
> Key: AIRFLOW-3173
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3173
> Project: Apache Airflow
> Issue Type: New Feature
>Reporter: Ash Berlin-Taylor
>Priority: Major
> Fix For: 1.10.1
>
>
> Right now we support specifying a command to get a small number of config
> options ({{sql_alchemy_conn_cmd}} for instance) but there are some new config
> options that contain a password that don't have a {{_cmd }} option.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] ashb opened a new pull request #4024: [AIRFLOW-3173] Add _cmd options for password config options
ashb opened a new pull request #4024: [AIRFLOW-3173] Add _cmd options for
password config options
URL: https://github.com/apache/incubator-airflow/pull/4024
Make sure you have checked _all_ steps below.
### Jira
- [x] https://issues.apache.org/jira/browse/AIRFLOW-3173
### Description
There were a few more "password" config options added over the last few
months that didn't have _cmd options. Any config option that is a
password should be able to be provided via a _cmd version.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason: None needed really?
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- Updated docs to mention all config options that are supported by _cmd
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ashb commented on a change in pull request #3992: [AIRFLOW-620] Feature to tail custom number of logs instead of rendering whole log
ashb commented on a change in pull request #3992: [AIRFLOW-620] Feature to tail
custom number of logs instead of rendering whole log
URL: https://github.com/apache/incubator-airflow/pull/3992#discussion_r223700638
##
File path: airflow/www/views.py
##
@@ -836,6 +839,21 @@ def log(self, session=None):
dttm = pendulum.parse(execution_date)
form = DateTimeForm(data={'execution_date': dttm})
dag = dagbag.get_dag(dag_id)
+metadata = {}
+tailing_required = False
+tail_lines_list = [100, 200, 500, 1000]
Review comment:
Don't hard-code defaults like this. Instead put it in to default-airflow.cfg.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ashb commented on a change in pull request #3992: [AIRFLOW-620] Feature to tail custom number of logs instead of rendering whole log
ashb commented on a change in pull request #3992: [AIRFLOW-620] Feature to tail
custom number of logs instead of rendering whole log
URL: https://github.com/apache/incubator-airflow/pull/3992#discussion_r223702250
##
File path: airflow/www/views.py
##
@@ -836,6 +839,21 @@ def log(self, session=None):
dttm = pendulum.parse(execution_date)
form = DateTimeForm(data={'execution_date': dttm})
dag = dagbag.get_dag(dag_id)
+metadata = {}
+tailing_required = False
+tail_lines_list = [100, 200, 500, 1000]
+num_lines = 100
+if conf.has_option('webserver', 'tail_lines_list'):
+tail_lines_list = conf.get('webserver', 'tail_lines_list')
+tail_lines_list = list(map(int, tail_lines_list.split(',')))
+if conf.has_option('webserver', 'tail_logs'):
+tailing_required = conf.getboolean('webserver', 'tail_logs')
+if tailing_required and conf.has_option('webserver', 'num_lines'):
+num_lines = conf.getint('webserver', 'num_lines')
+metadata["num_lines"] = num_lines
Review comment:
Please find a way of re-writing this. The nested conditions and checks make
it hard to see what the intent here is. Perhaps some comment saying when/why we
do things would help me understand this code.
I'm also notsure what the webserver -> tail_logs setting is for. Moving that
to default config means you can add a comment to it
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ashb commented on a change in pull request #3992: [AIRFLOW-620] Feature to tail custom number of logs instead of rendering whole log
ashb commented on a change in pull request #3992: [AIRFLOW-620] Feature to tail
custom number of logs instead of rendering whole log
URL: https://github.com/apache/incubator-airflow/pull/3992#discussion_r223702228
##
File path: airflow/www/views.py
##
@@ -836,6 +839,21 @@ def log(self, session=None):
dttm = pendulum.parse(execution_date)
form = DateTimeForm(data={'execution_date': dttm})
dag = dagbag.get_dag(dag_id)
+metadata = {}
+tailing_required = False
+tail_lines_list = [100, 200, 500, 1000]
+num_lines = 100
+if conf.has_option('webserver', 'tail_lines_list'):
+tail_lines_list = conf.get('webserver', 'tail_lines_list')
+tail_lines_list = list(map(int, tail_lines_list.split(',')))
+if conf.has_option('webserver', 'tail_logs'):
+tailing_required = conf.getboolean('webserver', 'tail_logs')
+if tailing_required and conf.has_option('webserver', 'num_lines'):
+num_lines = conf.getint('webserver', 'num_lines')
+metadata["num_lines"] = num_lines
+if num_lines in tail_lines_list:
+tail_lines_list.remove(num_lines)
Review comment:
What is this conditional for? I can't work it out
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ashb commented on a change in pull request #3992: [AIRFLOW-620] Feature to tail custom number of logs instead of rendering whole log
ashb commented on a change in pull request #3992: [AIRFLOW-620] Feature to tail
custom number of logs instead of rendering whole log
URL: https://github.com/apache/incubator-airflow/pull/3992#discussion_r223701009
##
File path: airflow/www/templates/airflow/ti_log.html
##
@@ -29,6 +29,28 @@ {{ title }}
{% endfor %}
+ {% if tailing_required %}
+
+
+
+
+Tail Logs
+
+
+{% for tail_line in tail_lines_list %}
Review comment:
I'm not a fan of this being shown for log storage methods (such as S3/GCP)
that don't support this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ashb commented on a change in pull request #3992: [AIRFLOW-620] Feature to tail custom number of logs instead of rendering whole log
ashb commented on a change in pull request #3992: [AIRFLOW-620] Feature to tail
custom number of logs instead of rendering whole log
URL: https://github.com/apache/incubator-airflow/pull/3992#discussion_r223699733
##
File path: airflow/bin/cli.py
##
@@ -1023,17 +1023,41 @@ def scheduler(args):
@cli_utils.action_logging
def serve_logs(args):
print("Starting flask")
-import flask
-flask_app = flask.Flask(__name__)
+from flask import Flask, request, Response, stream_with_context,
send_from_directory
+flask_app = Flask(__name__)
@flask_app.route('/log/')
def serve_logs(filename): # noqa
+def tail_logs(logdir, filename, num_lines):
+logpath = "{logdir}/{filename}".format(logdir=logdir,
filename=filename)
+logsize = os.path.getsize(logpath)
+if logsize >= 100 * 1024 * 1024:
+p1 = subprocess.Popen(["tail", "-n " + str(num_lines),
filename],
+ stdout=subprocess.PIPE, cwd=log)
+out, err = p1.communicate()
+out = "Tailing file\n\n" + out.decode("utf-8")
+else:
+fl = open("{log}//{filename}".format(log=log,
filename=filename), "r")
+lines = fl.readlines()
+fl.close()
+out = "".join(l for l in lines[-num_lines:])
+line = "* Showing only last {num_lines} lines from {filename}
*" \
Review comment:
Messages like this don't belong in here - that is something that the
FileTaskHandler etc should add.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ashb commented on issue #3992: [AIRFLOW-620] Feature to tail custom number of logs instead of rendering whole log
ashb commented on issue #3992: [AIRFLOW-620] Feature to tail custom number of logs instead of rendering whole log URL: https://github.com/apache/incubator-airflow/pull/3992#issuecomment-428190709 @msumit This isn't the equivalent of `tail -f` -- it doesn't stream the logs in as they happen, it's just `tail -n 300` etc. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-3173) Add _cmd options for password config options
Ash Berlin-Taylor created AIRFLOW-3173:
--
Summary: Add _cmd options for password config options
Key: AIRFLOW-3173
URL: https://issues.apache.org/jira/browse/AIRFLOW-3173
Project: Apache Airflow
Issue Type: New Feature
Reporter: Ash Berlin-Taylor
Fix For: 1.10.1
Right now we support specifying a command to get a small number of config
options ({{sql_alchemy_conn_cmd}} for instance) but there are some new config
options that contain a password that don't have a {{_cmd }} option.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] ashb commented on issue #3994: [AIRFLOW-3136] Add retry_number to TaskInstance Key property to avoid race condition
ashb commented on issue #3994: [AIRFLOW-3136] Add retry_number to TaskInstance Key property to avoid race condition URL: https://github.com/apache/incubator-airflow/pull/3994#issuecomment-428185966 Is there an easy way I can reproduce this locally to see the bug for myself? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb commented on a change in pull request #4014: [AIRFLOW-3168] add healthcheck for mysql setup in CI
ashb commented on a change in pull request #4014: [AIRFLOW-3168] add
healthcheck for mysql setup in CI
URL: https://github.com/apache/incubator-airflow/pull/4014#discussion_r223693000
##
File path: scripts/ci/3-setup-databases.sh
##
@@ -20,4 +20,9 @@ set -exuo pipefail
MYSQL_HOST=mysql
+# make sure mysql is done setting up
+while ! mysqladmin ping -h ${MYSQL_HOST} -u root --silent; do
+sleep 1
+done
Review comment:
Probably a good idea.
Though also: why do we set up mysql when we are running against sqlite or
postgres?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] codecov-io edited a comment on issue #3138: [AIRFLOW-2221] Create DagFetcher abstraction
codecov-io edited a comment on issue #3138: [AIRFLOW-2221] Create DagFetcher abstraction URL: https://github.com/apache/incubator-airflow/pull/3138#issuecomment-373965660 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3138?src=pr=h1) Report > Merging [#3138](https://codecov.io/gh/apache/incubator-airflow/pull/3138?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/b8be322d3badfeadfa8f08e0bf92a12a6cd26418?src=pr=desc) will **decrease** coverage by `2.62%`. > The diff coverage is `87.57%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3138?src=pr=tree) ```diff @@Coverage Diff @@ ## master#3138 +/- ## == - Coverage 75.79% 73.16% -2.63% == Files 199 187 -12 Lines 1594612716-3230 == - Hits12086 9304-2782 + Misses 3860 3412 -448 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3138?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/plugins\_manager.py](https://codecov.io/gh/apache/incubator-airflow/pull/3138/diff?src=pr=tree#diff-YWlyZmxvdy9wbHVnaW5zX21hbmFnZXIucHk=) | `92.85% <100%> (+1.49%)` | :arrow_up: | | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/3138/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `87.56% <100%> (-4.15%)` | :arrow_down: | | [airflow/dag/fetchers/s3.py](https://codecov.io/gh/apache/incubator-airflow/pull/3138/diff?src=pr=tree#diff-YWlyZmxvdy9kYWcvZmV0Y2hlcnMvczMucHk=) | `100% <100%> (ø)` | | | [airflow/dag/fetchers/git.py](https://codecov.io/gh/apache/incubator-airflow/pull/3138/diff?src=pr=tree#diff-YWlyZmxvdy9kYWcvZmV0Y2hlcnMvZ2l0LnB5) | `100% <100%> (ø)` | | | [airflow/\_\_init\_\_.py](https://codecov.io/gh/apache/incubator-airflow/pull/3138/diff?src=pr=tree#diff-YWlyZmxvdy9fX2luaXRfXy5weQ==) | `81.25% <100%> (+6.96%)` | :arrow_up: | | [airflow/dag/fetchers/hdfs.py](https://codecov.io/gh/apache/incubator-airflow/pull/3138/diff?src=pr=tree#diff-YWlyZmxvdy9kYWcvZmV0Y2hlcnMvaGRmcy5weQ==) | `100% <100%> (ø)` | | | [airflow/dag/fetchers/\_\_init\_\_.py](https://codecov.io/gh/apache/incubator-airflow/pull/3138/diff?src=pr=tree#diff-YWlyZmxvdy9kYWcvZmV0Y2hlcnMvX19pbml0X18ucHk=) | `100% <100%> (ø)` | | | [airflow/dag/fetchers/gcs.py](https://codecov.io/gh/apache/incubator-airflow/pull/3138/diff?src=pr=tree#diff-YWlyZmxvdy9kYWcvZmV0Y2hlcnMvZ2NzLnB5) | `100% <100%> (ø)` | | | [airflow/dag/fetchers/filesystem.py](https://codecov.io/gh/apache/incubator-airflow/pull/3138/diff?src=pr=tree#diff-YWlyZmxvdy9kYWcvZmV0Y2hlcnMvZmlsZXN5c3RlbS5weQ==) | `82.9% <82.9%> (ø)` | | | [airflow/dag/fetchers/base.py](https://codecov.io/gh/apache/incubator-airflow/pull/3138/diff?src=pr=tree#diff-YWlyZmxvdy9kYWcvZmV0Y2hlcnMvYmFzZS5weQ==) | `85.71% <85.71%> (ø)` | | | ... and [213 more](https://codecov.io/gh/apache/incubator-airflow/pull/3138/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3138?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3138?src=pr=footer). Last update [b8be322...6979f0d](https://codecov.io/gh/apache/incubator-airflow/pull/3138?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #3975: [AIRFLOW-XXX] Pin docker requirement version
codecov-io edited a comment on issue #3975: [AIRFLOW-XXX] Pin docker requirement version URL: https://github.com/apache/incubator-airflow/pull/3975#issuecomment-425622625 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3975?src=pr=h1) Report > Merging [#3975](https://codecov.io/gh/apache/incubator-airflow/pull/3975?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/b8be322d3badfeadfa8f08e0bf92a12a6cd26418?src=pr=desc) will **decrease** coverage by `0.74%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3975?src=pr=tree) ```diff @@Coverage Diff @@ ## master#3975 +/- ## == - Coverage 75.79% 75.04% -0.75% == Files 199 199 Lines 1594615965 +19 == - Hits1208611981 -105 - Misses 3860 3984 +124 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3975?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/sensors/s3\_prefix\_sensor.py](https://codecov.io/gh/apache/incubator-airflow/pull/3975/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL3MzX3ByZWZpeF9zZW5zb3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/sensors/s3\_key\_sensor.py](https://codecov.io/gh/apache/incubator-airflow/pull/3975/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL3MzX2tleV9zZW5zb3IucHk=) | `30.3% <0%> (-69.7%)` | :arrow_down: | | [airflow/sensors/sql\_sensor.py](https://codecov.io/gh/apache/incubator-airflow/pull/3975/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL3NxbF9zZW5zb3IucHk=) | `90.47% <0%> (-9.53%)` | :arrow_down: | | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/3975/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `88.65% <0%> (-3.06%)` | :arrow_down: | | [airflow/hooks/http\_hook.py](https://codecov.io/gh/apache/incubator-airflow/pull/3975/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9odHRwX2hvb2sucHk=) | `93.75% <0%> (-1.71%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/incubator-airflow/pull/3975/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `89.03% <0%> (-1.32%)` | :arrow_down: | | [airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/3975/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5) | `82.13% <0%> (-0.27%)` | :arrow_down: | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/incubator-airflow/pull/3975/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `79.67% <0%> (-0.17%)` | :arrow_down: | | [airflow/bin/cli.py](https://codecov.io/gh/apache/incubator-airflow/pull/3975/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `64.43% <0%> (-0.05%)` | :arrow_down: | | [airflow/default\_login.py](https://codecov.io/gh/apache/incubator-airflow/pull/3975/diff?src=pr=tree#diff-YWlyZmxvdy9kZWZhdWx0X2xvZ2luLnB5) | `58.97% <0%> (ø)` | :arrow_up: | | ... and [4 more](https://codecov.io/gh/apache/incubator-airflow/pull/3975/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3975?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3975?src=pr=footer). Last update [b8be322...c09eb40](https://codecov.io/gh/apache/incubator-airflow/pull/3975?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #3806: [AIRFLOW-2956] added kubernetes tolerations to kubernetes pod operator
codecov-io edited a comment on issue #3806: [AIRFLOW-2956] added kubernetes tolerations to kubernetes pod operator URL: https://github.com/apache/incubator-airflow/pull/3806#issuecomment-416340975 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3806?src=pr=h1) Report > Merging [#3806](https://codecov.io/gh/apache/incubator-airflow/pull/3806?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/b8be322d3badfeadfa8f08e0bf92a12a6cd26418?src=pr=desc) will **decrease** coverage by `0.3%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3806?src=pr=tree) ```diff @@Coverage Diff @@ ## master#3806 +/- ## == - Coverage 75.79% 75.48% -0.31% == Files 199 199 Lines 1594615947 +1 == - Hits1208612038 -48 - Misses 3860 3909 +49 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3806?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/sensors/s3\_prefix\_sensor.py](https://codecov.io/gh/apache/incubator-airflow/pull/3806/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL3MzX3ByZWZpeF9zZW5zb3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/sensors/s3\_key\_sensor.py](https://codecov.io/gh/apache/incubator-airflow/pull/3806/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL3MzX2tleV9zZW5zb3IucHk=) | `30.3% <0%> (-69.7%)` | :arrow_down: | | [airflow/sensors/sql\_sensor.py](https://codecov.io/gh/apache/incubator-airflow/pull/3806/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL3NxbF9zZW5zb3IucHk=) | `90.47% <0%> (-9.53%)` | :arrow_down: | | [airflow/hooks/http\_hook.py](https://codecov.io/gh/apache/incubator-airflow/pull/3806/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9odHRwX2hvb2sucHk=) | `93.75% <0%> (-1.71%)` | :arrow_down: | | [airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/3806/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5) | `82.13% <0%> (-0.27%)` | :arrow_down: | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/incubator-airflow/pull/3806/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `79.67% <0%> (-0.17%)` | :arrow_down: | | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/3806/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `91.62% <0%> (-0.09%)` | :arrow_down: | | [airflow/bin/cli.py](https://codecov.io/gh/apache/incubator-airflow/pull/3806/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `64.43% <0%> (-0.05%)` | :arrow_down: | | [airflow/default\_login.py](https://codecov.io/gh/apache/incubator-airflow/pull/3806/diff?src=pr=tree#diff-YWlyZmxvdy9kZWZhdWx0X2xvZ2luLnB5) | `58.97% <0%> (ø)` | :arrow_up: | | [airflow/operators/redshift\_to\_s3\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/3806/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcmVkc2hpZnRfdG9fczNfb3BlcmF0b3IucHk=) | `95.55% <0%> (+0.1%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3806?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3806?src=pr=footer). Last update [b8be322...a193629](https://codecov.io/gh/apache/incubator-airflow/pull/3806?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #4023: [AIRFLOW-XXX] Refine Docstring for SQL Operators
codecov-io commented on issue #4023: [AIRFLOW-XXX] Refine Docstring for SQL Operators URL: https://github.com/apache/incubator-airflow/pull/4023#issuecomment-428154092 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4023?src=pr=h1) Report > Merging [#4023](https://codecov.io/gh/apache/incubator-airflow/pull/4023?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/4e94997bc9b09df9ccd6d1a73abe0c5b091f480c?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4023?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#4023 +/- ## === Coverage 75.77% 75.77% === Files 199 199 Lines 1594615946 === Hits1208312083 Misses 3863 3863 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4023?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/operators/mssql\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4023/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXNzcWxfb3BlcmF0b3IucHk=) | `0% <ø> (ø)` | :arrow_up: | | [airflow/operators/sqlite\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4023/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvc3FsaXRlX29wZXJhdG9yLnB5) | `100% <ø> (ø)` | :arrow_up: | | [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4023/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `0% <ø> (ø)` | :arrow_up: | | [airflow/operators/oracle\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4023/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvb3JhY2xlX29wZXJhdG9yLnB5) | `0% <ø> (ø)` | :arrow_up: | | [airflow/operators/jdbc\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4023/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvamRiY19vcGVyYXRvci5weQ==) | `0% <ø> (ø)` | :arrow_up: | | [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4023/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `0% <ø> (ø)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4023?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4023?src=pr=footer). Last update [4e94997...988087d](https://codecov.io/gh/apache/incubator-airflow/pull/4023?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4023: [AIRFLOW-XXX] Refine Docstring for SQL Operators
codecov-io edited a comment on issue #4023: [AIRFLOW-XXX] Refine Docstring for SQL Operators URL: https://github.com/apache/incubator-airflow/pull/4023#issuecomment-428154092 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4023?src=pr=h1) Report > Merging [#4023](https://codecov.io/gh/apache/incubator-airflow/pull/4023?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/4e94997bc9b09df9ccd6d1a73abe0c5b091f480c?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4023?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#4023 +/- ## === Coverage 75.77% 75.77% === Files 199 199 Lines 1594615946 === Hits1208312083 Misses 3863 3863 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4023?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/operators/mssql\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4023/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXNzcWxfb3BlcmF0b3IucHk=) | `0% <ø> (ø)` | :arrow_up: | | [airflow/operators/sqlite\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4023/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvc3FsaXRlX29wZXJhdG9yLnB5) | `100% <ø> (ø)` | :arrow_up: | | [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4023/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `0% <ø> (ø)` | :arrow_up: | | [airflow/operators/oracle\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4023/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvb3JhY2xlX29wZXJhdG9yLnB5) | `0% <ø> (ø)` | :arrow_up: | | [airflow/operators/jdbc\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4023/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvamRiY19vcGVyYXRvci5weQ==) | `0% <ø> (ø)` | :arrow_up: | | [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4023/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `0% <ø> (ø)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4023?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4023?src=pr=footer). Last update [4e94997...988087d](https://codecov.io/gh/apache/incubator-airflow/pull/4023?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] msumit commented on a change in pull request #4014: [AIRFLOW-3168] add healthcheck for mysql setup in CI
msumit commented on a change in pull request #4014: [AIRFLOW-3168] add
healthcheck for mysql setup in CI
URL: https://github.com/apache/incubator-airflow/pull/4014#discussion_r223655917
##
File path: scripts/ci/3-setup-databases.sh
##
@@ -20,4 +20,9 @@ set -exuo pipefail
MYSQL_HOST=mysql
+# make sure mysql is done setting up
+while ! mysqladmin ping -h ${MYSQL_HOST} -u root --silent; do
+sleep 1
+done
Review comment:
shouldn't the while loop need to exist after some time instead of looping
forever?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] XD-DENG opened a new pull request #4023: [AIRFLOW-XXX] Refine Docstring for SQL Operators
XD-DENG opened a new pull request #4023: [AIRFLOW-XXX] Refine Docstring for SQL Operators URL: https://github.com/apache/incubator-airflow/pull/4023 ### Description - [x] Here are some details about my PR, including screenshots of any UI changes: There SQL operators (based on `DbApiHook`) receive `sql` and `parameters` (optional) as arguments. But `parameters` were not documented. The contents I added is based on the docstring in https://github.com/apache/incubator-airflow/blob/master/airflow/hooks/dbapi_hook.py#L88 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3172) AttributeError: 'DagModel' object has no attribute 'execution_date'
[
https://issues.apache.org/jira/browse/AIRFLOW-3172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16643092#comment-16643092
]
Vinnson Lee commented on AIRFLOW-3172:
--
MySQL Version is 5.7
> AttributeError: 'DagModel' object has no attribute 'execution_date'
> ---
>
> Key: AIRFLOW-3172
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3172
> Project: Apache Airflow
> Issue Type: Bug
> Environment: Docker Environment: python:3.6-stretch
>Reporter: Vinnson Lee
>Priority: Major
> Fix For: 1.10.0
>
>
> 2018-10-09 10:13:28,430] ERROR in app: Exception on /admin/dagmodel/ [GET]
> Traceback (most recent call last):
> File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1982, in
> wsgi_app
> response = self.full_dispatch_request()
> File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1614, in
> full_dispatch_request
> rv = self.handle_user_exception(e)
> File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1517, in
> handle_user_exception
> reraise(exc_type, exc_value, tb)
> File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 33, in
> reraise
> raise value
> File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1612, in
> full_dispatch_request
> rv = self.dispatch_request()
> File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1598, in
> dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
> File "/usr/local/lib/python3.6/site-packages/flask_admin/base.py", line 69,
> in inner
> return self._run_view(f, *args, **kwargs)
> File "/usr/local/lib/python3.6/site-packages/flask_admin/base.py", line 368,
> in _run_view
> return fn(self, *args, **kwargs)
> File "/usr/local/lib/python3.6/site-packages/flask_admin/model/base.py",
> line 1900, in index_view
> return_url=self._get_list_url(view_args),
> File "/usr/local/lib/python3.6/site-packages/flask_admin/base.py", line 308,
> in render
> return render_template(template, **kwargs)
> File "/usr/local/lib/python3.6/site-packages/flask/templating.py", line 134,
> in render_template
> context, ctx.app)
> File "/usr/local/lib/python3.6/site-packages/flask/templating.py", line 116,
> in _render
> rv = template.render(context)
> File "/usr/local/lib/python3.6/site-packages/jinja2/environment.py", line
> 989, in render
> return self.environment.handle_exception(exc_info, True)
> File "/usr/local/lib/python3.6/site-packages/jinja2/environment.py", line
> 754, in handle_exception
> reraise(exc_type, exc_value, tb)
> File "/usr/local/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in
> reraise
> raise value.with_traceback(tb)
> File
> "/usr/local/lib/python3.6/site-packages/airflow/www/templates/airflow/list_dags.html",
> line 22, in top-level template code
> \{% import 'admin/actions.html' as actionlib with context %}
> File
> "/usr/local/lib/python3.6/site-packages/airflow/www/templates/admin/master.html",
> line 18, in top-level template code
> \{% extends 'admin/base.html' %}
> File
> "/usr/local/lib/python3.6/site-packages/flask_admin/templates/bootstrap3/admin/base.html",
> line 30, in top-level template code
> \{% block page_body %}
> File
> "/usr/local/lib/python3.6/site-packages/airflow/www/templates/admin/master.html",
> line 107, in block "page_body"
> \{% block body %}
> File
> "/usr/local/lib/python3.6/site-packages/airflow/www/templates/airflow/list_dags.html",
> line 67, in block "body"
> \{% block model_list_table %}
> File
> "/usr/local/lib/python3.6/site-packages/airflow/www/templates/airflow/list_dags.html",
> line 115, in block "model_list_table"
> \{% block list_row scoped %}
> File
> "/usr/local/lib/python3.6/site-packages/airflow/www/templates/airflow/list_dags.html",
> line 143, in block "list_row"
> \{{ get_value(row, c) }}
> File "/usr/local/lib/python3.6/site-packages/flask_admin/model/base.py",
> line 1742, in get_list_value
> self.column_type_formatters,
> File "/usr/local/lib/python3.6/site-packages/flask_admin/model/base.py",
> line 1707, in _get_list_value
> value = column_fmt(self, context, model, name)
> File "/usr/local/lib/python3.6/site-packages/airflow/www/views.py", line
> 124, in dag_link
> execution_date=m.execution_date)
> AttributeError: 'DagModel' object has no attribute 'execution_date'
>
>
> Its fine to work with SQLlite, but not with mysql
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-3172) AttributeError: 'DagModel' object has no attribute 'execution_date'
Vinnson Lee created AIRFLOW-3172:
Summary: AttributeError: 'DagModel' object has no attribute
'execution_date'
Key: AIRFLOW-3172
URL: https://issues.apache.org/jira/browse/AIRFLOW-3172
Project: Apache Airflow
Issue Type: Bug
Environment: Docker Environment: python:3.6-stretch
Reporter: Vinnson Lee
Fix For: 1.10.0
2018-10-09 10:13:28,430] ERROR in app: Exception on /admin/dagmodel/ [GET]
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1982, in
wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1614, in
full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1517, in
handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 33, in
reraise
raise value
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1612, in
full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1598, in
dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.6/site-packages/flask_admin/base.py", line 69, in
inner
return self._run_view(f, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/flask_admin/base.py", line 368,
in _run_view
return fn(self, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/flask_admin/model/base.py", line
1900, in index_view
return_url=self._get_list_url(view_args),
File "/usr/local/lib/python3.6/site-packages/flask_admin/base.py", line 308,
in render
return render_template(template, **kwargs)
File "/usr/local/lib/python3.6/site-packages/flask/templating.py", line 134,
in render_template
context, ctx.app)
File "/usr/local/lib/python3.6/site-packages/flask/templating.py", line 116,
in _render
rv = template.render(context)
File "/usr/local/lib/python3.6/site-packages/jinja2/environment.py", line 989,
in render
return self.environment.handle_exception(exc_info, True)
File "/usr/local/lib/python3.6/site-packages/jinja2/environment.py", line 754,
in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in
reraise
raise value.with_traceback(tb)
File
"/usr/local/lib/python3.6/site-packages/airflow/www/templates/airflow/list_dags.html",
line 22, in top-level template code
\{% import 'admin/actions.html' as actionlib with context %}
File
"/usr/local/lib/python3.6/site-packages/airflow/www/templates/admin/master.html",
line 18, in top-level template code
\{% extends 'admin/base.html' %}
File
"/usr/local/lib/python3.6/site-packages/flask_admin/templates/bootstrap3/admin/base.html",
line 30, in top-level template code
\{% block page_body %}
File
"/usr/local/lib/python3.6/site-packages/airflow/www/templates/admin/master.html",
line 107, in block "page_body"
\{% block body %}
File
"/usr/local/lib/python3.6/site-packages/airflow/www/templates/airflow/list_dags.html",
line 67, in block "body"
\{% block model_list_table %}
File
"/usr/local/lib/python3.6/site-packages/airflow/www/templates/airflow/list_dags.html",
line 115, in block "model_list_table"
\{% block list_row scoped %}
File
"/usr/local/lib/python3.6/site-packages/airflow/www/templates/airflow/list_dags.html",
line 143, in block "list_row"
\{{ get_value(row, c) }}
File "/usr/local/lib/python3.6/site-packages/flask_admin/model/base.py", line
1742, in get_list_value
self.column_type_formatters,
File "/usr/local/lib/python3.6/site-packages/flask_admin/model/base.py", line
1707, in _get_list_value
value = column_fmt(self, context, model, name)
File "/usr/local/lib/python3.6/site-packages/airflow/www/views.py", line 124,
in dag_link
execution_date=m.execution_date)
AttributeError: 'DagModel' object has no attribute 'execution_date'
Its fine to work with SQLlite, but not with mysql
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] msumit closed pull request #4021: [AIRFLOW-XXX] - Added CarLabs to users
msumit closed pull request #4021: [AIRFLOW-XXX] - Added CarLabs to users URL: https://github.com/apache/incubator-airflow/pull/4021 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/README.md b/README.md index edeb46f6c5..d41ba25698 100644 --- a/README.md +++ b/README.md @@ -123,6 +123,7 @@ Currently **officially** using Airflow: 1. [Branch](https://branch.io) [[@sdebarshi](https://github.com/sdebarshi), [@dmitrig01](https://github.com/dmitrig01)] 1. [California Data Collaborative](https://github.com/California-Data-Collaborative) powered by [ARGO Labs](http://www.argolabs.org) 1. [Carbonite](https://www.carbonite.com) [[@ajbosco](https://github.com/ajbosco)] +1. [CarLabs](https://www.carlabs.ai/) [[@sganz](https://github.com/sganz) & [@odannyc](https://github.com/odannyc)] 1. [CAVA](https://www.cava.com) [[@minh5](http://github.com/minh5)] 1. [Celect](http://www.celect.com) [[@superdosh](https://github.com/superdosh) & [@chadcelect](https://github.com/chadcelect)] 1. [Censys](https://censys.io) [[@zakird](https://github.com/zakird), [@dadrian](https://github.com/dadrian), & [@andrewsardone](https://github.com/andrewsardone)] This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (AIRFLOW-3156) AWS Batch executor
[ https://issues.apache.org/jira/browse/AIRFLOW-3156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642949#comment-16642949 ] Jacopo Sabbatini edited comment on AIRFLOW-3156 at 10/9/18 8:24 AM: Well in a sense it's like the use case for the Mesos and Kubernetes executor. Correct me if wrong but when using the AWS Batch operator all the task does is continuously polling for the task status until done. This seems a very inefficient use of a worker. Furthermore if I wanted to submit many tasks using the Batch operator I suspect I would start seeing the polling API requests being throttled by AWS. Having a Batch executor it would mean that the polling of the job status can be batched using the DescribeJobs Batch API this would make having many tasks more efficient. Using Batch as executor would also simplify deployments since it remove the needs for workers that are now managed by AWS Batch. was (Author: dedalusj): Well in a sense it's like the use case for the Mesos and Kubernetes executor. Correct me if wrong but when using the AWS Batch operator all the task does is continuously polling for the task status until done. This seems a very inefficient use of a worker. Furthermore if I wanted to submit many tasks using the Batch operator I suspect I would start seeing the polling API requests being throttled by AWS. Having a Batch executor it would mean that the polling of the job status can be batched using the DescribeJobs Batch API this would make having many tasks more efficient. > AWS Batch executor > -- > > Key: AIRFLOW-3156 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3156 > Project: Apache Airflow > Issue Type: New Feature > Components: executor >Reporter: Jacopo Sabbatini >Priority: Major > Labels: aws > > Add support for AWS Batch as executor. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3156) AWS Batch executor
[ https://issues.apache.org/jira/browse/AIRFLOW-3156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642949#comment-16642949 ] Jacopo Sabbatini commented on AIRFLOW-3156: --- Well in a sense it's like the use case for the Mesos and Kubernetes executor. Correct me if wrong but when using the AWS Batch operator all the task does is continuously polling for the task status until done. This seems a very inefficient use of a worker. Furthermore if I wanted to submit many tasks using the Batch operator I suspect I would start seeing the polling API requests being throttled by AWS. Having a Batch executor it would mean that the polling of the job status can be batched using the DescribeJobs Batch API this would make having many tasks more efficient. > AWS Batch executor > -- > > Key: AIRFLOW-3156 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3156 > Project: Apache Airflow > Issue Type: New Feature > Components: executor >Reporter: Jacopo Sabbatini >Priority: Major > Labels: aws > > Add support for AWS Batch as executor. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] codecov-io edited a comment on issue #3684: [AIRFLOW-2840] - add update connections cli option
codecov-io edited a comment on issue #3684: [AIRFLOW-2840] - add update connections cli option URL: https://github.com/apache/incubator-airflow/pull/3684#issuecomment-409962956 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3684?src=pr=h1) Report > Merging [#3684](https://codecov.io/gh/apache/incubator-airflow/pull/3684?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/b8be322d3badfeadfa8f08e0bf92a12a6cd26418?src=pr=desc) will **increase** coverage by `1.71%`. > The diff coverage is `67.39%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3684?src=pr=tree) ```diff @@Coverage Diff@@ ## master #3684 +/- ## = + Coverage 75.79% 77.5% +1.71% = Files 199 205 +6 Lines 15946 15817 -129 = + Hits12086 12259 +173 + Misses 38603558 -302 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3684?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/bin/cli.py](https://codecov.io/gh/apache/incubator-airflow/pull/3684/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `64.52% <67.39%> (+0.03%)` | :arrow_up: | | [airflow/sensors/s3\_key\_sensor.py](https://codecov.io/gh/apache/incubator-airflow/pull/3684/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL3MzX2tleV9zZW5zb3IucHk=) | `31.03% <0%> (-68.97%)` | :arrow_down: | | [airflow/sensors/s3\_prefix\_sensor.py](https://codecov.io/gh/apache/incubator-airflow/pull/3684/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL3MzX3ByZWZpeF9zZW5zb3IucHk=) | `41.17% <0%> (-58.83%)` | :arrow_down: | | [airflow/utils/helpers.py](https://codecov.io/gh/apache/incubator-airflow/pull/3684/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9oZWxwZXJzLnB5) | `67.07% <0%> (-17.31%)` | :arrow_down: | | [airflow/hooks/mysql\_hook.py](https://codecov.io/gh/apache/incubator-airflow/pull/3684/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9teXNxbF9ob29rLnB5) | `78% <0%> (-12%)` | :arrow_down: | | [airflow/sensors/sql\_sensor.py](https://codecov.io/gh/apache/incubator-airflow/pull/3684/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL3NxbF9zZW5zb3IucHk=) | `90.47% <0%> (-9.53%)` | :arrow_down: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/incubator-airflow/pull/3684/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `73.91% <0%> (-7.52%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/incubator-airflow/pull/3684/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `83.95% <0%> (-5.31%)` | :arrow_down: | | [airflow/utils/state.py](https://codecov.io/gh/apache/incubator-airflow/pull/3684/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zdGF0ZS5weQ==) | `93.33% <0%> (-3.34%)` | :arrow_down: | | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/3684/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `88.58% <0%> (-3.13%)` | :arrow_down: | | ... and [75 more](https://codecov.io/gh/apache/incubator-airflow/pull/3684/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3684?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3684?src=pr=footer). Last update [b8be322...c5fa839](https://codecov.io/gh/apache/incubator-airflow/pull/3684?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] vardancse commented on a change in pull request #3994: [AIRFLOW-3136] Add retry_number to TaskInstance Key property to avoid race condition
vardancse commented on a change in pull request #3994: [AIRFLOW-3136] Add retry_number to TaskInstance Key property to avoid race condition URL: https://github.com/apache/incubator-airflow/pull/3994#discussion_r223579061 ## File path: airflow/models.py ## @@ -1230,7 +1230,7 @@ def key(self): """ Returns a tuple that identifies the task instance uniquely """ -return self.dag_id, self.task_id, self.execution_date +return self.dag_id, self.task_id, self.execution_date, self.try_number Review comment: This collision situation is arising because of the following steps happening periodically by scheduler. 1. In the _execute_helper method, heartbeat method is being called, which calls execute_async in further and putting key(dag_id, task_id, execution_date) and command to executor queue and then from there onwards based on executor type, task starts executing asynchronously. 2. Sync methods runs followed by execute_async method which push keys from result_queue to event_buffer 3. _process_executor_events method being called followed by hearbeat method which detects external killing of task, if TI.state is queued and event_buffer key status is success/failed Another suggestion for change could be taking out execute_async method from heartbeat method and call it after _process_executor_events method call in _execute_helper method in jobs.py This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
