[jira] [Commented] (AIRFLOW-2886) Secure Flask SECRET_KEY
[
https://issues.apache.org/jira/browse/AIRFLOW-2886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16580293#comment-16580293
]
ASF subversion and git services commented on AIRFLOW-2886:
--
Commit f7602f8266559e55bc602a9639e3e1ab640f30e8 in incubator-airflow's branch
refs/heads/master from Xiaodong
[ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=f7602f8 ]
[AIRFLOW-2886] Secure Flask SECRET_KEY (#3738)
The Flask SECRET_KEY should be as random as possible.
On the other hand, we can nott genrate random value when
we launch the webserver (the secret_key will be
inconsistent across the workers).
We can generate a random one in the configuration file
airflow.cfg, just like how we deal with FERNET_KEY.
The SECRET_KEY is generated using os.urandom, as
recommended by Flask community.
> Secure Flask SECRET_KEY
> ---
>
> Key: AIRFLOW-2886
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2886
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Critical
>
> In my earlier PRs, [https://github.com/apache/incubator-airflow/pull/3651]
> and [https://github.com/apache/incubator-airflow/pull/3729] , I proposed to
> generate random SECRET_KEY for Flask App.
> If we have multiple workers for the Flask webserver, we may encounter CSRF
> error {{The CSRF session token is missing}} .
> On the other hand, it's still very important to have as random SECRET_KEY as
> possible for security reasons. We can deal with it like how we dealt with
> FERNET_KEY (i.e. generate a random value when the airflow.cfg file is
> initiated).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2860) DruidHook: time variable is not updated correctly when checking for timeout
[ https://issues.apache.org/jira/browse/AIRFLOW-2860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578597#comment-16578597 ] ASF subversion and git services commented on AIRFLOW-2860: -- Commit 9d68fa337586a6a64b6a9f19fc8f2b079376a4db in incubator-airflow's branch refs/heads/master from awelsh93 [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=9d68fa3 ] [AIRFLOW-2860] DruidHook: time check is wrong (#3745) > DruidHook: time variable is not updated correctly when checking for timeout > > > Key: AIRFLOW-2860 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2860 > Project: Apache Airflow > Issue Type: Bug > Components: hooks >Reporter: Adam Welsh >Assignee: Adam Welsh >Priority: Trivial > Fix For: 2.0.0 > > > The variable that is used in the condition to check if the Druid ingestion > task has exceeded the max_ingestion_time is not updated correctly. It gets > incremented by one (which is the default value for timeout) but should be > incremented by timeout. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2524) Airflow integration with AWS Sagemaker
[ https://issues.apache.org/jira/browse/AIRFLOW-2524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577696#comment-16577696 ] ASF subversion and git services commented on AIRFLOW-2524: -- Commit 4d2f83b19af3489d6c9563d51210a3dab2f38b26 in incubator-airflow's branch refs/heads/master from Keliang Chen [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=4d2f83b ] [AIRFLOW-2524] Add Amazon SageMaker Training (#3658) Add SageMaker Hook, Training Operator & Sensor Co-authored-by: srrajeev-aws > Airflow integration with AWS Sagemaker > -- > > Key: AIRFLOW-2524 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2524 > Project: Apache Airflow > Issue Type: Improvement > Components: aws, contrib >Reporter: Rajeev Srinivasan >Assignee: Yang Yu >Priority: Major > Labels: AWS > Time Spent: 10m > Remaining Estimate: 0h > > Would it be possible to orchestrate an end to end AWS Sagemaker job using > Airflow. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2889) Fix typos detected by github.com/client9/misspell
[ https://issues.apache.org/jira/browse/AIRFLOW-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577407#comment-16577407 ] ASF subversion and git services commented on AIRFLOW-2889: -- Commit b78c7fb8512f7a40f58b46530e9b3d5562fe84ea in incubator-airflow's branch refs/heads/master from [~seratch] [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=b78c7fb ] [AIRFLOW-2889] Fix typos detected by github.com/client9/misspell (#3732) > Fix typos detected by github.com/client9/misspell > - > > Key: AIRFLOW-2889 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2889 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Kazuhiro Sera >Priority: Minor > > Fixing typos is sometimes very hard. It's not so easy to visually review > them. Recently, I discovered a very useful tool for it, > [misspell](https://github.com/client9/misspell). > This pull request fixes minor typos detected by > [misspell](https://github.com/client9/misspell) except for the false > positives. If you would like me to work on other files as well, let me know. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2878) "/www_rbac" Layout/UI Display Issue
[
https://issues.apache.org/jira/browse/AIRFLOW-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576596#comment-16576596
]
ASF subversion and git services commented on AIRFLOW-2878:
--
Commit f999ce22c16b11e8b9205d1bbb6af0683a0c7a3d in incubator-airflow's branch
refs/heads/master from [~TaoFeng]
[ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=f999ce2 ]
Revert "[AIRFLOW-2878] Fix www_rbac display issue" (#3737)
This reverts commit 1f57dafd70ac41da8295f20793f8913b7f5a5dff.
> "/www_rbac" Layout/UI Display Issue
> ---
>
> Key: AIRFLOW-2878
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2878
> Project: Apache Airflow
> Issue Type: Improvement
> Components: ui
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Critical
>
> The new RBAC UI has some issues about layout/UI display.
> The header () is not shown ("hidden" by the Nav Bar), or tables are not
> shown completely.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1874) Support standard SQL in Check, ValueCheck and IntervalCheck BigQuery operators
[ https://issues.apache.org/jira/browse/AIRFLOW-1874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576092#comment-16576092 ] ASF subversion and git services commented on AIRFLOW-1874: -- Commit 9952b23ba08fdc57c866e0f35f20a7a1a13ede74 in incubator-airflow's branch refs/heads/master from Yuliya Volkova [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=9952b23 ] [AIRFLOW-1874] use_legacy_sql added to BigQueryCheck operators (#3717) > Support standard SQL in Check, ValueCheck and IntervalCheck BigQuery operators > -- > > Key: AIRFLOW-1874 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1874 > Project: Apache Airflow > Issue Type: Bug > Components: contrib, gcp, operators >Reporter: Guillermo Rodríguez Cano >Assignee: Iuliia Volkova >Priority: Major > Fix For: 2.0.0 > > > BigQueryCheckOperator, BigQueryValueCheckOperator and > BigQueryIntervalCheckOperator do not support disabling use of default legacy > SQL in BigQuery. > This is a major blocker to support correct migration to standard SQL when > queries are complicated. For example, a query that can be queried in legacy > SQL may be blocked from any subsequent view done in standard SQL that this > view uses as the queries are bound to either standard or legacy SQL but not a > mix. > These operators inherit from base ones of the same name (without the BigQuery > prefix) from Airflow which may make the process more complicated as the flag > to use standard SQL should be enabled because the underlying BigQueryHook has > the corresponding parameter, use_legacy_sql, set to True, when running a > query. But it is not possible to pass parameters all the way to it via the > aforementioned operators. > The workaround of including #standardSQL and a new line before the query > doesn't work either as there is mismatch. BigQuery reports the following in > fact: "Query text specifies use_legacy_sql:false, while API options > specify:true" > A workaround for queries on views using standard SQL is to persist the result > of the query in a temporary table, then run the check operation and > thereafter delete the temporary table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2763) No precheck mechanism in place during worker initialisation for the connection to metadata database
[ https://issues.apache.org/jira/browse/AIRFLOW-2763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575618#comment-16575618 ] ASF subversion and git services commented on AIRFLOW-2763: -- Commit c7551f680a52b8b0242aa18872758693bda72955 in incubator-airflow's branch refs/heads/master from [~aimohan] [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=c7551f6 ] [AIRFLOW-2763] Add check to validate worker connectivity to metadata Database > No precheck mechanism in place during worker initialisation for the > connection to metadata database > --- > > Key: AIRFLOW-2763 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2763 > Project: Apache Airflow > Issue Type: Improvement > Components: db, worker >Reporter: Aishwarya Mohan >Assignee: Aishwarya Mohan >Priority: Major > > Currently when airflow worker is started, no initialization check is in place > to see if the connection to metadata database is healthy or not. The worker > threads remain alive(after throwing exceptions - OperationalError) leading to > a misconception that the worker is up and running the tasks allocated. > In the case of worker interaction with the broker, a reconnection mechanism > exists the exits the worker due to timeout operational error if it failed to > establish connection within the configured number of > attempts(connection_max_retries). > An exit/shutdown mechanism needs to be included during worker initialization > incase the connection to metadata DB is not established. This will assure > the airflow users that the connection to database is in place before they > proceed with task scheduling. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2878) "/www_rbac" Layout/UI Display Issue
[
https://issues.apache.org/jira/browse/AIRFLOW-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575501#comment-16575501
]
ASF subversion and git services commented on AIRFLOW-2878:
--
Commit 1f57dafd70ac41da8295f20793f8913b7f5a5dff in incubator-airflow's branch
refs/heads/master from Xiaodong
[ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=1f57daf ]
[AIRFLOW-2878] Fix www_rbac display issue
The new RBAC UI has some issues about layout/UI display.
The header () is not shown ("hidden" by the Nav Bar),
or tables are not shown completely.
This is addressed by a simple change on
templates/appbuilder/baselayout.html
> "/www_rbac" Layout/UI Display Issue
> ---
>
> Key: AIRFLOW-2878
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2878
> Project: Apache Airflow
> Issue Type: Improvement
> Components: ui
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Critical
>
> The new RBAC UI has some issues about layout/UI display.
> The header () is not shown ("hidden" by the Nav Bar), or tables are not
> shown completely.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2856) Update Docker Env Setup Docs to Account for verify_gpl_dependency()
[ https://issues.apache.org/jira/browse/AIRFLOW-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573627#comment-16573627 ] ASF subversion and git services commented on AIRFLOW-2856: -- Commit 55200b48bbc0bf86db72c2b7eab01109bb98102f in incubator-airflow's branch refs/heads/v1-10-stable from [~ajc] [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=55200b4 ] [AIRFLOW-2856] Pass in SLUGIFY_USES_TEXT_UNIDECODE=yes ENV to docker run (#3701) (cherry picked from commit 8687ab9271b7b93473584a720f225f20fa9a7aa4) Signed-off-by: Bolke de Bruin (cherry picked from commit 3670d49a4c5da4769be1df5b4f8ef3504b74a184) Signed-off-by: Bolke de Bruin > Update Docker Env Setup Docs to Account for verify_gpl_dependency() > --- > > Key: AIRFLOW-2856 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2856 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Andy Cooper >Assignee: Andy Cooper >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2869) Remove smart quote from default config
[ https://issues.apache.org/jira/browse/AIRFLOW-2869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573623#comment-16573623 ] ASF subversion and git services commented on AIRFLOW-2869: -- Commit 26f6faed690846c963f90fe06458b3711d841e25 in incubator-airflow's branch refs/heads/v1-10-stable from [~wdhorton] [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=26f6fae ] [AIRFLOW-2869] Remove smart quote from default config Closes #3716 from wdhorton/remove-smart-quote- from-cfg (cherry picked from commit 67e2bb96cdc5ea37226d11332362d3bd3778cea0) Signed-off-by: Bolke de Bruin (cherry picked from commit 700f5f088dbead866170c9a3fe7e021e86ab30bb) Signed-off-by: Bolke de Bruin > Remove smart quote from default config > -- > > Key: AIRFLOW-2869 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2869 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Siddharth Anand >Assignee: Siddharth Anand >Priority: Trivial > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2859) DateTimes returned from the database are not converted to UTC
[ https://issues.apache.org/jira/browse/AIRFLOW-2859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573625#comment-16573625 ] ASF subversion and git services commented on AIRFLOW-2859: -- Commit a6d5ee9ce57bcdd22dde3fdf00e3b1cd26274ede in incubator-airflow's branch refs/heads/v1-10-stable from bolkedebruin [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=a6d5ee9 ] [AIRFLOW-2859] Implement own UtcDateTime (#3708) The different UtcDateTime implementations all have issues. Either they replace tzinfo directly without converting or they do not convert to UTC at all. We also ensure all mysql connections are in UTC in order to keep sanity, as mysql will ignore the timezone of a field when inserting/updating. (cherry picked from commit 6fd4e6055e36e9867923b0b402363fcd8c30e297) Signed-off-by: Bolke de Bruin (cherry picked from commit 8fc8c7ae5483c002f5264b087b26a20fd8ae7b67) Signed-off-by: Bolke de Bruin > DateTimes returned from the database are not converted to UTC > - > > Key: AIRFLOW-2859 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2859 > Project: Apache Airflow > Issue Type: Bug > Components: database >Reporter: Bolke de Bruin >Priority: Blocker > Fix For: 1.10.0 > > > This is due to the fact that sqlalchemy-utcdatetime does not convert to UTC > when the database returns datetimes with tzinfo. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2856) Update Docker Env Setup Docs to Account for verify_gpl_dependency()
[ https://issues.apache.org/jira/browse/AIRFLOW-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573619#comment-16573619 ] ASF subversion and git services commented on AIRFLOW-2856: -- Commit 3670d49a4c5da4769be1df5b4f8ef3504b74a184 in incubator-airflow's branch refs/heads/v1-10-test from [~ajc] [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=3670d49 ] [AIRFLOW-2856] Pass in SLUGIFY_USES_TEXT_UNIDECODE=yes ENV to docker run (#3701) (cherry picked from commit 8687ab9271b7b93473584a720f225f20fa9a7aa4) Signed-off-by: Bolke de Bruin > Update Docker Env Setup Docs to Account for verify_gpl_dependency() > --- > > Key: AIRFLOW-2856 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2856 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Andy Cooper >Assignee: Andy Cooper >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2870) Migrations fail when upgrading from below cc1e65623dc7_add_max_tries_column_to_task_instance
[
https://issues.apache.org/jira/browse/AIRFLOW-2870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573618#comment-16573618
]
ASF subversion and git services commented on AIRFLOW-2870:
--
Commit 95aa49a71dcc69d2e9a8e32b69a2a61cacec2b1b in incubator-airflow's branch
refs/heads/v1-10-test from bolkedebruin
[ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=95aa49a ]
[AIRFLOW-2870] Use abstract TaskInstance for migration (#3720)
If we use the full model for migration it can have columns
added that are not available yet in the database. Using
an abstraction ensures only the columns that are required
for data migration are present.
(cherry picked from commit 546f1cdb5208ba8e1cf3bde36bbdbb639fa20b22)
Signed-off-by: Bolke de Bruin
> Migrations fail when upgrading from below
> cc1e65623dc7_add_max_tries_column_to_task_instance
>
>
> Key: AIRFLOW-2870
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2870
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: George Leslie-Waksman
>Priority: Blocker
>
> Running migrations from below
> cc1e65623dc7_add_max_tries_column_to_task_instance.py fail with:
> {noformat}
> INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
> INFO [alembic.runtime.migration] Will assume transactional DDL.
> INFO [alembic.runtime.migration] Running upgrade 127d2bf2dfa7 ->
> cc1e65623dc7, add max tries column to task instance
> Traceback (most recent call last):
> File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py",
> line 1182, in _execute_context
> context)
> File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py",
> line 470, in do_execute
> cursor.execute(statement, parameters)
> psycopg2.ProgrammingError: column task_instance.executor_config does not exist
> LINE 1: ...ued_dttm, task_instance.pid AS task_instance_pid, task_insta...
> {noformat}
> The failure is occurring because
> cc1e65623dc7_add_max_tries_column_to_task_instance.py imports TaskInstance
> from the current code version, which has changes to the task_instance table
> that are not expected by the migration.
> Specifically, 27c6a30d7c24_add_executor_config_to_task_instance.py adds an
> executor_config column that does not exist as of when
> cc1e65623dc7_add_max_tries_column_to_task_instance.py is run.
> It is worth noting that this will not be observed for new installs because
> the migration branches on table existence/non-existence at a point that will
> hide the issue from new installs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2861) Need index on log table
[ https://issues.apache.org/jira/browse/AIRFLOW-2861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573531#comment-16573531 ] ASF subversion and git services commented on AIRFLOW-2861: -- Commit 6f7fe74b9ff6c2abae2764988c152af8bcc8e199 in incubator-airflow's branch refs/heads/master from [~vardan] [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=6f7fe74 ] [AIRFLOW-2861] Add index on log table (#3709) > Need index on log table > --- > > Key: AIRFLOW-2861 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2861 > Project: Apache Airflow > Issue Type: Improvement > Components: database >Affects Versions: 1.10.0 >Reporter: Vardan Gupta >Assignee: Vardan Gupta >Priority: Major > > Delete dag functionality is added in v1-10-stable, whose implementation > during the metadata cleanup > [part|https://github.com/apache/incubator-airflow/blob/dc78b9196723ca6724185231ccd6f5bbe8edcaf3/airflow/api/common/experimental/delete_dag.py#L48], > look for classes which has attribute named as dag_id and then formulate the > query on matching model and then delete from metadata, we've few numbers > where we've observed slowness especially in log table because it doesn't have > any single or multiple-column index. Creating an index would boost the > performance though insertion will be a bit slower. Since deletion will be a > sync call, would be good idea to create index. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2870) Migrations fail when upgrading from below cc1e65623dc7_add_max_tries_column_to_task_instance
[
https://issues.apache.org/jira/browse/AIRFLOW-2870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573528#comment-16573528
]
ASF subversion and git services commented on AIRFLOW-2870:
--
Commit 546f1cdb5208ba8e1cf3bde36bbdbb639fa20b22 in incubator-airflow's branch
refs/heads/master from bolkedebruin
[ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=546f1cd ]
[AIRFLOW-2870] Use abstract TaskInstance for migration (#3720)
If we use the full model for migration it can have columns
added that are not available yet in the database. Using
an abstraction ensures only the columns that are required
for data migration are present.
> Migrations fail when upgrading from below
> cc1e65623dc7_add_max_tries_column_to_task_instance
>
>
> Key: AIRFLOW-2870
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2870
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: George Leslie-Waksman
>Priority: Blocker
>
> Running migrations from below
> cc1e65623dc7_add_max_tries_column_to_task_instance.py fail with:
> {noformat}
> INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
> INFO [alembic.runtime.migration] Will assume transactional DDL.
> INFO [alembic.runtime.migration] Running upgrade 127d2bf2dfa7 ->
> cc1e65623dc7, add max tries column to task instance
> Traceback (most recent call last):
> File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py",
> line 1182, in _execute_context
> context)
> File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py",
> line 470, in do_execute
> cursor.execute(statement, parameters)
> psycopg2.ProgrammingError: column task_instance.executor_config does not exist
> LINE 1: ...ued_dttm, task_instance.pid AS task_instance_pid, task_insta...
> {noformat}
> The failure is occurring because
> cc1e65623dc7_add_max_tries_column_to_task_instance.py imports TaskInstance
> from the current code version, which has changes to the task_instance table
> that are not expected by the migration.
> Specifically, 27c6a30d7c24_add_executor_config_to_task_instance.py adds an
> executor_config column that does not exist as of when
> cc1e65623dc7_add_max_tries_column_to_task_instance.py is run.
> It is worth noting that this will not be observed for new installs because
> the migration branches on table existence/non-existence at a point that will
> hide the issue from new installs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2856) Update Docker Env Setup Docs to Account for verify_gpl_dependency()
[ https://issues.apache.org/jira/browse/AIRFLOW-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573089#comment-16573089 ] ASF subversion and git services commented on AIRFLOW-2856: -- Commit 8687ab9271b7b93473584a720f225f20fa9a7aa4 in incubator-airflow's branch refs/heads/master from [~ajc] [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=8687ab9 ] [AIRFLOW-2856] Pass in SLUGIFY_USES_TEXT_UNIDECODE=yes ENV to docker run (#3701) > Update Docker Env Setup Docs to Account for verify_gpl_dependency() > --- > > Key: AIRFLOW-2856 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2856 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Andy Cooper >Assignee: Andy Cooper >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2874) Enable Flask App Builder theme support
[ https://issues.apache.org/jira/browse/AIRFLOW-2874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573086#comment-16573086 ] ASF subversion and git services commented on AIRFLOW-2874: -- Commit cbe7955d7e0dd352829f71e762026f4044a05a7d in incubator-airflow's branch refs/heads/master from [~verdan] [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=cbe7955 ] [AIRFLOW-2874] Enables FAB's theme support (#3719) > Enable Flask App Builder theme support > -- > > Key: AIRFLOW-2874 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2874 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Verdan Mahmood >Assignee: Verdan Mahmood >Priority: Major > > To customize the look and feel of Apache Airflow (an effort towards making > Airflow a whitelabel application), we should enable the support of FAB's > theme, which can be set in configuration. > Theme can be use in conjunction of existing `navbar_color` configuration or > can be used separately by simple unsetting the navbar_color config. > > http://flask-appbuilder.readthedocs.io/en/latest/customizing.html#changing-themes -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2859) DateTimes returned from the database are not converted to UTC
[ https://issues.apache.org/jira/browse/AIRFLOW-2859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572863#comment-16572863 ] ASF subversion and git services commented on AIRFLOW-2859: -- Commit 8fc8c7ae5483c002f5264b087b26a20fd8ae7b67 in incubator-airflow's branch refs/heads/v1-10-test from bolkedebruin [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=8fc8c7a ] [AIRFLOW-2859] Implement own UtcDateTime (#3708) The different UtcDateTime implementations all have issues. Either they replace tzinfo directly without converting or they do not convert to UTC at all. We also ensure all mysql connections are in UTC in order to keep sanity, as mysql will ignore the timezone of a field when inserting/updating. (cherry picked from commit 6fd4e6055e36e9867923b0b402363fcd8c30e297) Signed-off-by: Bolke de Bruin > DateTimes returned from the database are not converted to UTC > - > > Key: AIRFLOW-2859 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2859 > Project: Apache Airflow > Issue Type: Bug > Components: database >Reporter: Bolke de Bruin >Priority: Blocker > Fix For: 1.10.0 > > > This is due to the fact that sqlalchemy-utcdatetime does not convert to UTC > when the database returns datetimes with tzinfo. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2140) Add Kubernetes Scheduler to Spark Submit Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572855#comment-16572855 ] ASF subversion and git services commented on AIRFLOW-2140: -- Commit f58246d2ef265eb762c179a12c40e011ce62cea1 in incubator-airflow's branch refs/heads/v1-10-test from [~ashb] [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=f58246d ] [AIRFLOW-2140] Don't require kubernetes for the SparkSubmit hook (#3700) This extra dep is a quasi-breaking change when upgrading - previously there were no deps outside of Airflow itself for this hook. Importing the k8s libs breaks installs that aren't also using Kubernetes. This makes the dep optional for anyone who doesn't explicitly use the functionality (cherry picked from commit 0be002eebb182b607109a0390d7f6fb8795c668b) Signed-off-by: Bolke de Bruin > Add Kubernetes Scheduler to Spark Submit Operator > - > > Key: AIRFLOW-2140 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2140 > Project: Apache Airflow > Issue Type: New Feature >Affects Versions: 1.9.0 >Reporter: Rob Keevil >Assignee: Rob Keevil >Priority: Major > Fix For: 2.0.0 > > > Spark 2.3 adds the Kubernetes resource manager to Spark, alongside the > existing Standalone, Yarn and Mesos resource managers. > https://github.com/apache/spark/blob/master/docs/running-on-kubernetes.md > We should extend the spark submit operator to enable the new K8s spark submit > options, and to be able to monitor Spark jobs running within Kubernetes. > I already have working code for this, I need to test the monitoring/log > parsing code and make sure that Airflow is able to terminate Kubernetes pods > when jobs are cancelled etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2869) Remove smart quote from default config
[ https://issues.apache.org/jira/browse/AIRFLOW-2869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572854#comment-16572854 ] ASF subversion and git services commented on AIRFLOW-2869: -- Commit 700f5f088dbead866170c9a3fe7e021e86ab30bb in incubator-airflow's branch refs/heads/v1-10-test from William Horton [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=700f5f0 ] [AIRFLOW-2869] Remove smart quote from default config Closes #3716 from wdhorton/remove-smart-quote- from-cfg (cherry picked from commit 67e2bb96cdc5ea37226d11332362d3bd3778cea0) Signed-off-by: Bolke de Bruin > Remove smart quote from default config > -- > > Key: AIRFLOW-2869 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2869 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Siddharth Anand >Assignee: Siddharth Anand >Priority: Trivial > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2826) Add hook for Google Cloud KMS
[ https://issues.apache.org/jira/browse/AIRFLOW-2826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572848#comment-16572848 ] ASF subversion and git services commented on AIRFLOW-2826: -- Commit acca61c602e341da06ebee2eca3a26f4e7400238 in incubator-airflow's branch refs/heads/master from [~jakahn] [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=acca61c ] [AIRFLOW-2826] Add GoogleCloudKMSHook (#3677) Adds a hook enabling encryption and decryption through Google Cloud KMS. This should also contribute to AIRFLOW-2062. > Add hook for Google Cloud KMS > - > > Key: AIRFLOW-2826 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2826 > Project: Apache Airflow > Issue Type: Improvement > Components: hooks >Reporter: Jasper Kahn >Assignee: Jasper Kahn >Priority: Minor > Labels: features > > Add a hook to support interacting with Google Cloud KMS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2062) Support fine-grained Connection encryption
[ https://issues.apache.org/jira/browse/AIRFLOW-2062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572849#comment-16572849 ] ASF subversion and git services commented on AIRFLOW-2062: -- Commit acca61c602e341da06ebee2eca3a26f4e7400238 in incubator-airflow's branch refs/heads/master from [~jakahn] [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=acca61c ] [AIRFLOW-2826] Add GoogleCloudKMSHook (#3677) Adds a hook enabling encryption and decryption through Google Cloud KMS. This should also contribute to AIRFLOW-2062. > Support fine-grained Connection encryption > -- > > Key: AIRFLOW-2062 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2062 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib >Reporter: Wilson Lian >Priority: Minor > > This effort targets containerized tasks (e.g., those launched by > KubernetesExecutor). Under that paradigm, each task could potentially operate > under different credentials, and fine-grained Connection encryption will > enable an administrator to restrict which connections can be accessed by > which tasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2859) DateTimes returned from the database are not converted to UTC
[ https://issues.apache.org/jira/browse/AIRFLOW-2859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572716#comment-16572716 ] ASF subversion and git services commented on AIRFLOW-2859: -- Commit 6fd4e6055e36e9867923b0b402363fcd8c30e297 in incubator-airflow's branch refs/heads/master from bolkedebruin [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=6fd4e60 ] [AIRFLOW-2859] Implement own UtcDateTime (#3708) The different UtcDateTime implementations all have issues. Either they replace tzinfo directly without converting or they do not convert to UTC at all. We also ensure all mysql connections are in UTC in order to keep sanity, as mysql will ignore the timezone of a field when inserting/updating. > DateTimes returned from the database are not converted to UTC > - > > Key: AIRFLOW-2859 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2859 > Project: Apache Airflow > Issue Type: Bug > Components: database >Reporter: Bolke de Bruin >Priority: Blocker > Fix For: 1.10.0 > > > This is due to the fact that sqlalchemy-utcdatetime does not convert to UTC > when the database returns datetimes with tzinfo. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2855) Need to Check Validity of Cron Expression When Process DAG File/Zip File
[ https://issues.apache.org/jira/browse/AIRFLOW-2855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572528#comment-16572528 ] ASF subversion and git services commented on AIRFLOW-2855: -- Commit d47580feaf80eeebb416d0179dfa8db3f4e1d2c9 in incubator-airflow's branch refs/heads/master from Xiaodong [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=d47580f ] [AIRFLOW-2855] Check Cron Expression Validity in DagBag.process_file() (#3698) A DAG can be imported as a .py script properly, but the Cron expression inside as "schedule_interval" may be invalid, like "0 100 * * *". This commit helps check the validity of Cron expression in DAG files (.py) and packaged DAG files (.zip), and help show exception messages in web UI by add these exceptions into metadata "import_error". > Need to Check Validity of Cron Expression When Process DAG File/Zip File > > > Key: AIRFLOW-2855 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2855 > Project: Apache Airflow > Issue Type: Improvement > Components: DAG >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > > *schedule_interval* of DAGs can either be *timedelta* or a *Cron expression*. > When it's a Cron expression, there is no mechanism to check its validity at > this moment. If there is anything wrong with the Cron expression itself, it > will cause issues when methods _*following_schedule(**)*_ and > _*previous_schedule()*_ are invoked (will affect scheduling). However, > exceptions will only be written into logs. From Web UI, it’s hard for users > to identify this issue & the source while no new task can be initiated > (especially for users who’re not very familiar with Cron). > It may be good to show error messages in web UI when a DAG's Cron expression > (as schedule_interval) can not be parsed by *croniter* properly. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2869) Remove smart quote from default config
[ https://issues.apache.org/jira/browse/AIRFLOW-2869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572453#comment-16572453 ] ASF subversion and git services commented on AIRFLOW-2869: -- Commit 67e2bb96cdc5ea37226d11332362d3bd3778cea0 in incubator-airflow's branch refs/heads/master from William Horton [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=67e2bb9 ] [AIRFLOW-2869] Remove smart quote from default config Closes #3716 from wdhorton/remove-smart-quote- from-cfg > Remove smart quote from default config > -- > > Key: AIRFLOW-2869 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2869 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Siddharth Anand >Assignee: Siddharth Anand >Priority: Trivial > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2851) Canonicalize "as _..." etc imports
[ https://issues.apache.org/jira/browse/AIRFLOW-2851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572156#comment-16572156 ] ASF subversion and git services commented on AIRFLOW-2851: -- Commit 9131d6cc8fce515222da6e4ab9d86cce69f20d1e in incubator-airflow's branch refs/heads/master from Taylor D. Edmiston [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=9131d6c ] [AIRFLOW-2851] Canonicalize "as _..." etc imports (#3696) > Canonicalize "as _..." etc imports > -- > > Key: AIRFLOW-2851 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2851 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Taylor Edmiston >Assignee: Taylor Edmiston >Priority: Minor > > This PR: > 1. Replaces `import foo as _foo` style imports with the more common `import > foo` used everywhere else across the codebase. I dug through history and > couldn't find special reasons to maintain the as style imports here (I think > it's just old code). Currently (33dd33c89d4b6454d224ca34bab5ae37fb9812a6), > there are just a handful of import lines using `as _...` vs thousands not > using it, so the goal here is to improve consistency. > 2. It also simplifies `import foo.bar as bar` style imports to equivalent > `from foo import bar`. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2140) Add Kubernetes Scheduler to Spark Submit Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572144#comment-16572144 ] ASF subversion and git services commented on AIRFLOW-2140: -- Commit 0be002eebb182b607109a0390d7f6fb8795c668b in incubator-airflow's branch refs/heads/master from [~ashb] [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=0be002e ] [AIRFLOW-2140] Don't require kubernetes for the SparkSubmit hook (#3700) This extra dep is a quasi-breaking change when upgrading - previously there were no deps outside of Airflow itself for this hook. Importing the k8s libs breaks installs that aren't also using Kubernetes. This makes the dep optional for anyone who doesn't explicitly use the functionality > Add Kubernetes Scheduler to Spark Submit Operator > - > > Key: AIRFLOW-2140 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2140 > Project: Apache Airflow > Issue Type: New Feature >Affects Versions: 1.9.0 >Reporter: Rob Keevil >Assignee: Rob Keevil >Priority: Major > Fix For: 2.0.0 > > > Spark 2.3 adds the Kubernetes resource manager to Spark, alongside the > existing Standalone, Yarn and Mesos resource managers. > https://github.com/apache/spark/blob/master/docs/running-on-kubernetes.md > We should extend the spark submit operator to enable the new K8s spark submit > options, and to be able to monitor Spark jobs running within Kubernetes. > I already have working code for this, I need to test the monitoring/log > parsing code and make sure that Airflow is able to terminate Kubernetes pods > when jobs are cancelled etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2863) GKEClusterHook catches wrong exception
[ https://issues.apache.org/jira/browse/AIRFLOW-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571967#comment-16571967 ] ASF subversion and git services commented on AIRFLOW-2863: -- Commit 142a9425db3675fc433b0dbbb637f53638e70a8a in incubator-airflow's branch refs/heads/master from [~noremac201] [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=142a942 ] [AIRFLOW-2863] Fix GKEClusterHook catching wrong exception (#3711) > GKEClusterHook catches wrong exception > -- > > Key: AIRFLOW-2863 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2863 > Project: Apache Airflow > Issue Type: Bug >Reporter: Cameron Moberg >Assignee: Cameron Moberg >Priority: Minor > > Instead of successfully catching the error and reporting success, it reports > a failure, since it catches the wrong error. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2857) Broken builds for Read the Docs
[ https://issues.apache.org/jira/browse/AIRFLOW-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571791#comment-16571791 ] ASF subversion and git services commented on AIRFLOW-2857: -- Commit c010e4a0447da8eeb4364b8db471198739582944 in incubator-airflow's branch refs/heads/v1-10-test from Taylor D. Edmiston [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=c010e4a ] [AIRFLOW-2857] Fix Read the Docs env (#3703) The Read the Docs build process was broken due to #3660. This PR fixes this. (cherry picked from commit 8af0aa96bfe3caa51d67ab393db069d37b0c4169) Signed-off-by: Bolke de Bruin > Broken builds for Read the Docs > --- > > Key: AIRFLOW-2857 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2857 > Project: Apache Airflow > Issue Type: Bug > Components: ci, docs >Reporter: Taylor Edmiston >Assignee: Taylor Edmiston >Priority: Major > Fix For: 2.0.0 > > > Per the mailing list thread "Readthedocs is Broken" on Aug 5, 2018, the Read > the Docs build process is currently broken per the recent addition of > [https://github.com/apache/incubator-airflow/pull/3660|https://github.com/apache/incubator-airflow/pull/3660.]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2857) Broken builds for Read the Docs
[ https://issues.apache.org/jira/browse/AIRFLOW-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571792#comment-16571792 ] ASF subversion and git services commented on AIRFLOW-2857: -- Commit c764c1114a9831443ea8ce6af89a8dd10f346ded in incubator-airflow's branch refs/heads/v1-10-stable from Taylor D. Edmiston [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=c764c11 ] [AIRFLOW-2857] Fix Read the Docs env (#3703) The Read the Docs build process was broken due to #3660. This PR fixes this. (cherry picked from commit 8af0aa96bfe3caa51d67ab393db069d37b0c4169) Signed-off-by: Bolke de Bruin > Broken builds for Read the Docs > --- > > Key: AIRFLOW-2857 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2857 > Project: Apache Airflow > Issue Type: Bug > Components: ci, docs >Reporter: Taylor Edmiston >Assignee: Taylor Edmiston >Priority: Major > Fix For: 2.0.0 > > > Per the mailing list thread "Readthedocs is Broken" on Aug 5, 2018, the Read > the Docs build process is currently broken per the recent addition of > [https://github.com/apache/incubator-airflow/pull/3660|https://github.com/apache/incubator-airflow/pull/3660.]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2231) DAG with a relativedelta schedule_interval fails
[ https://issues.apache.org/jira/browse/AIRFLOW-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571782#comment-16571782 ] ASF subversion and git services commented on AIRFLOW-2231: -- Commit 8949aaca604f77ce8f24ff5255d18225a77eef5a in incubator-airflow's branch refs/heads/master from brookskd [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=8949aac ] [AIRFLOW-2231] Fix relativedelta DAG schedule_interval (#3174) Fixes issues when specifying a DAG with a schedule_interval of type relativedelta. > DAG with a relativedelta schedule_interval fails > > > Key: AIRFLOW-2231 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2231 > Project: Apache Airflow > Issue Type: Bug > Components: DAG >Reporter: Kyle Brooks >Priority: Major > Attachments: test_reldel.py > > > The documentation for the DAG class says using > dateutil.relativedelta.relativedelta as a schedule_interval is supported but > it fails: > > Traceback (most recent call last): > File "/usr/local/lib/python3.6/site-packages/airflow/models.py", line 285, > in process_file > m = imp.load_source(mod_name, filepath) > File > "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/imp.py", > line 172, in load_source > module = _load(spec) > File "", line 675, in _load > File "", line 655, in _load_unlocked > File "", line 678, in exec_module > File "", line 205, in _call_with_frames_removed > File "/Users/k398995/airflow/dags/test_reldel.py", line 33, in > dagrun_timeout=timedelta(minutes=60)) > File "/usr/local/lib/python3.6/site-packages/airflow/models.py", line 2914, > in __init__ > if schedule_interval in cron_presets: > TypeError: unhashable type: 'relativedelta' > > It looks like the __init__ function for class DAG assumes the > schedule_interval is hashable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2860) DruidHook: time variable is not updated correctly when checking for timeout
[ https://issues.apache.org/jira/browse/AIRFLOW-2860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571770#comment-16571770 ] ASF subversion and git services commented on AIRFLOW-2860: -- Commit 4f138dfd20da867029828371a121b47d9cab37ad in incubator-airflow's branch refs/heads/master from awelsh93 [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=4f138df ] [AIRFLOW-2860] Raise ValueError if timeout < 1 in druid hook > DruidHook: time variable is not updated correctly when checking for timeout > > > Key: AIRFLOW-2860 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2860 > Project: Apache Airflow > Issue Type: Bug > Components: hooks >Reporter: Adam Welsh >Assignee: Adam Welsh >Priority: Trivial > Fix For: 2.0.0 > > > The variable that is used in the condition to check if the Druid ingestion > task has exceeded the max_ingestion_time is not updated correctly. It gets > incremented by one (which is the default value for timeout) but should be > incremented by timeout. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2857) Broken builds for Read the Docs
[ https://issues.apache.org/jira/browse/AIRFLOW-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571738#comment-16571738 ] ASF subversion and git services commented on AIRFLOW-2857: -- Commit 8af0aa96bfe3caa51d67ab393db069d37b0c4169 in incubator-airflow's branch refs/heads/master from Taylor D. Edmiston [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=8af0aa9 ] [AIRFLOW-2857] Fix Read the Docs env (#3703) The Read the Docs build process was broken due to #3660. This PR fixes this. > Broken builds for Read the Docs > --- > > Key: AIRFLOW-2857 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2857 > Project: Apache Airflow > Issue Type: Bug > Components: ci, docs >Reporter: Taylor Edmiston >Assignee: Taylor Edmiston >Priority: Major > Fix For: 2.0.0 > > > Per the mailing list thread "Readthedocs is Broken" on Aug 5, 2018, the Read > the Docs build process is currently broken per the recent addition of > [https://github.com/apache/incubator-airflow/pull/3660|https://github.com/apache/incubator-airflow/pull/3660.]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2864) Incorrect docstrings for SubDagOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571728#comment-16571728 ] ASF subversion and git services commented on AIRFLOW-2864: -- Commit b74a7122d2e24d0a24508f6b5c710456a9d11740 in incubator-airflow's branch refs/heads/master from [~kaxilnaik] [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=b74a712 ] [AIRFLOW-2864] Fix docstrings for SubDagOperator (#3712) The docstrings are currently in the `__init__` method, due to which they are not automatically shown in the Sphinx documentation. > Incorrect docstrings for SubDagOperator > --- > > Key: AIRFLOW-2864 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2864 > Project: Apache Airflow > Issue Type: Improvement > Components: Documentation >Reporter: Kaxil Naik >Assignee: Kaxil Naik >Priority: Minor > Fix For: 2.0.0 > > Attachments: image-2018-08-07-14-55-10-275.png > > > The docstrings are currently in the __init__ method, due to which they are > not automatically shown in the Sphinx documentation. > !image-2018-08-07-14-55-10-275.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2787) Airflow scheduler dies on DAGs with NULL DagRun run_id
[
https://issues.apache.org/jira/browse/AIRFLOW-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570879#comment-16570879
]
ASF subversion and git services commented on AIRFLOW-2787:
--
Commit 27b436ed93f3b13134088a6a6cc13d6012c435f3 in incubator-airflow's branch
refs/heads/master from [~gwax]
[ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=27b436e ]
AIRFLOW-2787 Allow is_backfill to handle NULL DagRun.run_id (#3629)
> Airflow scheduler dies on DAGs with NULL DagRun run_id
> --
>
> Key: AIRFLOW-2787
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2787
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: George Leslie-Waksman
>Priority: Critical
>
> When a DagRun is created with NULL run_id, the scheduler subprocess will
> crash when checking `is_backfill`:
> {noformat}
> Got an exception! Propagating...
> Traceback (most recent call last):
> File "/usr/local/lib/python3.6/site-packages/airflow/jobs.py", line 347, in
> helper
> pickle_dags)
> File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 53,
> in wrapper
> result = func(*args, **kwargs)
> File "/usr/local/lib/python3.6/site-packages/airflow/jobs.py", line 1583,
> in process_file
> self._process_dags(dagbag, dags, ti_keys_to_schedule)
> File "/usr/local/lib/python3.6/site-packages/airflow/jobs.py", line 1175,
> in _process_dags
> self._process_task_instances(dag, tis_out)
> File "/usr/local/lib/python3.6/site-packages/airflow/jobs.py", line 873, in
> _process_task_instances
> if run.is_backfill:
> File "/usr/local/lib/python3.6/site-packages/airflow/models.py", line 4257,
> in is_backfill
> if "backfill" in self.run_id:
> TypeError: argument of type 'NoneType' is not iterable
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2755) k8s workers think DAGs are always in `/tmp/dags`
[ https://issues.apache.org/jira/browse/AIRFLOW-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570752#comment-16570752 ] ASF subversion and git services commented on AIRFLOW-2755: -- Commit 90e88dfe82d781236db39ce3b6dfd92d5f48f6e7 in incubator-airflow's branch refs/heads/master from Aldo Giambelluca [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=90e88df ] [AIRFLOW-2755] Added `kubernetes.worker_dags_folder` configuration (#3612) It was previously hardcoded to `/tmp/dags`. This causes problems with python import of modules in the DAGs folder. > k8s workers think DAGs are always in `/tmp/dags` > > > Key: AIRFLOW-2755 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2755 > Project: Apache Airflow > Issue Type: Bug > Components: configuration, worker >Reporter: Aldo >Assignee: Aldo >Priority: Minor > Fix For: 2.0.0 > > > We have Airflow configured to use the `KubernetesExecutor` and run tasks in > newly created pods. > I tried to use the `PythonOperator` to import the python callable from a > python module located in the DAGs directory as [that should be > possible|https://github.com/apache/incubator-airflow/blob/c7a472ed6b0d8a4720f57ba1140c8cf665757167/airflow/__init__.py#L42]. > Airflow complained that the module was not found. > After a fair amount of digging we found that the issue was that the workers > have the `AIRFLOW__CORE__DAGS_FOLDER` environment variable set to `/tmp/dags` > as [you can see from the > code|https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/kubernetes/worker_configuration.py#L84]. > Unset that environment variable from within the task's pod and running the > task manually worked as expected. I think that this path should be > configurable (I'll give it a try to add a `kubernetes.worker_dags_folder` > configuration). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2860) DruidHook: time variable is not updated correctly when checking for timeout
[ https://issues.apache.org/jira/browse/AIRFLOW-2860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570696#comment-16570696 ] ASF subversion and git services commented on AIRFLOW-2860: -- Commit aa7b25b6afc069077c3e42ab13ef2de3efa7d4fc in incubator-airflow's branch refs/heads/master from [~Fokko] [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=aa7b25b ] Revert "[AIRFLOW-2860] DruidHook: time variable is not updated correctly when checking for timeout (#3707)" This reverts commit d12aacd552878308f9b1c3663414bb7c00c0632b. > DruidHook: time variable is not updated correctly when checking for timeout > > > Key: AIRFLOW-2860 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2860 > Project: Apache Airflow > Issue Type: Bug > Components: hooks >Reporter: Adam Welsh >Assignee: Adam Welsh >Priority: Trivial > Fix For: 2.0.0 > > > The variable that is used in the condition to check if the Druid ingestion > task has exceeded the max_ingestion_time is not updated correctly. It gets > incremented by one (which is the default value for timeout) but should be > incremented by timeout. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2811) Fix scheduler_ops_metrics.py to work
[
https://issues.apache.org/jira/browse/AIRFLOW-2811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570457#comment-16570457
]
ASF subversion and git services commented on AIRFLOW-2811:
--
Commit 84a55f3e546cfbfd5f47302537444c8c8c4d2753 in incubator-airflow's branch
refs/heads/master from [~sekikn]
[ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=84a55f3 ]
[AIRFLOW-2811] Fix scheduler_ops_metrics.py to work (#3653)
This PR fixes timezone problem in
scheduler_ops_metrics.py and makes
its timeout configurable.
> Fix scheduler_ops_metrics.py to work
>
>
> Key: AIRFLOW-2811
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2811
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Major
>
> I tried to run {{scripts/perf/scheduler_ops_metrics.py}} but it failed with
> the following error:
> {code}
> $ python scripts/perf/scheduler_ops_metrics.py
> (snip)
> Traceback (most recent call last):
> File "scripts/perf/scheduler_ops_metrics.py", line 192, in
> main()
> File "scripts/perf/scheduler_ops_metrics.py", line 188, in main
> job.run()
> File "/home/sekikn/dev/incubator-airflow/airflow/jobs.py", line 202, in run
> self._execute()
> File "/home/sekikn/dev/incubator-airflow/airflow/jobs.py", line 1584, in
> _execute
> self._execute_helper(processor_manager)
> File "/home/sekikn/dev/incubator-airflow/airflow/jobs.py", line 1714, in
> _execute_helper
> self.heartbeat()
> File "scripts/perf/scheduler_ops_metrics.py", line 121, in heartbeat
> for dag in dags for task in dag.tasks])
> TypeError: can't subtract offset-naive and offset-aware datetimes
> {code}
> Also, it'd be nice if {{MAX_RUNTIME_SECS}} were configurable, since the
> default value (6 seconds) is too short for all TaskInstances to finish in my
> environment.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2860) DruidHook: time variable is not updated correctly when checking for timeout
[ https://issues.apache.org/jira/browse/AIRFLOW-2860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570019#comment-16570019 ] ASF subversion and git services commented on AIRFLOW-2860: -- Commit d12aacd552878308f9b1c3663414bb7c00c0632b in incubator-airflow's branch refs/heads/master from awelsh93 [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=d12aacd ] [AIRFLOW-2860] DruidHook: time variable is not updated correctly when checking for timeout (#3707) > DruidHook: time variable is not updated correctly when checking for timeout > > > Key: AIRFLOW-2860 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2860 > Project: Apache Airflow > Issue Type: Bug > Components: hooks >Reporter: Adam Welsh >Assignee: Adam Welsh >Priority: Trivial > Fix For: 2.0.0 > > > The variable that is used in the condition to check if the Druid ingestion > task has exceeded the max_ingestion_time is not updated correctly. It gets > incremented by one (which is the default value for timeout) but should be > incremented by timeout. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2845) Remove asserts from the contrib code (change to legal exceptions)
[ https://issues.apache.org/jira/browse/AIRFLOW-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569930#comment-16569930 ] ASF subversion and git services commented on AIRFLOW-2845: -- Commit 5b7a28c7e0dade62bb4d087f25687821e7f11083 in incubator-airflow's branch refs/heads/master from Yuliya Volkova [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=5b7a28c ] [AIRFLOW-2845] Asserts in contrib package code are changed on raise ValueError and TypeError (#3690) > Remove asserts from the contrib code (change to legal exceptions) > -- > > Key: AIRFLOW-2845 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2845 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib >Affects Versions: 1.9.0 >Reporter: Iuliia Volkova >Assignee: Iuliia Volkova >Priority: Minor > Labels: easyfix > Fix For: 2.0.0 > > > Hi guys. `asserts` is used in Airflow contrib package code . And from point > of view for which purposes asserts are really is, it's not correct. > If we look at documentation we could find information what asserts is debug > tool: > [https://docs.python.org/3/reference/simple_stmts.html#the-assert-statement] > and also it is could be disabled globally by default. > If you do not mind, I will be happy to prepare PR for remove asserts from the > contrib module with changing it to raising errors with correct Exceptions and > messages and not just "Assertion Error". > I talk only about src (not about asserts in tests). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2858) Make tox.ini indentation and whitespace consistent
[ https://issues.apache.org/jira/browse/AIRFLOW-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569879#comment-16569879 ] ASF subversion and git services commented on AIRFLOW-2858: -- Commit 6824c86e97d568304a02df51ecd806949a49723a in incubator-airflow's branch refs/heads/master from Taylor D. Edmiston [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=6824c86 ] [AIRFLOW-2858] Make tox.ini indentation and whitespace consistent (#3705) > Make tox.ini indentation and whitespace consistent > -- > > Key: AIRFLOW-2858 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2858 > Project: Apache Airflow > Issue Type: Improvement > Components: ci, tests >Reporter: Taylor Edmiston >Assignee: Taylor Edmiston >Priority: Minor > Fix For: 2.0.0 > > > The tox.ini file currently mixes multiple indentation and whitespace styles. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2853) Add maintainers to the README
[ https://issues.apache.org/jira/browse/AIRFLOW-2853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569625#comment-16569625 ] ASF subversion and git services commented on AIRFLOW-2853: -- Commit dc64649d98f9c6ddb508b49b52a61fc7799e4343 in incubator-airflow's branch refs/heads/master from [~ajc] [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=dc64649 ] [AIRFLOW-2853] Add official committers to README Closes #3699 from andscoop/Add-core-commiters-to- readme > Add maintainers to the README > - > > Key: AIRFLOW-2853 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2853 > Project: Apache Airflow > Issue Type: Task >Reporter: Andy Cooper >Assignee: Andy Cooper >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2821) Doc "Plugins" can be improved
[ https://issues.apache.org/jira/browse/AIRFLOW-2821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569539#comment-16569539 ] ASF subversion and git services commented on AIRFLOW-2821: -- Commit 3157287e8c1aba7649cb7de80dd402889725 in incubator-airflow's branch refs/heads/master from Xiaodong [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=3157287 ] [AIRFLOW-2821] Refine Doc "Plugins" (#3664) > Doc "Plugins" can be improved > - > > Key: AIRFLOW-2821 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2821 > Project: Apache Airflow > Issue Type: Improvement > Components: Documentation >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > > The doc "Plugins" is not very clear in some sense. > Something that users may need to take note is not mentioned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2850) Remove deprecated airflow.utils.apply_defaults
[ https://issues.apache.org/jira/browse/AIRFLOW-2850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569536#comment-16569536 ] ASF subversion and git services commented on AIRFLOW-2850: -- Commit b0e912cea90b63520a8e354e9dcc27471c50a738 in incubator-airflow's branch refs/heads/master from Taylor D. Edmiston [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=b0e912c ] [AIRFLOW-2850] Remove deprecated airflow.utils.apply_defaults (#3695) > Remove deprecated airflow.utils.apply_defaults > -- > > Key: AIRFLOW-2850 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2850 > Project: Apache Airflow > Issue Type: Improvement > Components: utils >Affects Versions: 2.0.0 >Reporter: Taylor Edmiston >Assignee: Taylor Edmiston >Priority: Minor > > This PR removes the wrapper function apply_defaults that's had a deprecation > warning since 2016. As similar "to be deprecated" stuff is removed for 2.0 > in #3692 ([AIRFLOW-2847]), this felt like a good time to take care of related > things. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2839) Answer a question in doc: what if I have connections with the same conn_id in both metadata and environment variables?
[ https://issues.apache.org/jira/browse/AIRFLOW-2839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569533#comment-16569533 ] ASF subversion and git services commented on AIRFLOW-2839: -- Commit c0c63ae2a495b8f808c36acc35d752e916393954 in incubator-airflow's branch refs/heads/master from Xiaodong [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=c0c63ae ] [AIRFLOW-2839] Refine Doc Concepts->Connections (#3678) > Answer a question in doc: what if I have connections with the same conn_id in > both metadata and environment variables? > -- > > Key: AIRFLOW-2839 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2839 > Project: Apache Airflow > Issue Type: Improvement > Components: configuration, Documentation >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > Fix For: 2.0.0 > > > We can define connections either in metadata database or environment > variables. > > However, how about there are connections with the same conn_id in both > metadata database and environment variables? Will one of them be picked? > Which one? Or a random one will be picked? > > The answer is: Airflow will check environment variables > first.[https://github.com/apache/incubator-airflow/blob/master/airflow/hooks/base_hook.py#L70] > The "random picking" mechanism will only be applicable for multiple > connections with the same conn_id in metadata database (if there is no > connection with the same conn_id defined in environment variables). > > This should be made clear in the documentation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2796) Improve code coverage for utils/helpers.py
[ https://issues.apache.org/jira/browse/AIRFLOW-2796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569328#comment-16569328 ] ASF subversion and git services commented on AIRFLOW-2796: -- Commit 012ebfd48203302be906caf949a85b0ccac4613c in incubator-airflow's branch refs/heads/master from [~ajc] [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=012ebfd ] [AIRFLOW-2796] Improve utils helpers code coverage (#3637) > Improve code coverage for utils/helpers.py > -- > > Key: AIRFLOW-2796 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2796 > Project: Apache Airflow > Issue Type: Bug >Reporter: Andy Cooper >Assignee: Andy Cooper >Priority: Trivial > Fix For: 2.0.0 > > > Improve code coverage by adding tests for > * is_container > * is_in > * as_tuple -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2849) devel requirement is not sufficient to check code quality locally
[ https://issues.apache.org/jira/browse/AIRFLOW-2849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569219#comment-16569219 ] ASF subversion and git services commented on AIRFLOW-2849: -- Commit 33dd33c89d4b6454d224ca34bab5ae37fb9812a6 in incubator-airflow's branch refs/heads/master from [~etrabelsi] [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=33dd33c ] [AIRFLOW-2849] Add flake8 to setup.py Closes #3694 from eyaltrabelsi/AIRFLOW-2849-add-flake8-to-setup.py-dev-req-to-run-quality-check-locally > devel requirement is not sufficient to check code quality locally > - > > Key: AIRFLOW-2849 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2849 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Reporter: Eyal Trabelsi >Assignee: Eyal Trabelsi >Priority: Trivial > Fix For: 2.0.0 > > > The devel requirement doesn't list flake8, but in order to check code quality > locally one need to install it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2849) devel requirement is not sufficient to check code quality locally
[ https://issues.apache.org/jira/browse/AIRFLOW-2849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569218#comment-16569218 ] ASF subversion and git services commented on AIRFLOW-2849: -- Commit 33dd33c89d4b6454d224ca34bab5ae37fb9812a6 in incubator-airflow's branch refs/heads/master from [~etrabelsi] [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=33dd33c ] [AIRFLOW-2849] Add flake8 to setup.py Closes #3694 from eyaltrabelsi/AIRFLOW-2849-add-flake8-to-setup.py-dev-req-to-run-quality-check-locally > devel requirement is not sufficient to check code quality locally > - > > Key: AIRFLOW-2849 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2849 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Reporter: Eyal Trabelsi >Assignee: Eyal Trabelsi >Priority: Trivial > Fix For: 2.0.0 > > > The devel requirement doesn't list flake8, but in order to check code quality > locally one need to install it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2836) Minor improvement of contrib.sensors.FileSensor
[
https://issues.apache.org/jira/browse/AIRFLOW-2836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16568224#comment-16568224
]
ASF subversion and git services commented on AIRFLOW-2836:
--
Commit 97bc70bd20d377d8eb372e4a7989ed48f6133ab1 in incubator-airflow's branch
refs/heads/master from Xiaodong
[ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=97bc70b ]
[AIRFLOW-2836] Minor improvement-contrib.sensors.FileSensor (#3674)
- The default value of fs_conn_id was not proper.
- Added a new test in which we try to ignore setting
fs_conn_id explicitly.
- a minor change on how a path is concatenated
> Minor improvement of contrib.sensors.FileSensor
> ---
>
> Key: AIRFLOW-2836
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2836
> Project: Apache Airflow
> Issue Type: Improvement
> Components: contrib
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Minor
>
> h4. *Background*
> The default *fs_conn_id* in contrib.sensors.FileSensor is '_*fs_default2*_'.
> However, when we initiate the database
> (https://github.com/apache/incubator-airflow/blob/master/airflow/utils/db.py#L88),
> there isn't such an entry. It doesn't exist anywhere else.
> h4. *Issue*
> The purpose of _contrib.sensors.FileSensor_ is mainly for checking local file
> system (of course can also be used for NAS). Then the path ("/") from default
> connection 'fs_default' would suffice.
> However, given the default value for *fs_conn_id* in
> contrib.sensors.FileSensor is "fs_default2" (a value doesn't exist), it will
> make the situation much more complex.
> When users intend to check local file system only, they should be able to
> leave *fs_conn_id* default directly, instead of going setting up another
> connection separately.
> h4. Proposal
> Change default value for *fs_conn_id* in contrib.sensors.FileSensor from
> "fs_default2" to "fs_default" (actually in the related test, the *fs_conn_id*
> are all specified to be "fs_default").
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2796) Improve code coverage for utils/helpers.py
[ https://issues.apache.org/jira/browse/AIRFLOW-2796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16568206#comment-16568206 ] ASF subversion and git services commented on AIRFLOW-2796: -- Commit 76a99597e2946d86ddf9a4b72479e6e4ddbac036 in incubator-airflow's branch refs/heads/master from [~noremac201] [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=76a9959 ] [AIRFLOW-2796] Expand code coverage for utils/helpers.py (#3686) > Improve code coverage for utils/helpers.py > -- > > Key: AIRFLOW-2796 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2796 > Project: Apache Airflow > Issue Type: Bug >Reporter: Andy Cooper >Priority: Trivial > > Improve code coverage by adding tests for > * is_container > * is_in > * as_tuple -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2805) Display user's local timezone and DAG's timezone on UI
[ https://issues.apache.org/jira/browse/AIRFLOW-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16568142#comment-16568142 ] ASF subversion and git services commented on AIRFLOW-2805: -- Commit c2927ac38840597c4817a63e744baa60306ff52f in incubator-airflow's branch refs/heads/master from [~verdan] [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=c2927ac ] [AIRFLOW-2805] Display multiple timezones on UI (#3687) > Display user's local timezone and DAG's timezone on UI > -- > > Key: AIRFLOW-2805 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2805 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Verdan Mahmood >Assignee: Verdan Mahmood >Priority: Major > Attachments: Screen Shot 2018-08-02 at 1.08.53 PM.png > > > The UI currently only displays the UTC timezone which is also not in human > readable forms on all places. > Make all the date times in human readable forms. > Also, we need to display user's local timezone and DAG's timezone along with > UTC. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2814) Default Arg "file_process_interval" for class SchedulerJob is inconsistent with doc
[ https://issues.apache.org/jira/browse/AIRFLOW-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567956#comment-16567956 ] ASF subversion and git services commented on AIRFLOW-2814: -- Commit 758be9db8316f4959f6f1ca2d61254190ba783ad in incubator-airflow's branch refs/heads/master from [~kaxilnaik] [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=758be9d ] Revert [AIRFLOW-2814] - Change `min_file_process_interval` to 0 (#3669) - Change the time (in seconds) after which a new DAG should be picked up from the filesystem > Default Arg "file_process_interval" for class SchedulerJob is inconsistent > with doc > --- > > Key: AIRFLOW-2814 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2814 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > Fix For: 2.0.0 > > > h2. Backgrond > In > [https://github.com/XD-DENG/incubator-airflow/blob/master/airflow/jobs.py#L592] > , it was mentioned the default value of argument *file_process_interval* > should be 3 minutes (*file_process_interval:* Parse and schedule each file no > faster than this interval). > The value is normally parsed from the default configuration. However, in the > default config_template, its value is 0 rather than 180 seconds > ([https://github.com/XD-DENG/incubator-airflow/blob/master/airflow/config_templates/default_airflow.cfg#L432] > ). > h2. Issue > This means that actually that each file is parsed and scheduled without > letting Airflow "rest". This conflicts with the design purpose (by default > let it be 180 seconds) and may affect performance significantly. > h2. My Proposal > Change the value in the config template from 0 to 180. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2817) Force explicit choice on GPL dependency
[ https://issues.apache.org/jira/browse/AIRFLOW-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567833#comment-16567833 ] ASF subversion and git services commented on AIRFLOW-2817: -- Commit cefcf4c61f64be3792cbfed509b82a9eb4cc47be in incubator-airflow's branch refs/heads/v1-10-stable from bolkedebruin [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=cefcf4c ] [AIRFLOW-2817] Force explicit choice on GPL dependency (#3660) By default one of Apache Airflow's dependencies pulls in a GPL library. Airflow should not install (and upgrade) without an explicit choice. This is part of the Apache requirements as we cannot depend on Category X software. (cherry picked from commit c37fc0b6ba19e3fe5656ae37cef9b59cef3c29e8) Signed-off-by: Bolke de Bruin (cherry picked from commit b39e4532d9d1086c60b31553d08972bcc68df641) Signed-off-by: Bolke de Bruin > Force explicit choice on GPL dependency > --- > > Key: AIRFLOW-2817 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2817 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Bolke de Bruin >Priority: Major > Fix For: 1.10.0 > > > A more explicit choice on GPL dependency was required by the IPMC -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2817) Force explicit choice on GPL dependency
[ https://issues.apache.org/jira/browse/AIRFLOW-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567832#comment-16567832 ] ASF subversion and git services commented on AIRFLOW-2817: -- Commit b39e4532d9d1086c60b31553d08972bcc68df641 in incubator-airflow's branch refs/heads/v1-10-test from bolkedebruin [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=b39e453 ] [AIRFLOW-2817] Force explicit choice on GPL dependency (#3660) By default one of Apache Airflow's dependencies pulls in a GPL library. Airflow should not install (and upgrade) without an explicit choice. This is part of the Apache requirements as we cannot depend on Category X software. (cherry picked from commit c37fc0b6ba19e3fe5656ae37cef9b59cef3c29e8) Signed-off-by: Bolke de Bruin > Force explicit choice on GPL dependency > --- > > Key: AIRFLOW-2817 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2817 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Bolke de Bruin >Priority: Major > Fix For: 1.10.0 > > > A more explicit choice on GPL dependency was required by the IPMC -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2658) Add GKE specific Kubernetes Pod Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-2658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567410#comment-16567410 ] ASF subversion and git services commented on AIRFLOW-2658: -- Commit b4f43e6c48b976a873448a3cc8a1490efe38a8e8 in incubator-airflow's branch refs/heads/master from [~noremac201] [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=b4f43e6 ] [AIRFLOW-2658] Add GCP specific k8s pod operator (#3532) Executes a task in a Kubernetes pod in the specified Google Kubernetes Engine cluster. This makes it easier to interact with GCP kubernetes engine service because it encapsulates acquiring credentials. > Add GKE specific Kubernetes Pod Operator > > > Key: AIRFLOW-2658 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2658 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Cameron Moberg >Assignee: Cameron Moberg >Priority: Minor > > Currently there is a Kubernetes Pod operator, but it is not really easy to > have it work with GCP Kubernetes Engine, it would be nice to have one. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2238) Update dev/airflow-pr to work with gitub for merge targets
[ https://issues.apache.org/jira/browse/AIRFLOW-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567408#comment-16567408 ] ASF subversion and git services commented on AIRFLOW-2238: -- Commit d843f65af24b237d5066292916cb4700f82f7729 in incubator-airflow's branch refs/heads/master from [~ashb] [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=d843f65 ] [AIRFLOW-2238] Use SSH protocol for pushing to Github (#3680) Since we have MFA enforced we won't be able to use passwords to push and will have to use SSH keypairs. > Update dev/airflow-pr to work with gitub for merge targets > -- > > Key: AIRFLOW-2238 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2238 > Project: Apache Airflow > Issue Type: Improvement > Components: PR tool >Reporter: Ash Berlin-Taylor >Priority: Major > > We are planning on migrating the to the Apache "GitBox" project which lets > committers work directly on github. This will mean we might not _need_ to use > the pr tool, but we should update it so that it merges and pushes back to > github, not the ASF repo. > I think we need to do this before we ask the ASF infra team to migrate our > repo over. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2099) Task details cannot be shown when PythonOperator calls partial function / class instance with __call__
[
https://issues.apache.org/jira/browse/AIRFLOW-2099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16566463#comment-16566463
]
ASF subversion and git services commented on AIRFLOW-2099:
--
Commit a4b9aa386c93d8894a41e986da91d3841bb37f43 in incubator-airflow's branch
refs/heads/master from Matt Revell
[ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=a4b9aa3 ]
[AIRFLOW-2099] Handle getsource() calls gracefully
There are several scenarios where Task Instance view tries to render
Python callables where 'x' is not the correct artefact to target.
This commit adds a helper fuction to test for known scenarios, and
derives the source from the correc artefact or as a default returns 'No
source available for '. This means that even in unknown or
unfixable edge cases, the Task Instance view still renders instead of
displaying an exception.
Closes #3571 from night0wl/AIRFLOW-2099_task_view_type_check
> Task details cannot be shown when PythonOperator calls partial function /
> class instance with __call__
> --
>
> Key: AIRFLOW-2099
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2099
> Project: Apache Airflow
> Issue Type: Bug
> Components: ui
>Affects Versions: Airflow 1.8
>Reporter: Matthew Revell
>Assignee: Matthew Revell
>Priority: Minor
>
> There are several scenarios where the inspect.getsource() method fails with:
> {{object at 0x> is not a module, class, method, function, traceback,
> frame, or code object}}
> One such scenario is described in
> [AIRFLOW-1027|https://issues.apache.org/jira/browse/AIRFLOW-1027] where a
> partial function is used. Another is when an instance of a class which
> implements __call__() is used.
> Example:
> {{class MyClass(object):}}
> {{ def __init__(self):}}
> {{ pass}}
> {{ def __call__(self):}}
> {{ pass}}
> {{my_class = MyClass()}}
> {{dag_task = PythonOperator(}}
> {{ task_id='dag_task',}}
> {{ dag=dag, }}
> {{ python_callable=my_class,}}
> {{)}}
> There exists a PR for AIRFLOW-1027, however, this fix does not address this
> other scenario, and also does not guard against any other edge cases which my
> result in this error in future.
> A better solution would be to catch known scenarios with work arounds, and
> default to reporting that the source is unavailable for unknown cases. This
> would at least display the Task Instance details in every case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2099) Task details cannot be shown when PythonOperator calls partial function / class instance with __call__
[
https://issues.apache.org/jira/browse/AIRFLOW-2099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16566462#comment-16566462
]
ASF subversion and git services commented on AIRFLOW-2099:
--
Commit a4b9aa386c93d8894a41e986da91d3841bb37f43 in incubator-airflow's branch
refs/heads/master from Matt Revell
[ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=a4b9aa3 ]
[AIRFLOW-2099] Handle getsource() calls gracefully
There are several scenarios where Task Instance view tries to render
Python callables where 'x' is not the correct artefact to target.
This commit adds a helper fuction to test for known scenarios, and
derives the source from the correc artefact or as a default returns 'No
source available for '. This means that even in unknown or
unfixable edge cases, the Task Instance view still renders instead of
displaying an exception.
Closes #3571 from night0wl/AIRFLOW-2099_task_view_type_check
> Task details cannot be shown when PythonOperator calls partial function /
> class instance with __call__
> --
>
> Key: AIRFLOW-2099
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2099
> Project: Apache Airflow
> Issue Type: Bug
> Components: ui
>Affects Versions: Airflow 1.8
>Reporter: Matthew Revell
>Assignee: Matthew Revell
>Priority: Minor
>
> There are several scenarios where the inspect.getsource() method fails with:
> {{object at 0x> is not a module, class, method, function, traceback,
> frame, or code object}}
> One such scenario is described in
> [AIRFLOW-1027|https://issues.apache.org/jira/browse/AIRFLOW-1027] where a
> partial function is used. Another is when an instance of a class which
> implements __call__() is used.
> Example:
> {{class MyClass(object):}}
> {{ def __init__(self):}}
> {{ pass}}
> {{ def __call__(self):}}
> {{ pass}}
> {{my_class = MyClass()}}
> {{dag_task = PythonOperator(}}
> {{ task_id='dag_task',}}
> {{ dag=dag, }}
> {{ python_callable=my_class,}}
> {{)}}
> There exists a PR for AIRFLOW-1027, however, this fix does not address this
> other scenario, and also does not guard against any other edge cases which my
> result in this error in future.
> A better solution would be to catch known scenarios with work arounds, and
> default to reporting that the source is unavailable for unknown cases. This
> would at least display the Task Instance details in every case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2829) Brush up the CI script for minikube
[
https://issues.apache.org/jira/browse/AIRFLOW-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16566016#comment-16566016
]
ASF subversion and git services commented on AIRFLOW-2829:
--
Commit 0d193ada44707ec6f50371cb1fc9cbb96d2466db in incubator-airflow's branch
refs/heads/master from [~sekikn]
[ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=0d193ad ]
[AIRFLOW-2829] Brush up the CI script for minikube
Fix scripts/ci/kubernetes/minikube/start_minikube.sh
as follows:
- Make minikube version configurable via
environment variable
- Remove unused variables for readability
- Reorder some lines to remove warnings
- Replace ineffective `return` with `exit`
- Add -E to `sudo minikube` so that non-root
users can use this script locally
> Brush up the CI script for minikube
> ---
>
> Key: AIRFLOW-2829
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2829
> Project: Apache Airflow
> Issue Type: Bug
> Components: ci
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Major
>
> Ran {{scripts/ci/kubernetes/minikube/start_minikube.sh}} locally and found
> some points that can be improved:
> - minikube version is hard-coded
> - Defined but unused variables: {{$_HELM_VERSION}}, {{$_VM_DRIVER}}
> - Undefined variables: {{$unameOut}}
> - The following lines cause warnings if download is skipped:
> {code}
> 69 sudo mv bin/minikube /usr/local/bin/minikube
> 70 sudo mv bin/kubectl /usr/local/bin/kubectl
> {code}
> - {{return}} s at line 81 and 96 won't work since it's outside of a function
> - To run this script as a non-root user, {{-E}} is required for {{sudo}}. See
> https://github.com/kubernetes/minikube/issues/1883.
> {code}
> 105 _MINIKUBE="sudo PATH=$PATH minikube"
> 106
> 107 $_MINIKUBE config set bootstrapper localkube
> 108 $_MINIKUBE start --kubernetes-version=${_KUBERNETES_VERSION}
> --vm-driver=none
> 109 $_MINIKUBE update-context
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2820) In Doc "Scheduling & Triggers", missed to mention manual Dag Trigger feature in web UI
[ https://issues.apache.org/jira/browse/AIRFLOW-2820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16565991#comment-16565991 ] ASF subversion and git services commented on AIRFLOW-2820: -- Commit b120427b652dc308b8ac6fb93fc096a5b951a084 in incubator-airflow's branch refs/heads/master from Xiaodong [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=b120427 ] [AIRFLOW-2820] Add Web UI triggger in doc "Scheduling & Triggers" In documentation page "Scheduling & Triggers", it only mentioned the CLI method to manually trigger a DAG run. However, the manual trigger feature in Web UI should be mentioned as well (it may be even more frequently used by users). > In Doc "Scheduling & Triggers", missed to mention manual Dag Trigger feature > in web UI > --- > > Key: AIRFLOW-2820 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2820 > Project: Apache Airflow > Issue Type: Improvement > Components: Documentation >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > > In documentation page *Scheduling & Triggers*, it only mentioned the CLI > method to manually trigger a DAG run. > However, the manual trigger feature in Web UI should be mentioned as well (it > may be even more frequently used by users). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2716) Replace new Python 3.7 keywords
[ https://issues.apache.org/jira/browse/AIRFLOW-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16565044#comment-16565044 ] ASF subversion and git services commented on AIRFLOW-2716: -- Commit 0423060bb20c54035fafecda8cc0a50e65953f27 in incubator-airflow's branch refs/heads/v1-10-test from [~jacobhayes] [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=0423060 ] [AIRFLOW-2716] Replace async and await py3.7 keywords Closes #3578 from JacobHayes/py37-keywords (cherry picked from commit fcd51f362c6804cb69dee8e2dd054304a0564dde) Signed-off-by: Bolke de Bruin > Replace new Python 3.7 keywords > --- > > Key: AIRFLOW-2716 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2716 > Project: Apache Airflow > Issue Type: Bug >Affects Versions: Airflow 2.0 >Reporter: Jacob Hayes >Assignee: Jacob Hayes >Priority: Major > Fix For: 2.0.0 > > > Python 3.7 added `async` and `await` as reserved keywords, so they need to be > replaced with alternative names. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2716) Replace new Python 3.7 keywords
[ https://issues.apache.org/jira/browse/AIRFLOW-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16565042#comment-16565042 ] ASF subversion and git services commented on AIRFLOW-2716: -- Commit c720c352f0762a483fa1202e1477e740631a8184 in incubator-airflow's branch refs/heads/v1-10-stable from [~jacobhayes] [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=c720c35 ] [AIRFLOW-2716] Replace async and await py3.7 keywords Closes #3578 from JacobHayes/py37-keywords (cherry picked from commit fcd51f362c6804cb69dee8e2dd054304a0564dde) Signed-off-by: Bolke de Bruin (cherry picked from commit 0423060bb20c54035fafecda8cc0a50e65953f27) Signed-off-by: Bolke de Bruin > Replace new Python 3.7 keywords > --- > > Key: AIRFLOW-2716 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2716 > Project: Apache Airflow > Issue Type: Bug >Affects Versions: Airflow 2.0 >Reporter: Jacob Hayes >Assignee: Jacob Hayes >Priority: Major > Fix For: 2.0.0 > > > Python 3.7 added `async` and `await` as reserved keywords, so they need to be > replaced with alternative names. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2817) Force explicit choice on GPL dependency
[ https://issues.apache.org/jira/browse/AIRFLOW-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16565037#comment-16565037 ] ASF subversion and git services commented on AIRFLOW-2817: -- Commit c37fc0b6ba19e3fe5656ae37cef9b59cef3c29e8 in incubator-airflow's branch refs/heads/master from bolkedebruin [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=c37fc0b ] [AIRFLOW-2817] Force explicit choice on GPL dependency (#3660) By default one of Apache Airflow's dependencies pulls in a GPL library. Airflow should not install (and upgrade) without an explicit choice. This is part of the Apache requirements as we cannot depend on Category X software. > Force explicit choice on GPL dependency > --- > > Key: AIRFLOW-2817 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2817 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Bolke de Bruin >Priority: Major > > A more explicit choice on GPL dependency was required by the IPMC -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2756) Marking DAG run does not set start_time and end_time correctly
[ https://issues.apache.org/jira/browse/AIRFLOW-2756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564959#comment-16564959 ] ASF subversion and git services commented on AIRFLOW-2756: -- Commit c26c56487649f8d912c20ebf7aec1677e9996ea4 in incubator-airflow's branch refs/heads/master from yrqls21 [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=c26c564 ] [AIRFLOW-2756] Fix bug in set DAG run state workflow (#3606) > Marking DAG run does not set start_time and end_time correctly > -- > > Key: AIRFLOW-2756 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2756 > Project: Apache Airflow > Issue Type: Bug >Reporter: Kevin Yang >Assignee: Kevin Yang >Priority: Major > > Marking DAG run right now always set end_time while it should set start_time > when marking RUNNING and otherwise end_time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2832) Inconsistencies and linter errors across markdown files
[ https://issues.apache.org/jira/browse/AIRFLOW-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564911#comment-16564911 ] ASF subversion and git services commented on AIRFLOW-2832: -- Commit 6d31c9e4160ce3907e23e53023f30678485be78c in incubator-airflow's branch refs/heads/master from Taylor D. Edmiston [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=6d31c9e ] [AIRFLOW-2832] Lint and resolve inconsistencies in Markdown files (#3670) Clean up the Markdown files and make the formatting consistent > Inconsistencies and linter errors across markdown files > --- > > Key: AIRFLOW-2832 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2832 > Project: Apache Airflow > Issue Type: Improvement > Components: docs, Documentation >Reporter: Taylor Edmiston >Assignee: Taylor Edmiston >Priority: Minor > > There are a number of inconsistencies within and across markdown files in the > Airflow project. Most of these are simple formatting issues easily fixed by > linting (e.g., with mdl). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2814) Default Arg "file_process_interval" for class SchedulerJob is inconsistent with doc
[ https://issues.apache.org/jira/browse/AIRFLOW-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564434#comment-16564434 ] ASF subversion and git services commented on AIRFLOW-2814: -- Commit 1ee1fc4ec0bab25d9e75a8ca1943fc1a91a85546 in incubator-airflow's branch refs/heads/revert-2814 from [~kaxilnaik] [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=1ee1fc4 ] Revert [AIRFLOW-2814] - Change `min_file_process_interval` to 0 > Default Arg "file_process_interval" for class SchedulerJob is inconsistent > with doc > --- > > Key: AIRFLOW-2814 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2814 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > Fix For: 2.0.0 > > > h2. Backgrond > In > [https://github.com/XD-DENG/incubator-airflow/blob/master/airflow/jobs.py#L592] > , it was mentioned the default value of argument *file_process_interval* > should be 3 minutes (*file_process_interval:* Parse and schedule each file no > faster than this interval). > The value is normally parsed from the default configuration. However, in the > default config_template, its value is 0 rather than 180 seconds > ([https://github.com/XD-DENG/incubator-airflow/blob/master/airflow/config_templates/default_airflow.cfg#L432] > ). > h2. Issue > This means that actually that each file is parsed and scheduled without > letting Airflow "rest". This conflicts with the design purpose (by default > let it be 180 seconds) and may affect performance significantly. > h2. My Proposal > Change the value in the config template from 0 to 180. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1104) Concurrency check in scheduler should count queued tasks as well as running
[ https://issues.apache.org/jira/browse/AIRFLOW-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564331#comment-16564331 ] ASF subversion and git services commented on AIRFLOW-1104: -- Commit ed972042a864cd010137190e0bbb1d25a9dcfe83 in incubator-airflow's branch refs/heads/master from Dan Fowler [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=ed97204 ] [AIRFLOW-1104] Update jobs.py so Airflow does not over schedule tasks (#3568) This change will prevent tasks from getting scheduled and queued over the concurrency limits set for the dag > Concurrency check in scheduler should count queued tasks as well as running > --- > > Key: AIRFLOW-1104 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1104 > Project: Apache Airflow > Issue Type: Bug > Environment: see https://github.com/apache/incubator-airflow/pull/2221 > "Tasks with the QUEUED state should also be counted below, but for now we > cannot count them. This is because there is no guarantee that queued tasks in > failed dagruns will or will not eventually run and queued tasks that will > never run will consume slots and can stall a DAG. Once we can guarantee that > all queued tasks in failed dagruns will never run (e.g. make sure that all > running/newly queued TIs have running dagruns), then we can include QUEUED > tasks here, with the constraint that they are in running dagruns." >Reporter: Alex Guziel >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2670) SSHOperator's timeout parameter doesn't affect SSHook timeoot
[ https://issues.apache.org/jira/browse/AIRFLOW-2670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564248#comment-16564248 ] ASF subversion and git services commented on AIRFLOW-2670: -- Commit 3b35d360f6ff8694b6fb4387901c182ca39160b5 in incubator-airflow's branch refs/heads/master from [~noremac201] [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=3b35d36 ] [AIRFLOW-2670] Update SSH Operator's Hook to respect timeout (#3666) > SSHOperator's timeout parameter doesn't affect SSHook timeoot > - > > Key: AIRFLOW-2670 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2670 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib >Affects Versions: Airflow 2.0 >Reporter: jin zhang >Priority: Major > > when I use SSHOperator, SSHOperator's timeout parameter can't set in SSHHook > and it's just effect exce_command. > old version: > self.ssh_hook = SSHHook(ssh_conn_id=self.ssh_conn_id) > I change it to : > self.ssh_hook = SSHHook(ssh_conn_id=self.ssh_conn_id, timeout=self.timeout) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2795) Oracle to Oracle Transfer Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564245#comment-16564245 ] ASF subversion and git services commented on AIRFLOW-2795: -- Commit 9983466fd1f82faad7d74506fd428f2d007e3daf in incubator-airflow's branch refs/heads/master from [~marcus.r...@gmail.com] [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=9983466 ] [AIRFLOW-2795] Oracle to Oracle Transfer Operator (#3639) > Oracle to Oracle Transfer Operator > --- > > Key: AIRFLOW-2795 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2795 > Project: Apache Airflow > Issue Type: New Feature > Components: operators >Reporter: Marcus Rehm >Assignee: Marcus Rehm >Priority: Trivial > > This operator should help in transfer data from one Oracle instance to > another or between tables in the same instance. t's suitable in use cases > where you don't want to or it's not allowed use dblink. > The operator needs a sql query and a destination table in order to work. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2825) S3ToHiveTransfer operator may not may able to handle GZIP file with uppercase ext in S3
[ https://issues.apache.org/jira/browse/AIRFLOW-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564242#comment-16564242 ] ASF subversion and git services commented on AIRFLOW-2825: -- Commit c7e54461c68c70e11b5cd47e9dee9d52f6ee357b in incubator-airflow's branch refs/heads/master from XD-DENG [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=c7e5446 ] [AIRFLOW-2825]Fix S3ToHiveTransfer bug due to case Because upper/lower case was not considered in the file extension check, S3ToHiveTransfer operator may mistakenly think a GZIP file with uppercase ext ".GZ" is not a GZIP file and raise exception. > S3ToHiveTransfer operator may not may able to handle GZIP file with uppercase > ext in S3 > --- > > Key: AIRFLOW-2825 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2825 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > > Because upper/lower case was not considered in the extension check, > S3ToHiveTransfer operator may think a GZIP file with uppercase ext `.GZ` is > not a GZIP file and raise exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2825) S3ToHiveTransfer operator may not may able to handle GZIP file with uppercase ext in S3
[ https://issues.apache.org/jira/browse/AIRFLOW-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564243#comment-16564243 ] ASF subversion and git services commented on AIRFLOW-2825: -- Commit 8d2f57cd104736f4a9b2b87182358a8c2e406c1a in incubator-airflow's branch refs/heads/master from [~Fokko] [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=8d2f57c ] Merge pull request #3665 from XD-DENG/patch-6 [AIRFLOW-2825] Fix S3ToHiveTransfer bug due to case > S3ToHiveTransfer operator may not may able to handle GZIP file with uppercase > ext in S3 > --- > > Key: AIRFLOW-2825 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2825 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > > Because upper/lower case was not considered in the extension check, > S3ToHiveTransfer operator may think a GZIP file with uppercase ext `.GZ` is > not a GZIP file and raise exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2822) PendingDeprecationWarning Invalid arguments: HipChatAPISendRoomNotificationOperator
[
https://issues.apache.org/jira/browse/AIRFLOW-2822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563903#comment-16563903
]
ASF subversion and git services commented on AIRFLOW-2822:
--
Commit 3eb0454cb1da1e96ae5d7ad88db7c1cca71109f3 in incubator-airflow's branch
refs/heads/master from [~elgalu]
[ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=3eb0454 ]
[AIRFLOW-2822] Fix HipChat Deprecation Warning
Fixes PendingDeprecationWarning on HipChatAPISendRoomNotificationOperator
Using `HipChatAPISendRoomNotificationOperator` on Airflow master branch (2.0)
gives:
airflow/models.py:2390: PendingDeprecationWarning:
Invalid arguments were passed to HipChatAPISendRoomNotificationOperator.
Support for passing such arguments will be dropped in Airflow 2.0.
Invalid arguments were:
*args: ()
**kwargs: {'color': 'green'}
category=PendingDeprecationWarning
> PendingDeprecationWarning Invalid arguments:
> HipChatAPISendRoomNotificationOperator
> ---
>
> Key: AIRFLOW-2822
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2822
> Project: Apache Airflow
> Issue Type: Bug
> Components: contrib, operators
>Affects Versions: Airflow 2.0
>Reporter: Leo Gallucci
>Assignee: Leo Gallucci
>Priority: Trivial
> Labels: easyfix
>
> Using `HipChatAPISendRoomNotificationOperator` on Airflow master branch (2.0)
> gives:
> {code:python}
> airflow/models.py:2390: PendingDeprecationWarning:
> Invalid arguments were passed to HipChatAPISendRoomNotificationOperator.
> Support for passing such arguments will be dropped in Airflow 2.0.
> Invalid arguments were:
> *args: ()
> **kwargs: {'color': 'green'}
> category=PendingDeprecationWarning
> {code}
> I've fixed this in my fork:
> https://github.com/elgalu/apache-airflow/commit/83fc940f54e5d6531f66bff256f66765899dc055
> I will send a PR
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2800) Remove airflow/ low-hanging linting errors
[ https://issues.apache.org/jira/browse/AIRFLOW-2800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563863#comment-16563863 ] ASF subversion and git services commented on AIRFLOW-2800: -- Commit 06584fc4b1d82a2dbba98e484d0b4515a169a818 in incubator-airflow's branch refs/heads/master from [~ajc] [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=06584fc ] [AIRFLOW-2800] Remove low-hanging linting errors > Remove airflow/ low-hanging linting errors > -- > > Key: AIRFLOW-2800 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2800 > Project: Apache Airflow > Issue Type: Bug >Reporter: Andy Cooper >Assignee: Andy Cooper >Priority: Major > > Removing low hanging linting errors from airflow directory > Focuses on > * E226 > * W291 > as well as *some* E501 (line too long) where it did not risk reducing > readability -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2238) Update dev/airflow-pr to work with gitub for merge targets
[ https://issues.apache.org/jira/browse/AIRFLOW-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563402#comment-16563402 ] ASF subversion and git services commented on AIRFLOW-2238: -- Commit 6fdc79980b378222bb0706035bedfe5fcefb982d in incubator-airflow's branch refs/heads/master from [~ashb] [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=6fdc799 ] Merge pull request #3413 from ashb/pr-tool-git-config [AIRFLOW-2238] Switch PR tool to push to Github > Update dev/airflow-pr to work with gitub for merge targets > -- > > Key: AIRFLOW-2238 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2238 > Project: Apache Airflow > Issue Type: Improvement > Components: PR tool >Reporter: Ash Berlin-Taylor >Priority: Major > > We are planning on migrating the to the Apache "GitBox" project which lets > committers work directly on github. This will mean we might not _need_ to use > the pr tool, but we should update it so that it merges and pushes back to > github, not the ASF repo. > I think we need to do this before we ask the ASF infra team to migrate our > repo over. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2238) Update dev/airflow-pr to work with gitub for merge targets
[ https://issues.apache.org/jira/browse/AIRFLOW-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563400#comment-16563400 ] ASF subversion and git services commented on AIRFLOW-2238: -- Commit 4484286e49b7272d2f82e022c0ee5a8690ccc564 in incubator-airflow's branch refs/heads/master from [~ashb] [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=4484286 ] [AIRFLOW-2238] Flake8 fixes on dev/airflow-pr > Update dev/airflow-pr to work with gitub for merge targets > -- > > Key: AIRFLOW-2238 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2238 > Project: Apache Airflow > Issue Type: Improvement > Components: PR tool >Reporter: Ash Berlin-Taylor >Priority: Major > > We are planning on migrating the to the Apache "GitBox" project which lets > committers work directly on github. This will mean we might not _need_ to use > the pr tool, but we should update it so that it merges and pushes back to > github, not the ASF repo. > I think we need to do this before we ask the ASF infra team to migrate our > repo over. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2238) Update dev/airflow-pr to work with gitub for merge targets
[ https://issues.apache.org/jira/browse/AIRFLOW-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563401#comment-16563401 ] ASF subversion and git services commented on AIRFLOW-2238: -- Commit d3793c0a5021df6555a720e9038ccf14b79a1196 in incubator-airflow's branch refs/heads/master from [~ashb] [ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=d3793c0 ] [AIRFLOW-2238] Update PR tool to push directly to Github > Update dev/airflow-pr to work with gitub for merge targets > -- > > Key: AIRFLOW-2238 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2238 > Project: Apache Airflow > Issue Type: Improvement > Components: PR tool >Reporter: Ash Berlin-Taylor >Priority: Major > > We are planning on migrating the to the Apache "GitBox" project which lets > committers work directly on github. This will mean we might not _need_ to use > the pr tool, but we should update it so that it merges and pushes back to > github, not the ASF repo. > I think we need to do this before we ask the ASF infra team to migrate our > repo over. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2809) Fix security issue regarding Flask SECRET_KEY
[ https://issues.apache.org/jira/browse/AIRFLOW-2809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561060#comment-16561060 ] ASF subversion and git services commented on AIRFLOW-2809: -- Commit dfa7b26ddaca80ee8fd9915ee9f6eac50fac77f6 in incubator-airflow's branch refs/heads/master from XD-DENG [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=dfa7b26 ] [AIRFLOW-2809] Fix security issue regarding Flask SECRET_KEY It's recommended by Falsk community to use random SECRET_KEY for security reason. However, in Airflow there is a default value for secret_key and most users will ignore to change it. This may cause security concern. Closes #3651 from XD-DENG/patch-2 > Fix security issue regarding Flask SECRET_KEY > - > > Key: AIRFLOW-2809 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2809 > Project: Apache Airflow > Issue Type: Bug > Components: webserver >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Major > > h2. Background > Currently there is a configuration item *secret_key* in the configuration > .cfg file, with a default value "temporary_key". > h2. Issue > Most admins would ignore it and just use the default value "temporary_key". > However, this may be very dangerous. User may modify the cookie if they try > the default SECRET_KEY while the admin didn't change it. > In Flask documentation, it's suggested to have a SECRET_KEY which is as > random as possible ([http://flask.pocoo.org/docs/1.0/quickstart/] ). > h2. My Proposal > If Admin explicitly specified the SECRET_KEY in *.cfg* file, we use this > SECRET_KEY given by Admin. > If the default SECRET_KEY is not changed in *.cfg* file, randomly generate > SECRET_KEY. Meanwhile, print INFO to remind that a randomly generated > SECRET_KEY is used. > This solution will not affect user experience at all. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2716) Replace new Python 3.7 keywords
[ https://issues.apache.org/jira/browse/AIRFLOW-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561058#comment-16561058 ] ASF subversion and git services commented on AIRFLOW-2716: -- Commit fcd51f362c6804cb69dee8e2dd054304a0564dde in incubator-airflow's branch refs/heads/master from [~jacobhayes] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=fcd51f3 ] [AIRFLOW-2716] Replace async and await py3.7 keywords Closes #3578 from JacobHayes/py37-keywords > Replace new Python 3.7 keywords > --- > > Key: AIRFLOW-2716 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2716 > Project: Apache Airflow > Issue Type: Bug >Affects Versions: Airflow 2.0 >Reporter: Jacob Hayes >Assignee: Jacob Hayes >Priority: Major > Fix For: 2.0.0 > > > Python 3.7 added `async` and `await` as reserved keywords, so they need to be > replaced with alternative names. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2814) Default Arg "file_process_interval" for class SchedulerJob is inconsistent with doc
[ https://issues.apache.org/jira/browse/AIRFLOW-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560865#comment-16560865 ] ASF subversion and git services commented on AIRFLOW-2814: -- Commit 9b7525f251f93e625214ff680710f92306df9433 in incubator-airflow's branch refs/heads/master from XD-DENG [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=9b7525f ] [AIRFLOW-2814] Fix inconsistent default config value of min_file_process_interval in config template is 0 However it's supposed to be 180 according to airflow/jobs.py line 592 Closes #3659 from XD-DENG/patch-3 > Default Arg "file_process_interval" for class SchedulerJob is inconsistent > with doc > --- > > Key: AIRFLOW-2814 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2814 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > Fix For: 2.0.0 > > > h2. Backgrond > In > [https://github.com/XD-DENG/incubator-airflow/blob/master/airflow/jobs.py#L592] > , it was mentioned the default value of argument *file_process_interval* > should be 3 minutes (*file_process_interval:* Parse and schedule each file no > faster than this interval). > The value is normally parsed from the default configuration. However, in the > default config_template, its value is 0 rather than 180 seconds > ([https://github.com/XD-DENG/incubator-airflow/blob/master/airflow/config_templates/default_airflow.cfg#L432] > ). > h2. Issue > This means that actually that each file is parsed and scheduled without > letting Airflow "rest". This conflicts with the design purpose (by default > let it be 180 seconds) and may affect performance significantly. > h2. My Proposal > Change the value in the config template from 0 to 180. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2807) Add support for External ID when using STS Assume Role
[ https://issues.apache.org/jira/browse/AIRFLOW-2807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560780#comment-16560780 ] ASF subversion and git services commented on AIRFLOW-2807: -- Commit 449a7fd1b72639f1eb2bbeb033e1642e8eaac96c in incubator-airflow's branch refs/heads/master from [~vvondra] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=449a7fd ] [AIRFLOW-2807] Support STS Assume Role External ID Currently the role assumption method works only if the granting account does not specify an External ID. The external ID is used to solved the confused deputy problem. When using the AWS hook to export data to multiple customers, it's good security practice to use the external ID. There is no backwards compatibility break, the ID will be `None` in existing cases. Moto doesn't provide any convenient way to verify the value was passed in the credential response in tests, so existing test cases are kept. Documentation: https://docs.aws.amazon.com/IAM/lat est/UserGuide/id_roles_create_for- user_externalid.html Closes #3647 from vvondra/support_sts_external_id > Add support for External ID when using STS Assume Role > -- > > Key: AIRFLOW-2807 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2807 > Project: Apache Airflow > Issue Type: Improvement > Components: aws, boto3, hooks >Affects Versions: 1.10.1 >Reporter: Vojtech Vondra >Priority: Minor > Fix For: 2.0.0 > > > Currently the role assumption method works only if the granting account does > not specify an External ID. The external ID is used to solved the confused > deputy problem. When using the AWS hook to export data to multiple customers, > it's good security practice to use the external ID. > Documentation: > https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2816) Remove github copyright clause as it shouldnt be there
[ https://issues.apache.org/jira/browse/AIRFLOW-2816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560771#comment-16560771 ] ASF subversion and git services commented on AIRFLOW-2816: -- Commit 5f3a4cf3037a6caa2f464f3c9c43cb02513478aa in incubator-airflow's branch refs/heads/v1-10-stable from [~bolke] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=5f3a4cf ] [AIRFLOW-2816] Fix license text in docs/license.rst (cherry picked from commit af15f1150d3c2e02df9a8d330ac3dcc18c9d3e97) Signed-off-by: Bolke de Bruin > Remove github copyright clause as it shouldnt be there > -- > > Key: AIRFLOW-2816 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2816 > Project: Apache Airflow > Issue Type: Sub-task >Reporter: Bolke de Bruin >Priority: Major > > probably copy paste error -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2779) Verify and correct licenses
[
https://issues.apache.org/jira/browse/AIRFLOW-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560770#comment-16560770
]
ASF subversion and git services commented on AIRFLOW-2779:
--
Commit eb34cd596b5995474aeaa6a13829df0d52673cde in incubator-airflow's branch
refs/heads/v1-10-stable from [~bolke]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=eb34cd5 ]
[AIRFLOW-2779] Update licenses and remove copyright headers
* ssh_hook is not a port anymore
* auth backends should never had a copyright clause
* minihivecluster isnt used
(cherry picked from commit a5fa1bf8886ee3ced1306f8c2ff58541a72db96c)
Signed-off-by: Bolke de Bruin
> Verify and correct licenses
> ---
>
> Key: AIRFLOW-2779
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2779
> Project: Apache Airflow
> Issue Type: Improvement
>Reporter: Bolke de Bruin
>Priority: Major
> Labels: licenses
> Fix For: 2.0.0, 1.10.1
>
>
> # {color:#00}/airflow/security/utils.py{color}
> {color:#00}2. ./airflow/security/kerberos.py{color}
> {color:#00}3. ./airflow/www_rbac/static/jqClock.min.js{color}
> {color:#00}4. ./airflow/www/static/bootstrap3-typeahead.min.js{color}
> {color:#00}5.
> ./apache-airflow-1.10.0rc2+incubating/scripts/ci/flake8_diff.sh{color}
> {color:#00}6. {color}[https://www.apache.org/legal/resolved.html#optional]
> {color:#00}7. ./docs/license.rst{color}
> {color:#00}8. airflow/contrib/auth/backends/google_auth.py{color}
> {color:#00}9.
> /airflow/contrib/auth/backends/github_enterprise_auth.py{color}
> {color:#00}10. /airflow/contrib/hooks/ssh_hook.py{color}
> {color:#00}11. /airflow/minihivecluster.py{color}
> {color:#00}This files [1][2] seem to be 3rd party ALv2 licensed files
> that refers to a NOTICE file, that information in that NOTICE file (at the
> very least the copyright into) should be in your NOTICE file. This should
> also be noted in LICENSE.{color}
>
> {color:#00}LICENSE is:
> - missing jQuery clock [3] and typeahead [4], as they are ALv2 it’s not
> required to list them but it’s a good idea to do so.
> - missing the license for this [5]
> - this file [7] oddly has © 2016 GitHub, [Inc.at|http://inc.at/] the bottom
> of it{color}
>
> * {color:#00}Year in NOTICE is not correct "2016 and onwards” isn’t
> valid as copyright has an expiry date{color}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2779) Verify and correct licenses
[
https://issues.apache.org/jira/browse/AIRFLOW-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560769#comment-16560769
]
ASF subversion and git services commented on AIRFLOW-2779:
--
Commit fa7574700627cc381b8135392a1c78a76349dc83 in incubator-airflow's branch
refs/heads/v1-10-test from [~bolke]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=fa75747 ]
[AIRFLOW-2779] Update licenses and remove copyright headers
* ssh_hook is not a port anymore
* auth backends should never had a copyright clause
* minihivecluster isnt used
(cherry picked from commit a5fa1bf8886ee3ced1306f8c2ff58541a72db96c)
Signed-off-by: Bolke de Bruin
> Verify and correct licenses
> ---
>
> Key: AIRFLOW-2779
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2779
> Project: Apache Airflow
> Issue Type: Improvement
>Reporter: Bolke de Bruin
>Priority: Major
> Labels: licenses
> Fix For: 2.0.0, 1.10.1
>
>
> # {color:#00}/airflow/security/utils.py{color}
> {color:#00}2. ./airflow/security/kerberos.py{color}
> {color:#00}3. ./airflow/www_rbac/static/jqClock.min.js{color}
> {color:#00}4. ./airflow/www/static/bootstrap3-typeahead.min.js{color}
> {color:#00}5.
> ./apache-airflow-1.10.0rc2+incubating/scripts/ci/flake8_diff.sh{color}
> {color:#00}6. {color}[https://www.apache.org/legal/resolved.html#optional]
> {color:#00}7. ./docs/license.rst{color}
> {color:#00}8. airflow/contrib/auth/backends/google_auth.py{color}
> {color:#00}9.
> /airflow/contrib/auth/backends/github_enterprise_auth.py{color}
> {color:#00}10. /airflow/contrib/hooks/ssh_hook.py{color}
> {color:#00}11. /airflow/minihivecluster.py{color}
> {color:#00}This files [1][2] seem to be 3rd party ALv2 licensed files
> that refers to a NOTICE file, that information in that NOTICE file (at the
> very least the copyright into) should be in your NOTICE file. This should
> also be noted in LICENSE.{color}
>
> {color:#00}LICENSE is:
> - missing jQuery clock [3] and typeahead [4], as they are ALv2 it’s not
> required to list them but it’s a good idea to do so.
> - missing the license for this [5]
> - this file [7] oddly has © 2016 GitHub, [Inc.at|http://inc.at/] the bottom
> of it{color}
>
> * {color:#00}Year in NOTICE is not correct "2016 and onwards” isn’t
> valid as copyright has an expiry date{color}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2816) Remove github copyright clause as it shouldnt be there
[ https://issues.apache.org/jira/browse/AIRFLOW-2816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560768#comment-16560768 ] ASF subversion and git services commented on AIRFLOW-2816: -- Commit 868f9855739cd899509d40017361130f4609 in incubator-airflow's branch refs/heads/v1-10-test from [~bolke] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=868f985 ] [AIRFLOW-2816] Fix license text in docs/license.rst (cherry picked from commit af15f1150d3c2e02df9a8d330ac3dcc18c9d3e97) Signed-off-by: Bolke de Bruin > Remove github copyright clause as it shouldnt be there > -- > > Key: AIRFLOW-2816 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2816 > Project: Apache Airflow > Issue Type: Sub-task >Reporter: Bolke de Bruin >Priority: Major > > probably copy paste error -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2816) Remove github copyright clause as it shouldnt be there
[ https://issues.apache.org/jira/browse/AIRFLOW-2816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560741#comment-16560741 ] ASF subversion and git services commented on AIRFLOW-2816: -- Commit af15f1150d3c2e02df9a8d330ac3dcc18c9d3e97 in incubator-airflow's branch refs/heads/master from [~bolke] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=af15f11 ] [AIRFLOW-2816] Fix license text in docs/license.rst > Remove github copyright clause as it shouldnt be there > -- > > Key: AIRFLOW-2816 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2816 > Project: Apache Airflow > Issue Type: Sub-task >Reporter: Bolke de Bruin >Priority: Major > > probably copy paste error -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2815) Notice cannot contain "onwards" and needs to be specific
[ https://issues.apache.org/jira/browse/AIRFLOW-2815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560719#comment-16560719 ] ASF subversion and git services commented on AIRFLOW-2815: -- Commit fdca7196d40d89a1a1a61454f4fb338eb24bebc5 in incubator-airflow's branch refs/heads/v1-10-test from [~bolke] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=fdca719 ] [AIRFLOW-2815] Use correct copyright period "onwards" is not specific enough (cherry picked from commit 4f562ba8c7f12b8de74e40fe67cac7b4f38413b1) Signed-off-by: Bolke de Bruin > Notice cannot contain "onwards" and needs to be specific > > > Key: AIRFLOW-2815 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2815 > Project: Apache Airflow > Issue Type: Sub-task >Reporter: Bolke de Bruin >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2815) Notice cannot contain "onwards" and needs to be specific
[ https://issues.apache.org/jira/browse/AIRFLOW-2815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560720#comment-16560720 ] ASF subversion and git services commented on AIRFLOW-2815: -- Commit f0549211f5b026a5b2d5c3bc6b3b74d0bf0a6536 in incubator-airflow's branch refs/heads/v1-10-stable from [~bolke] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=f054921 ] [AIRFLOW-2815] Use correct copyright period "onwards" is not specific enough (cherry picked from commit 4f562ba8c7f12b8de74e40fe67cac7b4f38413b1) Signed-off-by: Bolke de Bruin > Notice cannot contain "onwards" and needs to be specific > > > Key: AIRFLOW-2815 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2815 > Project: Apache Airflow > Issue Type: Sub-task >Reporter: Bolke de Bruin >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2815) Notice cannot contain "onwards" and needs to be specific
[ https://issues.apache.org/jira/browse/AIRFLOW-2815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560718#comment-16560718 ] ASF subversion and git services commented on AIRFLOW-2815: -- Commit 4f562ba8c7f12b8de74e40fe67cac7b4f38413b1 in incubator-airflow's branch refs/heads/master from [~bolke] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=4f562ba ] [AIRFLOW-2815] Use correct copyright period "onwards" is not specific enough > Notice cannot contain "onwards" and needs to be specific > > > Key: AIRFLOW-2815 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2815 > Project: Apache Airflow > Issue Type: Sub-task >Reporter: Bolke de Bruin >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2810) Typo in Xcom model timestamp field
[ https://issues.apache.org/jira/browse/AIRFLOW-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560717#comment-16560717 ] ASF subversion and git services commented on AIRFLOW-2810: -- Commit 3b7e71f820bd312f2013d41eb7b2f9486a532036 in incubator-airflow's branch refs/heads/v1-10-stable from [~andywilco] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=3b7e71f ] [AIRFLOW-2810] Fix typo in Xcom model timestamp Fix typo in Xcom model timestamp field No new testing - the field is already represented in migrations Closes #3652 from andywilco/fix_datetime_typo (cherry picked from commit a338f3276835af45765d24a6e6d43ad4ba4d66ba) Signed-off-by: Bolke de Bruin > Typo in Xcom model timestamp field > -- > > Key: AIRFLOW-2810 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2810 > Project: Apache Airflow > Issue Type: Bug > Components: models >Affects Versions: 1.10 >Reporter: Andy Wilcox >Assignee: Andy Wilcox >Priority: Minor > > > Looks like a find/replace error. Should be type UtcDateTime, is legacy > DateTime -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2810) Typo in Xcom model timestamp field
[ https://issues.apache.org/jira/browse/AIRFLOW-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560716#comment-16560716 ] ASF subversion and git services commented on AIRFLOW-2810: -- Commit d8869a2610cc71d72d89e23a989c1050b4d8824e in incubator-airflow's branch refs/heads/v1-10-test from [~andywilco] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=d8869a2 ] [AIRFLOW-2810] Fix typo in Xcom model timestamp Fix typo in Xcom model timestamp field No new testing - the field is already represented in migrations Closes #3652 from andywilco/fix_datetime_typo (cherry picked from commit a338f3276835af45765d24a6e6d43ad4ba4d66ba) Signed-off-by: Bolke de Bruin > Typo in Xcom model timestamp field > -- > > Key: AIRFLOW-2810 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2810 > Project: Apache Airflow > Issue Type: Bug > Components: models >Affects Versions: 1.10 >Reporter: Andy Wilcox >Assignee: Andy Wilcox >Priority: Minor > > > Looks like a find/replace error. Should be type UtcDateTime, is legacy > DateTime -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2812) Fix error in Updating.md for upgrading to 1.10
[
https://issues.apache.org/jira/browse/AIRFLOW-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560714#comment-16560714
]
ASF subversion and git services commented on AIRFLOW-2812:
--
Commit b5ba06dd57433d0b116e1b64f95093e078a41495 in incubator-airflow's branch
refs/heads/v1-10-test from [~nrhvyc]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=b5ba06d ]
[AIRFLOW-2812] Fix error in Updating.md for upgrading to 1.10
Closes #3654 from nrhvyc/AIRFLOW-2812
(cherry picked from commit 3b8c4ff7a02bb7914ffab8edc810a69e77cca0e9)
Signed-off-by: Bolke de Bruin
> Fix error in Updating.md for upgrading to 1.10
> --
>
> Key: AIRFLOW-2812
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2812
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.10
>Reporter: Nick Hughes
>Assignee: Nick Hughes
>Priority: Minor
>
> Error in Updating.md directions under Logging Configuration having too many
> brackets which results in errors:
> * {{log_filename_template}}
> * {{log_processor_filename_template}}{{}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2812) Fix error in Updating.md for upgrading to 1.10
[
https://issues.apache.org/jira/browse/AIRFLOW-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560712#comment-16560712
]
ASF subversion and git services commented on AIRFLOW-2812:
--
Commit 3b8c4ff7a02bb7914ffab8edc810a69e77cca0e9 in incubator-airflow's branch
refs/heads/v1-10-stable from [~nrhvyc]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=3b8c4ff ]
[AIRFLOW-2812] Fix error in Updating.md for upgrading to 1.10
Closes #3654 from nrhvyc/AIRFLOW-2812
> Fix error in Updating.md for upgrading to 1.10
> --
>
> Key: AIRFLOW-2812
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2812
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.10
>Reporter: Nick Hughes
>Assignee: Nick Hughes
>Priority: Minor
>
> Error in Updating.md directions under Logging Configuration having too many
> brackets which results in errors:
> * {{log_filename_template}}
> * {{log_processor_filename_template}}{{}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2812) Fix error in Updating.md for upgrading to 1.10
[
https://issues.apache.org/jira/browse/AIRFLOW-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560711#comment-16560711
]
ASF subversion and git services commented on AIRFLOW-2812:
--
Commit 3b8c4ff7a02bb7914ffab8edc810a69e77cca0e9 in incubator-airflow's branch
refs/heads/v1-10-stable from [~nrhvyc]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=3b8c4ff ]
[AIRFLOW-2812] Fix error in Updating.md for upgrading to 1.10
Closes #3654 from nrhvyc/AIRFLOW-2812
> Fix error in Updating.md for upgrading to 1.10
> --
>
> Key: AIRFLOW-2812
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2812
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.10
>Reporter: Nick Hughes
>Assignee: Nick Hughes
>Priority: Minor
>
> Error in Updating.md directions under Logging Configuration having too many
> brackets which results in errors:
> * {{log_filename_template}}
> * {{log_processor_filename_template}}{{}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2563) Pig Hook Doesn't work for Python 3
[ https://issues.apache.org/jira/browse/AIRFLOW-2563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560455#comment-16560455 ] ASF subversion and git services commented on AIRFLOW-2563: -- Commit 3e7e42f028279a628d9e15d1ae4b6005593f8afb in incubator-airflow's branch refs/heads/master from [~jakahn] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=3e7e42f ] [AIRFLOW-2563] Fix PigCliHook Python 3 string/bytes use Unit tests added for PigCliHook as well to prevent future issues. Closes #3594 from jakahn/master > Pig Hook Doesn't work for Python 3 > -- > > Key: AIRFLOW-2563 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2563 > Project: Apache Airflow > Issue Type: Bug >Reporter: Murium Iqbal >Assignee: Jasper Kahn >Priority: Major > > Pig Hook doesn't work in Python3 due to differences in handling string and > bytes as described in this stackO post > https://stackoverflow.com/questions/50652034/pig-hook-in-airflow-doesnt-work-for-python3 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2810) Typo in Xcom model timestamp field
[ https://issues.apache.org/jira/browse/AIRFLOW-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560015#comment-16560015 ] ASF subversion and git services commented on AIRFLOW-2810: -- Commit a338f3276835af45765d24a6e6d43ad4ba4d66ba in incubator-airflow's branch refs/heads/master from [~andywilco] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=a338f32 ] [AIRFLOW-2810] Fix typo in Xcom model timestamp Fix typo in Xcom model timestamp field No new testing - the field is already represented in migrations Closes #3652 from andywilco/fix_datetime_typo > Typo in Xcom model timestamp field > -- > > Key: AIRFLOW-2810 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2810 > Project: Apache Airflow > Issue Type: Bug > Components: models >Affects Versions: 1.10 >Reporter: Andy Wilcox >Assignee: Andy Wilcox >Priority: Minor > > > Looks like a find/replace error. Should be type UtcDateTime, is legacy > DateTime -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2766) Replicated *Base date* in each tab on DAG detail view
[ https://issues.apache.org/jira/browse/AIRFLOW-2766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558769#comment-16558769 ] ASF subversion and git services commented on AIRFLOW-2766: -- Commit 25f31c9b773b1bf61738282c8e4ab997fce57e45 in incubator-airflow's branch refs/heads/master from [~verdan] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=25f31c9 ] [AIRFLOW-2766][Airflow-2766] Respect shared datetime across tabs Closes #3615 from verdan/AIRFLOW-2766-shared- datetime > Replicated *Base date* in each tab on DAG detail view > - > > Key: AIRFLOW-2766 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2766 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Verdan Mahmood >Assignee: Verdan Mahmood >Priority: Minor > Fix For: 2.0.0 > > Attachments: Screen Shot 2018-07-18 at 1.57.22 PM.png > > > At the moment, most of the tabs on DAG detail page has the *Base date* > feature which behaves only for that particular tab/view. From end-user's > perspective it is confusing as a user assume to select a date and then see > all views based on that date > The idea is to move the *Base date* outside the tabs, and each tab will then > behave on the *Base date* on the Global level. Users will then only need to > select the base date once and every tab will behave on that date. > Please see attached screenshot for reference > Or, the second approach could be to make sure the change of Base Date in each > tab remains the same and doesn't change on page refresh. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2766) Replicated *Base date* in each tab on DAG detail view
[ https://issues.apache.org/jira/browse/AIRFLOW-2766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558768#comment-16558768 ] ASF subversion and git services commented on AIRFLOW-2766: -- Commit 25f31c9b773b1bf61738282c8e4ab997fce57e45 in incubator-airflow's branch refs/heads/master from [~verdan] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=25f31c9 ] [AIRFLOW-2766][Airflow-2766] Respect shared datetime across tabs Closes #3615 from verdan/AIRFLOW-2766-shared- datetime > Replicated *Base date* in each tab on DAG detail view > - > > Key: AIRFLOW-2766 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2766 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Verdan Mahmood >Assignee: Verdan Mahmood >Priority: Minor > Fix For: 2.0.0 > > Attachments: Screen Shot 2018-07-18 at 1.57.22 PM.png > > > At the moment, most of the tabs on DAG detail page has the *Base date* > feature which behaves only for that particular tab/view. From end-user's > perspective it is confusing as a user assume to select a date and then see > all views based on that date > The idea is to move the *Base date* outside the tabs, and each tab will then > behave on the *Base date* on the Global level. Users will then only need to > select the base date once and every tab will behave on that date. > Please see attached screenshot for reference > Or, the second approach could be to make sure the change of Base Date in each > tab remains the same and doesn't change on page refresh. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2776) Tree view JSON is unnecessarily large
[ https://issues.apache.org/jira/browse/AIRFLOW-2776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558765#comment-16558765 ] ASF subversion and git services commented on AIRFLOW-2776: -- Commit 68535d1886f22b0fa833727aac4d0ee3ce12edf1 in incubator-airflow's branch refs/heads/master from [~abdul-stripe] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=68535d1 ] [AIRFLOW-2776] Compress tree view JSON The tree view generates JSON that can be massive for bigger DAGs, up to 10s of MBs. The JSON is currently prettified, which both takes up more CPU time during serialization, and slows down everything else that uses it. Considering the JSON is only meant to be used programmatically, this is an easy win Closes #3620 from abdul-stripe/smaller-tree-view- json > Tree view JSON is unnecessarily large > - > > Key: AIRFLOW-2776 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2776 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Abdul Nimeri >Assignee: Abdul Nimeri >Priority: Minor > Fix For: 2.0.0 > > > The tree view generates JSON that can be massive for bigger DAGs –– some of > our tree views at stripe have 10s of MBs of JSON. > The [generated JSON is > prettified|https://github.com/apache/incubator-airflow/blob/52c745da71a6da798f7322956967b5e818b56e48/airflow/www/views.py#L1480], > which both takes up more CPU time during serialization, as well as slowing > everything else that uses it. We patched this on stripe's fork by removing > all whitespace and had a ton of speedup for bigger tree views. Considering > the JSON is only meant to be used programmatically, this is probably an easy > win. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2777) dag.sub_dag(...) speedups
[ https://issues.apache.org/jira/browse/AIRFLOW-2777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558761#comment-16558761 ] ASF subversion and git services commented on AIRFLOW-2777: -- Commit 4cd72b91bd024e6be9abb30aee345c0a90a6561b in incubator-airflow's branch refs/heads/master from [~abdul-stripe] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=4cd72b9 ] [AIRFLOW-2777] speed up dag.sub_dag(...) previous version created the subdag by copying over all the tasks, and then filtering them down. it's a lot faster if we only copy over the tasks we need Closes #3621 from abdul-stripe/faster-subdag > dag.sub_dag(...) speedups > - > > Key: AIRFLOW-2777 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2777 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Abdul Nimeri >Assignee: Abdul Nimeri >Priority: Minor > Fix For: 2.0.0 > > > dag.sub_dag(...) currently works by first deep copying the entire dag, and > then filtering down to the appropriate tasks. that can be slow since deep > copying a big dag takes a while. specifically, copying over all the tasks is > the bottleneck > this can be a lot faster by instead only copying over the filtered down tasks > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2765) Set default mime_charset in email_operator.EmailOperator to UTF-8, to match email utils
[
https://issues.apache.org/jira/browse/AIRFLOW-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558755#comment-16558755
]
ASF subversion and git services commented on AIRFLOW-2765:
--
Commit 8c441a0d88627fb97b465c5045bdcad24f1d305c in incubator-airflow's branch
refs/heads/master from Jeffrey Scott Keone Payne
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=8c441a0 ]
[AIRFLOW-2765] Set default mime_charset to UTF-8
Closes #3627 from jeffkpayne/AIRFLOW-2765
> Set default mime_charset in email_operator.EmailOperator to UTF-8, to match
> email utils
> ---
>
> Key: AIRFLOW-2765
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2765
> Project: Apache Airflow
> Issue Type: Improvement
> Components: operators
>Reporter: Jeffrey Payne
>Assignee: Jeffrey Payne
>Priority: Major
> Fix For: 2.0.0
>
>
> Assuming we would want both {{EmailOperator()}} and
> {{utils.email.send_email()}} to use the same default value for the
> {{mime_charset}} parameter. The default for {{utils.email.send_email()}} was
> recently changed to {{UTF-8}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
