[GitHub] [airflow] codecov-io commented on issue #6312: [AIRFLOW-5641] support running git sync container as root
codecov-io commented on issue #6312: [AIRFLOW-5641] support running git sync container as root URL: https://github.com/apache/airflow/pull/6312#issuecomment-541285583 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6312?src=pr=h1) Report > Merging [#6312](https://codecov.io/gh/apache/airflow/pull/6312?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/c0d98a72394b68df27db4d1cab4a94021980e0fb?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6312?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6312 +/- ## == + Coverage 80.34% 80.35% +<.01% == Files 616 616 Lines 3573335734 +1 == + Hits2871128713 +2 + Misses 7022 7021 -1 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6312?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/kubernetes/worker\_configuration.py](https://codecov.io/gh/apache/airflow/pull/6312/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3dvcmtlcl9jb25maWd1cmF0aW9uLnB5) | `96.4% <ø> (ø)` | :arrow_up: | | [airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/6312/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==) | `58.99% <100%> (+0.09%)` | :arrow_up: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6312/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `56.55% <0%> (-0.35%)` | :arrow_down: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6312/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.77% <0%> (+0.5%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6312?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6312?src=pr=footer). Last update [c0d98a7...8bd02cd](https://codecov.io/gh/apache/airflow/pull/6312?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-5642) sqlalchemy.exc.OperationalError happens when sql is too long
[
https://issues.apache.org/jira/browse/AIRFLOW-5642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chen xianxin updated AIRFLOW-5642:
--
Description:
h2. code
airflow\jobs\scheduler_job.py:_change_state_for_tasks_failed_to_execute(self,
session)
<
...
filter_for_ti_state_change = (
[and_(
TI.dag_id == dag_id,
TI.task_id == task_id,
TI.execution_date == execution_date,
TI._try_number == try_number - 1,
TI.state == State.QUEUED)
for dag_id, task_id, execution_date, try_number
in self.executor.queued_tasks.keys()])
ti_query = (session.query(TI)
.filter(or_(*filter_for_ti_state_change)))
tis_to_set_to_scheduled = (ti_query
.with_for_update()
.all())
...
>
h2. Error description
{{if there is too many queued_tasks, sqlalchemy.exc.OperationalError will
occur. details are as follow:}}
{{<}}
{{sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2006, "MySQL
server has gone away (BrokenPipeError(32, 'Broken pipe'))")}}
[SQL: SELECT ...]
(Background on this error at: http://sqlalche.me/e/e3q8)
{{> }}
> sqlalchemy.exc.OperationalError happens when sql is too long
>
>
> Key: AIRFLOW-5642
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5642
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 1.10.5
>Reporter: chen xianxin
>Priority: Minor
>
> h2. code
> airflow\jobs\scheduler_job.py:_change_state_for_tasks_failed_to_execute(self,
> session)
> <
> ...
> filter_for_ti_state_change = (
> [and_(
> TI.dag_id == dag_id,
> TI.task_id == task_id,
> TI.execution_date == execution_date,
> TI._try_number == try_number - 1,
> TI.state == State.QUEUED)
> for dag_id, task_id, execution_date, try_number
> in self.executor.queued_tasks.keys()])
> ti_query = (session.query(TI)
> .filter(or_(*filter_for_ti_state_change)))
> tis_to_set_to_scheduled = (ti_query
> .with_for_update()
> .all())
> ...
> >
> h2. Error description
> {{if there is too many queued_tasks, sqlalchemy.exc.OperationalError will
> occur. details are as follow:}}
> {{<}}
> {{sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2006,
> "MySQL server has gone away (BrokenPipeError(32, 'Broken pipe'))")}}
> [SQL: SELECT ...]
> (Background on this error at: http://sqlalche.me/e/e3q8)
> {{> }}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5639) DagFileProcessor: parse dag files every time, which consume lots of resources and is unnecessary
[ https://issues.apache.org/jira/browse/AIRFLOW-5639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chen xianxin updated AIRFLOW-5639: -- Description: h2. Code airflow\models\dagbag.py: process_file(self, filepath, only_if_updated=True, safe_mode=True) < file_last_changed_on_disk = datetime.fromtimestamp(os.path.getmtime(filepath)) if only_if_updated \ and filepath in self.file_last_changed \ and file_last_changed_on_disk == *self.file_last_changed*[filepath]: return found_dags > h2. Problem description *self.file_last_changed* doesn't work, beacause it's set to empty when initializing. so even no changes were made to the file, it will also parse was: airflow\models\dagbag.py: process_file(self, filepath, only_if_updated=True, safe_mode=True) file_last_changed_on_disk = datetime.fromtimestamp(os.path.getmtime(filepath)) if only_if_updated \ and filepath in self.file_last_changed \ and file_last_changed_on_disk == self.file_last_changed[filepath]: return found_dags self.file_last_changed doesn't work, beacause it's set to empty when initializing. so even no changes were made to the file, it will parse again > DagFileProcessor: parse dag files every time, which consume lots of resources > and is unnecessary > > > Key: AIRFLOW-5639 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5639 > Project: Apache Airflow > Issue Type: Improvement > Components: scheduler >Affects Versions: 1.10.5 >Reporter: chen xianxin >Priority: Major > > h2. Code > airflow\models\dagbag.py: process_file(self, filepath, only_if_updated=True, > safe_mode=True) > < > file_last_changed_on_disk = datetime.fromtimestamp(os.path.getmtime(filepath)) > if only_if_updated \ > and filepath in self.file_last_changed \ > and file_last_changed_on_disk == *self.file_last_changed*[filepath]: > return found_dags > > > h2. Problem description > *self.file_last_changed* doesn't work, beacause it's set to empty when > initializing. so even no changes were made to the file, it will also parse -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5642) sqlalchemy.exc.OperationalError happens when sql is too long, which cause scheduler processor will be exited
[
https://issues.apache.org/jira/browse/AIRFLOW-5642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chen xianxin updated AIRFLOW-5642:
--
Summary: sqlalchemy.exc.OperationalError happens when sql is too long,
which cause scheduler processor will be exited (was:
sqlalchemy.exc.OperationalError happens when sql is too long, which cause
scheduler processor be exited)
> sqlalchemy.exc.OperationalError happens when sql is too long, which cause
> scheduler processor will be exited
>
>
> Key: AIRFLOW-5642
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5642
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 1.10.5
>Reporter: chen xianxin
>Priority: Minor
>
> h2. code
> airflow\jobs\scheduler_job.py:_change_state_for_tasks_failed_to_execute(self,
> session)
> <
> ...
> filter_for_ti_state_change = (
> [and_(
> TI.dag_id == dag_id,
> TI.task_id == task_id,
> TI.execution_date == execution_date,
> TI._try_number == try_number - 1,
> TI.state == State.QUEUED)
> for dag_id, task_id, execution_date, try_number
> in self.executor.queued_tasks.keys()])
> ti_query = (session.query(TI)
> .filter(or_(*filter_for_ti_state_change)))
> tis_to_set_to_scheduled = (ti_query
> .with_for_update()
> .all())
> ...
> >
> h2. Error description
> {{if there is too many queued_tasks, sqlalchemy.exc.OperationalError will
> occur. details are as follow:}}
> {{<}}
> {{sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2006,
> "MySQL server has gone away (BrokenPipeError(32, 'Broken pipe'))")}}
> [SQL: SELECT ...]
> (Background on this error at: http://sqlalche.me/e/e3q8)
> {{> }}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5642) sqlalchemy.exc.OperationalError happens when sql is too long, which cause scheduler processor be exited
[
https://issues.apache.org/jira/browse/AIRFLOW-5642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chen xianxin updated AIRFLOW-5642:
--
Summary: sqlalchemy.exc.OperationalError happens when sql is too long,
which cause scheduler processor be exited (was:
sqlalchemy.exc.OperationalError happens when sql is too long)
> sqlalchemy.exc.OperationalError happens when sql is too long, which cause
> scheduler processor be exited
> ---
>
> Key: AIRFLOW-5642
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5642
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 1.10.5
>Reporter: chen xianxin
>Priority: Minor
>
> h2. code
> airflow\jobs\scheduler_job.py:_change_state_for_tasks_failed_to_execute(self,
> session)
> <
> ...
> filter_for_ti_state_change = (
> [and_(
> TI.dag_id == dag_id,
> TI.task_id == task_id,
> TI.execution_date == execution_date,
> TI._try_number == try_number - 1,
> TI.state == State.QUEUED)
> for dag_id, task_id, execution_date, try_number
> in self.executor.queued_tasks.keys()])
> ti_query = (session.query(TI)
> .filter(or_(*filter_for_ti_state_change)))
> tis_to_set_to_scheduled = (ti_query
> .with_for_update()
> .all())
> ...
> >
> h2. Error description
> {{if there is too many queued_tasks, sqlalchemy.exc.OperationalError will
> occur. details are as follow:}}
> {{<}}
> {{sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2006,
> "MySQL server has gone away (BrokenPipeError(32, 'Broken pipe'))")}}
> [SQL: SELECT ...]
> (Background on this error at: http://sqlalche.me/e/e3q8)
> {{> }}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (AIRFLOW-5642) sqlalchemy.exc.OperationalError happens when sql is too long
chen xianxin created AIRFLOW-5642: - Summary: sqlalchemy.exc.OperationalError happens when sql is too long Key: AIRFLOW-5642 URL: https://issues.apache.org/jira/browse/AIRFLOW-5642 Project: Apache Airflow Issue Type: Bug Components: scheduler Affects Versions: 1.10.5 Reporter: chen xianxin -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-5606) Create Production-ready Docker image for Airflow
[ https://issues.apache.org/jira/browse/AIRFLOW-5606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk reassigned AIRFLOW-5606: - Assignee: Jarek Potiuk > Create Production-ready Docker image for Airflow > > > Key: AIRFLOW-5606 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5606 > Project: Apache Airflow > Issue Type: Improvement > Components: ci >Affects Versions: 2.0.0, 1.10.6 >Reporter: Jarek Potiuk >Assignee: Jarek Potiuk >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (AIRFLOW-5606) Create Production-ready Docker image for Airflow
[ https://issues.apache.org/jira/browse/AIRFLOW-5606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-5606 started by Jarek Potiuk. - > Create Production-ready Docker image for Airflow > > > Key: AIRFLOW-5606 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5606 > Project: Apache Airflow > Issue Type: Improvement > Components: ci >Affects Versions: 2.0.0, 1.10.6 >Reporter: Jarek Potiuk >Assignee: Jarek Potiuk >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5641) Not able to run git sync init container as root
[
https://issues.apache.org/jira/browse/AIRFLOW-5641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949882#comment-16949882
]
ASF GitHub Bot commented on AIRFLOW-5641:
-
houqp commented on pull request #6312: [AIRFLOW-5641] support running git sync
container as root
URL: https://github.com/apache/airflow/pull/6312
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5641
### Description
Fixed a regression that prevents user from using root as user for git sync
container.
Also changed `_get_security_context_val` so that it behaves as documented.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
See changed files in tests folder
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Not able to run git sync init container as root
> ---
>
> Key: AIRFLOW-5641
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5641
> Project: Apache Airflow
> Issue Type: Bug
> Components: executor-kubernetes
>Affects Versions: 2.0.0
>Reporter: QP Hou
>Assignee: QP Hou
>Priority: Critical
>
> We used to be able to run git sync init container as root user. A recent
> refactoring of the code base (AIRFLOW-4851) introduced a regression that
> makes it impossible to run git sync container as root. As a result, we are
> not able to start any task using kubernetes executor.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] houqp opened a new pull request #6312: [AIRFLOW-5641] support running git sync container as root
houqp opened a new pull request #6312: [AIRFLOW-5641] support running git sync
container as root
URL: https://github.com/apache/airflow/pull/6312
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5641
### Description
Fixed a regression that prevents user from using root as user for git sync
container.
Also changed `_get_security_context_val` so that it behaves as documented.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
See changed files in tests folder
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-5641) Not able to run git sync init container as root
QP Hou created AIRFLOW-5641: --- Summary: Not able to run git sync init container as root Key: AIRFLOW-5641 URL: https://issues.apache.org/jira/browse/AIRFLOW-5641 Project: Apache Airflow Issue Type: Bug Components: executor-kubernetes Affects Versions: 2.0.0 Reporter: QP Hou Assignee: QP Hou We used to be able to run git sync init container as root user. A recent refactoring of the code base (AIRFLOW-4851) introduced a regression that makes it impossible to run git sync container as root. As a result, we are not able to start any task using kubernetes executor. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] milton0825 commented on a change in pull request #6309: [AIRFLOW-3783][AIRFLOW-5395] Switch to HEADER arg to unload data from Redshift to S3
milton0825 commented on a change in pull request #6309:
[AIRFLOW-3783][AIRFLOW-5395] Switch to HEADER arg to unload data from Redshift
to S3
URL: https://github.com/apache/airflow/pull/6309#discussion_r334209505
##
File path: airflow/operators/redshift_to_s3_operator.py
##
@@ -85,52 +93,16 @@ def __init__(
self.autocommit = autocommit
self.include_header = include_header
-if self.include_header and 'PARALLEL OFF' not in [uo.upper().strip()
for uo in self.unload_options]:
-self.unload_options = list(self.unload_options) + ['PARALLEL OFF',
]
+if self.include_header and 'HEADER' not in [uo.upper().strip() for uo
in self.unload_options]:
+self.unload_options = list(self.unload_options) + ['HEADER', ]
def execute(self, context):
-self.hook = PostgresHook(postgres_conn_id=self.redshift_conn_id)
-self.s3 = S3Hook(aws_conn_id=self.aws_conn_id, verify=self.verify)
-credentials = self.s3.get_credentials()
-unload_options = '\n\t\t\t'.join(self.unload_options)
-
-if self.include_header:
-self.log.info("Retrieving headers from %s.%s...",
- self.schema, self.table)
-
-columns_query = """SELECT column_name
-FROM information_schema.columns
-WHERE table_schema = '{schema}'
-AND table_name = '{table}'
-ORDER BY ordinal_position
-""".format(schema=self.schema,
- table=self.table)
-
-cursor = self.hook.get_conn().cursor()
-cursor.execute(columns_query)
-rows = cursor.fetchall()
-columns = [row[0] for row in rows]
-column_names = ', '.join("{0}".format(c) for c in columns)
-column_headers = ', '.join("\\'{0}\\'".format(c) for c in columns)
-column_castings = ', '.join("CAST({0} AS text) AS {0}".format(c)
-for c in columns)
-
-select_query = """SELECT {column_names} FROM
-(SELECT 2 sort_order, {column_castings}
- FROM {schema}.{table}
-UNION ALL
-SELECT 1 sort_order, {column_headers})
- ORDER BY sort_order"""\
-.format(column_names=column_names,
-column_castings=column_castings,
-column_headers=column_headers,
-schema=self.schema,
-table=self.table)
-else:
-select_query = "SELECT * FROM {schema}.{table}"\
-.format(schema=self.schema,
-table=self.table)
+postgres_hook = PostgresHook(postgres_conn_id=self.redshift_conn_id)
+s3_hook = S3Hook(aws_conn_id=self.aws_conn_id, verify=self.verify)
+credentials = s3_hook.get_credentials()
+unload_options = '\n\t\t\t'.join(self.unload_options)
+select_query = "SELECT * FROM
{schema}.{table}".format(schema=self.schema, table=self.table)
Review comment:
Should we allow user to choose which columns to select?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Resolved] (AIRFLOW-5588) Add celery's architecture diagram
[ https://issues.apache.org/jira/browse/AIRFLOW-5588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor resolved AIRFLOW-5588. Fix Version/s: 1.10.6 Resolution: Fixed > Add celery's architecture diagram > - > > Key: AIRFLOW-5588 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5588 > Project: Apache Airflow > Issue Type: Improvement > Components: documentation >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Priority: Major > Fix For: 1.10.6 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-5628) airflow[sentry] is not being installed and not visible at all
[ https://issues.apache.org/jira/browse/AIRFLOW-5628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor closed AIRFLOW-5628. -- Resolution: Information Provided > airflow[sentry] is not being installed and not visible at all > - > > Key: AIRFLOW-5628 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5628 > Project: Apache Airflow > Issue Type: Task > Components: webserver >Affects Versions: 1.10.5 >Reporter: Aidar Mamytov >Priority: Major > > So, I've followed this page of your docs > [https://airflow.readthedocs.io/en/latest/errors.html]. The doc seems to be > quite straight-forward, expect I couldn't get it work. Configuration setting > is set exactly as described, but nothing is logged at all. > I've also noticed that "sentry" extra is not listed among submodules > [https://airflow.apache.org/installation.html] > Furthermore, I launched python console, imported airflow and *dir*-ed it. > This module [https://github.com/apache/airflow/blob/master/airflow/sentry.py] > is not there. > I couldn't find anyone using your sentry integration after extensive > googling. Is this package still under development and not usable by general > public? If so, why is there a doc page for it? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5628) airflow[sentry] is not being installed and not visible at all
[ https://issues.apache.org/jira/browse/AIRFLOW-5628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949823#comment-16949823 ] Ash Berlin-Taylor commented on AIRFLOW-5628: You are looking at the docs for two different versions: readthedocs.io is for the latest master branch, but airflow.apache.org is for the released versions. This is why the code isn't there yet. We are working on improving our website and will make sure to make it much clearer that readthedocs.io is for unreleased versions. 1.10.6 (out in a week or two) will include the sentry code. Sorry for the confusion! > airflow[sentry] is not being installed and not visible at all > - > > Key: AIRFLOW-5628 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5628 > Project: Apache Airflow > Issue Type: Task > Components: webserver >Affects Versions: 1.10.5 >Reporter: Aidar Mamytov >Priority: Major > > So, I've followed this page of your docs > [https://airflow.readthedocs.io/en/latest/errors.html]. The doc seems to be > quite straight-forward, expect I couldn't get it work. Configuration setting > is set exactly as described, but nothing is logged at all. > I've also noticed that "sentry" extra is not listed among submodules > [https://airflow.apache.org/installation.html] > Furthermore, I launched python console, imported airflow and *dir*-ed it. > This module [https://github.com/apache/airflow/blob/master/airflow/sentry.py] > is not there. > I couldn't find anyone using your sentry integration after extensive > googling. Is this package still under development and not usable by general > public? If so, why is there a doc page for it? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] ashb commented on issue #6288: AIRFLOW-5540 adding rise exception when the _resolve_connection method fails
ashb commented on issue #6288: AIRFLOW-5540 adding rise exception when the _resolve_connection method fails URL: https://github.com/apache/airflow/pull/6288#issuecomment-541243888 I also don't see how this change has anything to do with https://issues.apache.org/jira/browse/AIRFLOW-5540 - am I missing something there? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #6288: AIRFLOW-5540 adding rise exception when the _resolve_connection method fails
ashb commented on a change in pull request #6288: AIRFLOW-5540 adding rise
exception when the _resolve_connection method fails
URL: https://github.com/apache/airflow/pull/6288#discussion_r334192786
##
File path: airflow/contrib/hooks/spark_submit_hook.py
##
@@ -200,6 +201,10 @@ def _resolve_connection(self):
self._conn_id, conn_data['master']
)
+raise AirflowException("Could not load connection string %s,
defaulting to %s",
Review comment:
This message doesn't seem to match the behavoiur. It says "defaulting" but
it's raising an error, so it can't continue.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #6303: AIRFLOW-5126 Read aws_session_token in extra_config of the aws hook
ashb commented on a change in pull request #6303: AIRFLOW-5126 Read
aws_session_token in extra_config of the aws hook
URL: https://github.com/apache/airflow/pull/6303#discussion_r334192142
##
File path: airflow/contrib/hooks/aws_hook.py
##
@@ -132,7 +134,7 @@ def _get_credentials(self, region_name):
role_arn = "arn:aws:iam::{}:role/{}" \
.format(aws_account_id, aws_iam_role)
-if role_arn is not None:
+if role_arn is not None and aws_session_token is None:
Review comment:
I'm not sure this condition makes sense. What if I want to use
key+secret+token to assume a role?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Work started] (AIRFLOW-5615) BaseJob subclasses shouldn't implement own heartbeat logic
[ https://issues.apache.org/jira/browse/AIRFLOW-5615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-5615 started by Ash Berlin-Taylor. -- > BaseJob subclasses shouldn't implement own heartbeat logic > -- > > Key: AIRFLOW-5615 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5615 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 1.10.5 >Reporter: Ash Berlin-Taylor >Assignee: Ash Berlin-Taylor >Priority: Trivial > Fix For: 1.10.6 > > > Both SchedulerJob and LocalTaskJob have their own timers and decide when to > call heartbeat based upon that. > That logic should be removed and live in BaseJob to simplify the code. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] ashb opened a new pull request #6311: [AIRFLOW-5615] Reduce duplicated logic around job heartbeating
ashb opened a new pull request #6311: [AIRFLOW-5615] Reduce duplicated logic
around job heartbeating
URL: https://github.com/apache/airflow/pull/6311
### Jira
- [x] https://issues.apache.org/jira/browse/AIRFLOW-5615
### Description
- [] Both SchedulerJob and LocalTaskJob have their own timers and decide when
to call heartbeat based upon that. THis makes those jobs harder to
follow, so I've moved the logic to BaseJob
### Tests
- [x] Tests added
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-5615) BaseJob subclasses shouldn't implement own heartbeat logic
[
https://issues.apache.org/jira/browse/AIRFLOW-5615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949812#comment-16949812
]
ASF GitHub Bot commented on AIRFLOW-5615:
-
ashb commented on pull request #6311: [AIRFLOW-5615] Reduce duplicated logic
around job heartbeating
URL: https://github.com/apache/airflow/pull/6311
### Jira
- [x] https://issues.apache.org/jira/browse/AIRFLOW-5615
### Description
- [] Both SchedulerJob and LocalTaskJob have their own timers and decide when
to call heartbeat based upon that. THis makes those jobs harder to
follow, so I've moved the logic to BaseJob
### Tests
- [x] Tests added
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> BaseJob subclasses shouldn't implement own heartbeat logic
> --
>
> Key: AIRFLOW-5615
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5615
> Project: Apache Airflow
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.10.5
>Reporter: Ash Berlin-Taylor
>Assignee: Ash Berlin-Taylor
>Priority: Trivial
> Fix For: 1.10.6
>
>
> Both SchedulerJob and LocalTaskJob have their own timers and decide when to
> call heartbeat based upon that.
> That logic should be removed and live in BaseJob to simplify the code.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (AIRFLOW-3611) Simplified development workflow
[ https://issues.apache.org/jira/browse/AIRFLOW-3611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor updated AIRFLOW-3611: --- Component/s: build Fix Version/s: (was: 1.10.6) 1.10.5 Affects Version/s: 1.10.0 > Simplified development workflow > --- > > Key: AIRFLOW-3611 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3611 > Project: Apache Airflow > Issue Type: Improvement > Components: build >Affects Versions: 1.10.0 >Reporter: Gerardo Curiel >Priority: Minor > Fix For: 1.10.5 > > > This issue is meant to track work done to implement > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-7+Simplified+development+workflow -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5000) Remove task instance duplicate end_date and reorder template context
[ https://issues.apache.org/jira/browse/AIRFLOW-5000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor updated AIRFLOW-5000: --- Fix Version/s: (was: 1.10.6) 2.0.0 Not pulling this in to 1.10.x as it is potentially a breaking change. > Remove task instance duplicate end_date and reorder template context > > > Key: AIRFLOW-5000 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5000 > Project: Apache Airflow > Issue Type: Improvement > Components: models >Affects Versions: 1.10.3 >Reporter: zhongjiajie >Assignee: zhongjiajie >Priority: Minor > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow-site] mik-laj opened a new issue #11: Create a list component
mik-laj opened a new issue #11: Create a list component URL: https://github.com/apache/airflow-site/issues/11 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow-site] mik-laj closed issue #20: Create a unordered list component
mik-laj closed issue #20: Create a unordered list component URL: https://github.com/apache/airflow-site/issues/20 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow-site] mik-laj closed issue #19: Create a ordered list component
mik-laj closed issue #19: Create a ordered list component URL: https://github.com/apache/airflow-site/issues/19 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow-site] mik-laj closed issue #11: Create a list component
mik-laj closed issue #11: Create a list component URL: https://github.com/apache/airflow-site/issues/11 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow-site] mik-laj closed issue #10: Create a button component (three variants)
mik-laj closed issue #10: Create a button component (three variants) URL: https://github.com/apache/airflow-site/issues/10 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow-site] mik-laj commented on issue #70: Fix file permissions
mik-laj commented on issue #70: Fix file permissions URL: https://github.com/apache/airflow-site/issues/70#issuecomment-541208771 Done. https://github.com/apache/airflow-site/pull/73 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow-site] mik-laj closed issue #70: Fix file permissions
mik-laj closed issue #70: Fix file permissions URL: https://github.com/apache/airflow-site/issues/70 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[airflow-site] branch aip-11 updated: Add ordered and unordered list components (#72)
This is an automated email from the ASF dual-hosted git repository.
kamilbregula pushed a commit to branch aip-11
in repository https://gitbox.apache.org/repos/asf/airflow-site.git
The following commit(s) were added to refs/heads/aip-11 by this push:
new 8be273a Add ordered and unordered list components (#72)
8be273a is described below
commit 8be273ac6d21a07a81aab4d1527df2ba96661ffe
Author: Kamil Gabryjelski
AuthorDate: Fri Oct 11 21:40:20 2019 +0200
Add ordered and unordered list components (#72)
---
landing-pages/site/assets/scss/_ol-ul.scss | 79 ++
landing-pages/site/assets/scss/main-custom.scss| 1 +
landing-pages/site/content/en/examples/_index.html | 16 +
landing-pages/site/static/icons/tick.svg | 8 +++
4 files changed, 104 insertions(+)
diff --git a/landing-pages/site/assets/scss/_ol-ul.scss
b/landing-pages/site/assets/scss/_ol-ul.scss
new file mode 100644
index 000..f1cf865
--- /dev/null
+++ b/landing-pages/site/assets/scss/_ol-ul.scss
@@ -0,0 +1,79 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+@import "typography";
+@import "colors";
+
+ol.counter-blue, ul.ticks-blue {
+ list-style: none;
+
+ li {
+@include bodytext("xsmall", $brownish-grey);
+position: relative;
+padding-left: 10px;
+
+&::before {
+ position: absolute;
+ border: solid 1px $cerulean-blue;
+ border-radius: 50%;
+}
+ }
+}
+
+ol.counter-blue {
+ counter-reset: custom-counter;
+
+ li {
+counter-increment: custom-counter;
+margin-bottom: 25px;
+
+&::before {
+ @include bodytext("small", $cerulean-blue, 500);
+ $size: 28px;
+ content: counter(custom-counter);
+
+ top: -2px;
+ left: calc(-1 * #{$size});
+ width: $size;
+ height: $size;
+
+ text-align: center;
+ line-height: $size;
+}
+ }
+}
+
+ul.ticks-blue {
+ li {
+margin-bottom: 22px;
+
+&::before {
+ $size: 24px;
+ content: "";
+
+ left: calc(-1 * #{$size});
+ width: $size;
+ height: $size;
+
+ background-position: center;
+ background-repeat: no-repeat;
+ background-image: url("/icons/tick.svg");
+}
+ }
+}
diff --git a/landing-pages/site/assets/scss/main-custom.scss
b/landing-pages/site/assets/scss/main-custom.scss
index 3075707..f887a63 100644
--- a/landing-pages/site/assets/scss/main-custom.scss
+++ b/landing-pages/site/assets/scss/main-custom.scss
@@ -20,3 +20,4 @@
@import "accordion";
@import "typography";
@import "buttons";
+@import "ol-ul";
diff --git a/landing-pages/site/content/en/examples/_index.html
b/landing-pages/site/content/en/examples/_index.html
index 9554c28..a7aa8d9 100644
--- a/landing-pages/site/content/en/examples/_index.html
+++ b/landing-pages/site/content/en/examples/_index.html
@@ -42,3 +42,19 @@ menu:
{{< /accordion >}}
{{< /blocks/section >}}
+
+{{< blocks/section color="white" >}}
+
+
+Item 1
+Item 2
+Item 3
+Item 4
+
+
+Item 1
+Item 2
+Item 3
+Item 4
+
+{{< /blocks/section >}}
diff --git a/landing-pages/site/static/icons/tick.svg
b/landing-pages/site/static/icons/tick.svg
new file mode 100644
index 000..f1c04b8
--- /dev/null
+++ b/landing-pages/site/static/icons/tick.svg
@@ -0,0 +1,8 @@
+http://www.w3.org/2000/svg; width="15.06" height="10.828"
viewBox="0 0 15.06 10.828">
+
+
+.cls-1{fill:#017cee;stroke:#017cee;stroke-width:2px}
+
+
+
+
[GitHub] [airflow-site] mik-laj merged pull request #72: Add ordered and unordered list components
mik-laj merged pull request #72: Add ordered and unordered list components URL: https://github.com/apache/airflow-site/pull/72 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Reopened] (AIRFLOW-5634) Disabled DagModelView fields can be edited (classic UI)
[ https://issues.apache.org/jira/browse/AIRFLOW-5634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor reopened AIRFLOW-5634: > Disabled DagModelView fields can be edited (classic UI) > --- > > Key: AIRFLOW-5634 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5634 > Project: Apache Airflow > Issue Type: Bug > Components: webserver >Affects Versions: 1.10.5 >Reporter: Ash Berlin-Taylor >Priority: Minor > Fix For: 1.10.6 > > > If someone plays silly games and edits the HTML (or crafts a request) to the > DAG model it is possible to edit fields that aren't meant to be edited. > We should stop that. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4222) Add cli autocomplete
[ https://issues.apache.org/jira/browse/AIRFLOW-4222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949742#comment-16949742 ] ASF subversion and git services commented on AIRFLOW-4222: -- Commit 208fbdeb5d000b98fc5ebfa99303972e11fc6d7c in airflow's branch refs/heads/v1-10-stable from Kaxil Naik [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=208fbde ] [AIRFLOW-4222] Add cli autocomplete for bash & zsh (#5789) (cherry picked from commit 44eb89d672784107c317cfe882c45d45d302621f) > Add cli autocomplete > > > Key: AIRFLOW-4222 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4222 > Project: Apache Airflow > Issue Type: Improvement > Components: cli >Affects Versions: 1.10.2 >Reporter: zhongjiajie >Assignee: Kaxil Naik >Priority: Major > Labels: patch > Fix For: 1.10.6 > > > Airflow cli can not autocomplete so far, this ticket will make it happen to > let cli more user-friendliness -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5101) Inconsistent owner value in examples
[ https://issues.apache.org/jira/browse/AIRFLOW-5101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor resolved AIRFLOW-5101. Fix Version/s: 1.10.6 Resolution: Fixed > Inconsistent owner value in examples > > > Key: AIRFLOW-5101 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5101 > Project: Apache Airflow > Issue Type: Improvement > Components: ci, documentation, examples >Affects Versions: 1.10.3 >Reporter: Jonathan de Bruin >Assignee: Jonathan de Bruin >Priority: Trivial > Labels: documentation, starter > Fix For: 1.10.6 > > > By default, the owner of a dag is 'Airflow' (Capital A). Nevertheless, a > large part (but not all) of the examples/tests/perf has 'airflow' as the > owner (lowercase A). I think it is better to use 'Airflow' in examples. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5634) Disabled DagModelView fields can be edited (classic UI)
[ https://issues.apache.org/jira/browse/AIRFLOW-5634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949729#comment-16949729 ] ASF GitHub Bot commented on AIRFLOW-5634: - ashb commented on pull request #6307: [AIRFLOW-5634] Don't allow disabled fields to be edited in DagModelView URL: https://github.com/apache/airflow/pull/6307 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Disabled DagModelView fields can be edited (classic UI) > --- > > Key: AIRFLOW-5634 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5634 > Project: Apache Airflow > Issue Type: Bug > Components: webserver >Affects Versions: 1.10.5 >Reporter: Ash Berlin-Taylor >Priority: Minor > Fix For: 1.10.6 > > > If someone plays silly games and edits the HTML (or crafts a request) to the > DAG model it is possible to edit fields that aren't meant to be edited. > We should stop that. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] ashb commented on issue #6308: [AIRFLOW-5634] Don't allow editing of DagModelView
ashb commented on issue #6308: [AIRFLOW-5634] Don't allow editing of DagModelView URL: https://github.com/apache/airflow/pull/6308#issuecomment-541192431 Similar to #6307 but for the RBAC UI this time (and I took a different approach here) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5292) Allow ECSOperator to tag tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-5292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949733#comment-16949733 ] ASF subversion and git services commented on AIRFLOW-5292: -- Commit aec8d5cf6c13d95d5174c054ea9baf42cbcdb7e9 in airflow's branch refs/heads/v1-10-stable from Philipp Hoffmann [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=aec8d5c ] [AIRFLOW-5292] Allow ECSOperator to tag tasks (#5891) This commit introduces a parameter for the `ECSOperator` in order to pass along tags to any tasks it is running. (cherry picked from commit 2856dee51095f04e5fe0441ced476d52725f6ce2) > Allow ECSOperator to tag tasks > -- > > Key: AIRFLOW-5292 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5292 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib >Affects Versions: 1.10.4 >Reporter: Philipp Hoffmann >Assignee: Philipp Hoffmann >Priority: Major > Fix For: 1.10.6 > > > In its current state the ECSOperator does not allow tagging tasks. Goal of > this ticket is to enable the ECSOperator to tag tasks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5127) Gzip support for CassandraToGoogleCloudStorageOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-5127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949740#comment-16949740 ] ASF subversion and git services commented on AIRFLOW-5127: -- Commit 1abaaa3b35c02559ed93c67eeab82900d3c55103 in airflow's branch refs/heads/v1-10-stable from Tomek [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=1abaaa3 ] [AIRFLOW-5127] Gzip support for CassandraToGoogleCloudStorageOperator (#5738) (cherry picked from commit 5ed0e1d7ed74a725f1dc276180ec0f3fe9d87114) > Gzip support for CassandraToGoogleCloudStorageOperator > -- > > Key: AIRFLOW-5127 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5127 > Project: Apache Airflow > Issue Type: Sub-task > Components: gcp >Affects Versions: 1.10.4 >Reporter: Tomasz Urbaszek >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5218) AWS Batch Operator - status polling too often, esp. for high concurrency
[
https://issues.apache.org/jira/browse/AIRFLOW-5218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949736#comment-16949736
]
ASF subversion and git services commented on AIRFLOW-5218:
--
Commit a198969b5e3acaee67479ebab145d29866607453 in airflow's branch
refs/heads/v1-10-stable from Darren Weber
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=a198969 ]
[AIRFLOW-5218] Less polling of AWS Batch job status (#5825)
https://issues.apache.org/jira/browse/AIRFLOW-5218
- avoid the AWS API throttle limits for highly concurrent tasks
- a small increase in the backoff factor could avoid excessive polling
- random sleep before polling to allow the batch task to spin-up
- the random sleep helps to avoid API throttling
- revise the retry logic slightly to avoid unnecessary pause
when there are no more retries required
(cherry picked from commit fc972fb6c82010f9809a437eb6b9772918a584d2)
> AWS Batch Operator - status polling too often, esp. for high concurrency
>
>
> Key: AIRFLOW-5218
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5218
> Project: Apache Airflow
> Issue Type: Improvement
> Components: aws, contrib
>Affects Versions: 1.10.4
>Reporter: Darren Weber
>Assignee: Darren Weber
>Priority: Major
> Fix For: 2.0.0, 1.10.6
>
>
> The AWS Batch Operator attempts to use a boto3 feature that is not available
> and has not been merged in years, see
> - [https://github.com/boto/botocore/pull/1307]
> - see also [https://github.com/broadinstitute/cromwell/issues/4303]
> This is a curious case of premature optimization. So, in the meantime, this
> means that the fallback is the exponential backoff routine for the status
> checks on the batch job. Unfortunately, when the concurrency of Airflow jobs
> is very high (100's of tasks), this fallback polling hits the AWS Batch API
> too hard and the AWS API throttle throws an error, which fails the Airflow
> task, simply because the status is polled too frequently.
> Check the output from the retry algorithm, e.g. within the first 10 retries,
> the status of an AWS batch job is checked about 10 times at a rate that is
> approx 1 retry/sec. When an Airflow instance is running 10's or 100's of
> concurrent batch jobs, this hits the API too frequently and crashes the
> Airflow task (plus it occupies a worker in too much busy work).
> {code:java}
> In [4]: [1 + pow(retries * 0.1, 2) for retries in range(20)]
> Out[4]:
> [1.0,
> 1.01,
> 1.04,
> 1.09,
> 1.1601,
> 1.25,
> 1.36,
> 1.4902,
> 1.6401,
> 1.81,
> 2.0,
> 2.21,
> 2.4404,
> 2.6904,
> 2.9604,
> 3.25,
> 3.5605,
> 3.8906,
> 4.24,
> 4.61]{code}
> Possible solutions are to introduce an initial sleep (say 60 sec?) right
> after issuing the request, so that the batch job has some time to spin up.
> The job progresses through a through phases before it gets to RUNNING state
> and polling for each phase of that sequence might help. Since batch jobs tend
> to be long-running jobs (rather than near-real time jobs), it might help to
> issue less frequent polls when it's in the RUNNING state. Something on the
> order of 10's seconds might be reasonable for batch jobs? Maybe the class
> could expose a parameter for the rate of polling (or a callable)?
>
> Another option is to use something like the sensor-poke approach, with
> rescheduling, e.g.
> -
> [https://github.com/apache/airflow/blob/master/airflow/sensors/base_sensor_operator.py#L117]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5142) Cassandra GCS Test is flaky
[
https://issues.apache.org/jira/browse/AIRFLOW-5142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949741#comment-16949741
]
ASF subversion and git services commented on AIRFLOW-5142:
--
Commit 4fbd8f4f2e786c54238f838306f9f395eca15426 in airflow's branch
refs/heads/v1-10-stable from Jarek Potiuk
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=4fbd8f4 ]
[AIRFLOW-5142] Fixed flaky cassandra test (#5758)
(cherry picked from commit fc6967b2ada3985dc2c53b9ae90135fc0c1b27f4)
> Cassandra GCS Test is flaky
> ---
>
> Key: AIRFLOW-5142
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5142
> Project: Apache Airflow
> Issue Type: Improvement
> Components: ci
>Affects Versions: 2.0.0
>Reporter: Jarek Potiuk
>Priority: Major
> Fix For: 2.0.0
>
>
> {{We have a flaky test in master:}}
>
> {{== 21)
> FAIL: test_execute
> (tests.contrib.operators.test_cassandra_to_gcs_operator.CassandraToGCSTest)
> --
> Traceback (most recent call last): /usr/local/lib/python3.5/unittest/mock.py
> line 1159 in patched return func(*args, **keywargs)
> tests/contrib/operators/test_cassandra_to_gcs_operator.py line 53 in
> test_execute test_bucket, schema, TMP_FILE_NAME, "application/json", gzip
> /usr/local/lib/python3.5/unittest/mock.py line 794 in assert_called_with
> raise AssertionError(_error_message()) from cause AssertionError: Expected
> call: upload('test-bucket', 'schema.json', 'temp-file', 'application/json',
> True) Actual call: upload('test-bucket', 'data.json', 'temp-file',
> 'application/json', True)}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-3705) PostgresHook's get_conn method calls incorrect attribute by default
[
https://issues.apache.org/jira/browse/AIRFLOW-3705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949734#comment-16949734
]
ASF subversion and git services commented on AIRFLOW-3705:
--
Commit f98ce4a122e959035d6d91b24dc253fab66f4d0c in airflow's branch
refs/heads/v1-10-stable from Cooper Gillan
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=f98ce4a ]
[AIRFLOW-3705] Fix PostgresHook get_conn to use conn_name_attr (#5841)
Update PostgresHook's get_conn method to directly call the specified
conn_name_attr rather that always using self.postgres_conn_id.
Currently subclassing PostgresHook requires overriding the
postgres_conn_id attribute in order to establish a separate connection.
Add an additional unit test for this case checking that the subclassed
PostgresHook's get_conn calls the correct arguments and that the hook
calls the correction connection_id in get_connection.
(cherry picked from commit f823a66001c9f3682153acbf27bca137d0f4a7ec)
> PostgresHook's get_conn method calls incorrect attribute by default
>
>
> Key: AIRFLOW-3705
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3705
> Project: Apache Airflow
> Issue Type: Bug
> Components: database
>Affects Versions: 1.10.2
>Reporter: Cooper Gillan
>Assignee: Cooper Gillan
>Priority: Minor
> Fix For: 1.10.6
>
>
> Sub-classing {{PostgresHook}} to specify separate connections is currently
> not possible due to [{{get_conn}} calling {{self.postgres_conn_id}}
> directly|https://github.com/apache/airflow/blob/c030729dcbe35a2a4869e46b57a761c8376ede8d/airflow/hooks/postgres_hook.py#L46].
> It should be possible to set different values for
> [{{self.conn_name_attr}}|https://github.com/apache/airflow/blob/c030729dcbe35a2a4869e46b57a761c8376ede8d/airflow/hooks/postgres_hook.py#L37],
> in sub-classes of the hook.
> For example, if sub-classing {{PostgresHook}} to set up separate named
> connections to multiple Postgres instances, the {{postgres_conn_id}} of the
> sub-class must be set specifically. This is true despite the fact that
> {{conn_name_attr}} is set up in a way that should allow any connection to be
> specified.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5634) Disabled DagModelView fields can be edited (classic UI)
[ https://issues.apache.org/jira/browse/AIRFLOW-5634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor resolved AIRFLOW-5634. Resolution: Fixed > Disabled DagModelView fields can be edited (classic UI) > --- > > Key: AIRFLOW-5634 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5634 > Project: Apache Airflow > Issue Type: Bug > Components: webserver >Affects Versions: 1.10.5 >Reporter: Ash Berlin-Taylor >Priority: Minor > Fix For: 1.10.6 > > > If someone plays silly games and edits the HTML (or crafts a request) to the > DAG model it is possible to edit fields that aren't meant to be edited. > We should stop that. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5581) KubernetesJobWatcher dies prematurely with num_runs or run_duration
[
https://issues.apache.org/jira/browse/AIRFLOW-5581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949731#comment-16949731
]
ASF subversion and git services commented on AIRFLOW-5581:
--
Commit 39f20a0b3168e045c5ae1a277981ddff03c3ae93 in airflow's branch
refs/heads/v1-10-stable from Kiran Pathak
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=39f20a0 ]
[AIRFLOW-5581] Cleanly shutdown KubernetesJobWatcher for safe Scheduler
shutdown on SIGTERM (#6237)
(cherry picked from commit 12f916a432e7937f6f72d1b556196e863920f334)
> KubernetesJobWatcher dies prematurely with num_runs or run_duration
> ---
>
> Key: AIRFLOW-5581
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5581
> Project: Apache Airflow
> Issue Type: Bug
> Components: contrib, executor-kubernetes, executors, scheduler
>Affects Versions: 1.10.5
>Reporter: Kiran
>Assignee: Kiran
>Priority: Major
> Fix For: 1.10.6
>
>
> If run_duration or num_runs is enabled, KubernetesJobWatcher dies prematurely
> causing below error:
> [2019-10-01 19:40:35,323] \{kubernetes_executor.py:327} ERROR - Unknown error
> in KubernetesJobWatcher. Failing
> Traceback (most recent call last):
> File
> "/home/vccorp/.local/lib/python3.6/site-packages/airflow/contrib/executors/kubernetes_executor.py",
> line 325, in run
> self.worker_uuid, self.kube_config)
> File
> "/home/vccorp/.local/lib/python3.6/site-packages/airflow/contrib/executors/kubernetes_executor.py",
> line 359, in _run
> task.metadata.resource_version
> File
> "/home/vccorp/.local/lib/python3.6/site-packages/airflow/contrib/executors/kubernetes_executor.py",
> line 391, in process_status
> self.watcher_queue.put((pod_id, None, labels, resource_version))
> File "", line 2, in put
> File "/usr/lib64/python3.6/multiprocessing/managers.py", line 756, in
> _callmethod
> conn.send((self._id, methodname, args, kwds))
> File "/usr/lib64/python3.6/multiprocessing/connection.py", line 206, in send
> self._send_bytes(_ForkingPickler.dumps(obj))
> File "/usr/lib64/python3.6/multiprocessing/connection.py", line 404, in
> _send_bytes
> self._send(header + buf)
> File "/usr/lib64/python3.6/multiprocessing/connection.py", line 368, in _send
> n = write(self._handle, buf)
> BrokenPipeError: [Errno 32] Broken pipe
> Process KubernetesJobWatcher-3:
> Traceback (most recent call last):
> File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in
> _bootstrap
> self.run()
> File
> "/home/vccorp/.local/lib/python3.6/site-packages/airflow/contrib/executors/kubernetes_executor.py",
> line 325, in run
> self.worker_uuid, self.kube_config)
> File
> "/home/vccorp/.local/lib/python3.6/site-packages/airflow/contrib/executors/kubernetes_executor.py",
> line 359, in _run
> task.metadata.resource_version
> File
> "/home/vccorp/.local/lib/python3.6/site-packages/airflow/contrib/executors/kubernetes_executor.py",
> line 391, in process_status
> self.watcher_queue.put((pod_id, None, labels, resource_version))
> File "", line 2, in put
> File "/usr/lib64/python3.6/multiprocessing/managers.py", line 756, in
> _callmethod
> conn.send((self._id, methodname, args, kwds))
> File "/usr/lib64/python3.6/multiprocessing/connection.py", line 206, in send
> self._send_bytes(_ForkingPickler.dumps(obj))
> File "/usr/lib64/python3.6/multiprocessing/connection.py", line 404, in
> _send_bytes
> self._send(header + buf)
> File "/usr/lib64/python3.6/multiprocessing/connection.py", line 368, in _send
> n = write(self._handle, buf)
> BrokenPipeError: [Errno 32] Broken pipe
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5124) Gzip support for S3ToGoogleCloudStorageOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-5124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949737#comment-16949737 ] ASF subversion and git services commented on AIRFLOW-5124: -- Commit afa5d820452a4382ba17521e7fc41193814a1724 in airflow's branch refs/heads/v1-10-stable from Kamil Breguła [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=afa5d82 ] [AIRFLOW-5124] Add gzip support for S3ToGoogleCloudStorageOperator (#5736) (cherry picked from commit 60a032f4b829eb41b84c907ff663560d50284989) > Gzip support for S3ToGoogleCloudStorageOperator > --- > > Key: AIRFLOW-5124 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5124 > Project: Apache Airflow > Issue Type: Sub-task > Components: gcp, operators >Affects Versions: 1.10.3 >Reporter: Kamil Bregula >Priority: Major > Fix For: 1.10.6 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4473) Add papermill operator
[ https://issues.apache.org/jira/browse/AIRFLOW-4473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949732#comment-16949732 ] ASF subversion and git services commented on AIRFLOW-4473: -- Commit e6ccf975e7498268a94c2927852ebaee26ce9bc5 in airflow's branch refs/heads/v1-10-stable from Kamil Breguła [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=e6ccf97 ] [AIRFLOW-4473] Move Papermill guide (#5371) (cherry picked from commit 6cf722182a8ce0afaab058df45137aa1b1a4ec79) > Add papermill operator > -- > > Key: AIRFLOW-4473 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4473 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Reporter: Bolke de Bruin >Priority: Major > Fix For: 1.10.6 > > > Papermill is a way to parameterize and productize python notebooks -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5125) Gzip support for AdlsToGoogleCloudStorageOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-5125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949739#comment-16949739 ] ASF subversion and git services commented on AIRFLOW-5125: -- Commit eb7060d6669c6fdedac16145b8f2ff9d990f1900 in airflow's branch refs/heads/v1-10-stable from Kamil Breguła [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=eb7060d ] [AIRFLOW-5125] Add gzip support for AdlsToGoogleCloudStorageOperator (#5737) (cherry picked from commit 147346192b7fde0fdf35f08695f4ecd29f210e5d) > Gzip support for AdlsToGoogleCloudStorageOperator > - > > Key: AIRFLOW-5125 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5125 > Project: Apache Airflow > Issue Type: Sub-task > Components: gcp, operators >Affects Versions: 1.10.3 >Reporter: Kamil Bregula >Priority: Major > Fix For: 1.10.6 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5218) AWS Batch Operator - status polling too often, esp. for high concurrency
[
https://issues.apache.org/jira/browse/AIRFLOW-5218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949735#comment-16949735
]
ASF subversion and git services commented on AIRFLOW-5218:
--
Commit a198969b5e3acaee67479ebab145d29866607453 in airflow's branch
refs/heads/v1-10-stable from Darren Weber
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=a198969 ]
[AIRFLOW-5218] Less polling of AWS Batch job status (#5825)
https://issues.apache.org/jira/browse/AIRFLOW-5218
- avoid the AWS API throttle limits for highly concurrent tasks
- a small increase in the backoff factor could avoid excessive polling
- random sleep before polling to allow the batch task to spin-up
- the random sleep helps to avoid API throttling
- revise the retry logic slightly to avoid unnecessary pause
when there are no more retries required
(cherry picked from commit fc972fb6c82010f9809a437eb6b9772918a584d2)
> AWS Batch Operator - status polling too often, esp. for high concurrency
>
>
> Key: AIRFLOW-5218
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5218
> Project: Apache Airflow
> Issue Type: Improvement
> Components: aws, contrib
>Affects Versions: 1.10.4
>Reporter: Darren Weber
>Assignee: Darren Weber
>Priority: Major
> Fix For: 2.0.0, 1.10.6
>
>
> The AWS Batch Operator attempts to use a boto3 feature that is not available
> and has not been merged in years, see
> - [https://github.com/boto/botocore/pull/1307]
> - see also [https://github.com/broadinstitute/cromwell/issues/4303]
> This is a curious case of premature optimization. So, in the meantime, this
> means that the fallback is the exponential backoff routine for the status
> checks on the batch job. Unfortunately, when the concurrency of Airflow jobs
> is very high (100's of tasks), this fallback polling hits the AWS Batch API
> too hard and the AWS API throttle throws an error, which fails the Airflow
> task, simply because the status is polled too frequently.
> Check the output from the retry algorithm, e.g. within the first 10 retries,
> the status of an AWS batch job is checked about 10 times at a rate that is
> approx 1 retry/sec. When an Airflow instance is running 10's or 100's of
> concurrent batch jobs, this hits the API too frequently and crashes the
> Airflow task (plus it occupies a worker in too much busy work).
> {code:java}
> In [4]: [1 + pow(retries * 0.1, 2) for retries in range(20)]
> Out[4]:
> [1.0,
> 1.01,
> 1.04,
> 1.09,
> 1.1601,
> 1.25,
> 1.36,
> 1.4902,
> 1.6401,
> 1.81,
> 2.0,
> 2.21,
> 2.4404,
> 2.6904,
> 2.9604,
> 3.25,
> 3.5605,
> 3.8906,
> 4.24,
> 4.61]{code}
> Possible solutions are to introduce an initial sleep (say 60 sec?) right
> after issuing the request, so that the batch job has some time to spin up.
> The job progresses through a through phases before it gets to RUNNING state
> and polling for each phase of that sequence might help. Since batch jobs tend
> to be long-running jobs (rather than near-real time jobs), it might help to
> issue less frequent polls when it's in the RUNNING state. Something on the
> order of 10's seconds might be reasonable for batch jobs? Maybe the class
> could expose a parameter for the rate of polling (or a callable)?
>
> Another option is to use something like the sensor-poke approach, with
> rescheduling, e.g.
> -
> [https://github.com/apache/airflow/blob/master/airflow/sensors/base_sensor_operator.py#L117]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5634) Disabled DagModelView fields can be edited (classic UI)
[ https://issues.apache.org/jira/browse/AIRFLOW-5634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949730#comment-16949730 ] ASF subversion and git services commented on AIRFLOW-5634: -- Commit 2599cda2b5450090f5e561c50a34a0bd7ff92533 in airflow's branch refs/heads/v1-10-stable from Ash Berlin-Taylor [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=2599cda ] [AIRFLOW-5634] Don't allow disabled fields to be edited in DagModelView (#6307) If someone plays silly games and edits the HTML (or crafts a request) to the DAG model it is possible to edit fields that aren't meant to be edited, leading to odd to debug behaviour > Disabled DagModelView fields can be edited (classic UI) > --- > > Key: AIRFLOW-5634 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5634 > Project: Apache Airflow > Issue Type: Bug > Components: webserver >Affects Versions: 1.10.5 >Reporter: Ash Berlin-Taylor >Priority: Minor > Fix For: 1.10.6 > > > If someone plays silly games and edits the HTML (or crafts a request) to the > DAG model it is possible to edit fields that aren't meant to be edited. > We should stop that. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] ashb merged pull request #6307: [AIRFLOW-5634] Don't allow disabled fields to be edited in DagModelView
ashb merged pull request #6307: [AIRFLOW-5634] Don't allow disabled fields to be edited in DagModelView URL: https://github.com/apache/airflow/pull/6307 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on issue #6309: [AIRFLOW-3783][AIRFLOW-5395] Switch to HEADER arg to unload data from Redshift to S3
ashb commented on issue #6309: [AIRFLOW-3783][AIRFLOW-5395] Switch to HEADER arg to unload data from Redshift to S3 URL: https://github.com/apache/airflow/pull/6309#issuecomment-541187240 This fixes AIRFLOW-5395 by simply not having to mention the columns anymore right? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow-site] mik-laj merged pull request #71: Add button components
mik-laj merged pull request #71: Add button components URL: https://github.com/apache/airflow-site/pull/71 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[airflow-site] branch aip-11 updated: Add button components (#71)
This is an automated email from the ASF dual-hosted git repository.
kamilbregula pushed a commit to branch aip-11
in repository https://gitbox.apache.org/repos/asf/airflow-site.git
The following commit(s) were added to refs/heads/aip-11 by this push:
new 008eb06 Add button components (#71)
008eb06 is described below
commit 008eb06c5e6897cba306deb7f86294d1cc465e19
Author: Kamil Gabryjelski
AuthorDate: Fri Oct 11 19:46:29 2019 +0200
Add button components (#71)
---
.../site/assets/icons/accordion-arrow.svg | 10 +--
landing-pages/site/assets/icons/github.svg | 4 +
landing-pages/site/assets/scss/_buttons.scss | 86 ++
landing-pages/site/assets/scss/main-custom.scss| 1 +
landing-pages/site/content/en/examples/_index.html | 2 +-
landing-pages/site/layouts/examples/baseof.html| 18 +
landing-pages/site/layouts/examples/list.html | 10 +++
.../layouts/partials/buttons/button-filled.html| 20 +
.../layouts/partials/buttons/button-hollow.html| 20 +
.../layouts/partials/buttons/button-with-icon.html | 24 ++
10 files changed, 187 insertions(+), 8 deletions(-)
diff --git a/landing-pages/site/assets/icons/accordion-arrow.svg
b/landing-pages/site/assets/icons/accordion-arrow.svg
index c57d756..441e6fd 100644
--- a/landing-pages/site/assets/icons/accordion-arrow.svg
+++ b/landing-pages/site/assets/icons/accordion-arrow.svg
@@ -1,8 +1,4 @@
-http://www.w3.org/2000/svg; width="18.904" height="11.451"
viewBox="0 0 18.904 11.451">
-
-
-.cls-1{fill:#017cee;stroke:#017cee;stroke-width:2px}
-
-
-
+http://www.w3.org/2000/svg; width="18.904" height="11.451">
+
diff --git a/landing-pages/site/assets/icons/github.svg
b/landing-pages/site/assets/icons/github.svg
new file mode 100644
index 000..08cc33e
--- /dev/null
+++ b/landing-pages/site/assets/icons/github.svg
@@ -0,0 +1,4 @@
+http://www.w3.org/2000/svg; width="30.76" height="30">
+
+
diff --git a/landing-pages/site/assets/scss/_buttons.scss
b/landing-pages/site/assets/scss/_buttons.scss
new file mode 100644
index 000..7276474
--- /dev/null
+++ b/landing-pages/site/assets/scss/_buttons.scss
@@ -0,0 +1,86 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+@import "colors";
+
+button {
+ cursor: pointer;
+ border: 1px solid;
+ border-radius: 5px;
+ padding: 9px 29px;
+ transition: all ease-out 0.2s;
+
+ &.btn-filled {
+border-color: $cerulean-blue;
+background-color: $cerulean-blue;
+
+&:hover {
+ border-color: $bright-sky-blue;
+ background-color: $bright-sky-blue;
+}
+ }
+
+ &.btn-with-icon {
+$icon-height: 30px;
+
+padding: 14px 20px;
+
+svg {
+ height: $icon-height;
+ width: auto;
+ padding-right: 15px;
+}
+
+span {
+ display: inline-block;
+ line-height: $icon-height;
+ vertical-align: middle;
+}
+ }
+
+ &.btn-hollow {
+&.btn-blue {
+ color: $cerulean-blue;
+ border-color: $cerulean-blue;
+ background-color: transparent;
+
+ &:hover {
+color: $white;
+background-color: $cerulean-blue;
+ }
+}
+
+&.btn-brown {
+ border-color: $very-light-pink;
+
+ &:hover {
+background-color: $greyish-brown;
+border-color: $greyish-brown;
+
+span {
+ color: $white;
+}
+svg {
+ path {
+fill: $white;
+ }
+}
+ }
+}
+ }
+}
diff --git a/landing-pages/site/assets/scss/main-custom.scss
b/landing-pages/site/assets/scss/main-custom.scss
index ca88a7d..3075707 100644
--- a/landing-pages/site/assets/scss/main-custom.scss
+++ b/landing-pages/site/assets/scss/main-custom.scss
@@ -19,3 +19,4 @@
@import "accordion";
@import "typography";
+@import "buttons";
diff --git a/landing-pages/site/content/en/examples/_index.html
b/landing-pages/site/content/en/examples/_index.html
index a6bf6a6..9554c28 100644
--- a/landing-pages/site/content/en/examples/_index.html
+++ b/landing-pages/site/content/en/examples/_index.html
@@ -26,9 +26,9 @@ menu:
This is a shamrock monotext
This is a vermillion
monotext
-
{{< /blocks/section
[GitHub] [airflow] TobKed commented on a change in pull request #6306: [AIRFLOW-5632] Rename ComputeEngine operators
TobKed commented on a change in pull request #6306: [AIRFLOW-5632] Rename ComputeEngine operators URL: https://github.com/apache/airflow/pull/6306#discussion_r334100151 ## File path: airflow/contrib/hooks/gcp_compute_hook.py ## @@ -16,14 +16,30 @@ # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. -"""This module is deprecated. Please use `airflow.gcp.hooks.compute`.""" +"""This module is deprecated. Please use `airflow.gcp.hooks.compute.ComputeEngineHook`.""" Review comment: I think information here shouldn't contain specific hook name, just path to module (as it was before). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] TobKed commented on a change in pull request #6306: [AIRFLOW-5632] Rename ComputeEngine operators
TobKed commented on a change in pull request #6306: [AIRFLOW-5632] Rename ComputeEngine operators URL: https://github.com/apache/airflow/pull/6306#discussion_r334098944 ## File path: UPDATING.md ## @@ -163,7 +163,7 @@ The following table shows changes in import paths. |airflow.contrib.hooks.datastore_hook.DatastoreHook |airflow.gcp.hooks.datastore.DatastoreHook | |airflow.contrib.hooks.gcp_bigtable_hook.BigtableHook |airflow.gcp.hooks.bigtable.BigtableHook | |airflow.contrib.hooks.gcp_cloud_build_hook.CloudBuildHook |airflow.gcp.hooks.cloud_build.CloudBuildHook | -|airflow.contrib.hooks.gcp_compute_hook.GceHook |airflow.gcp.hooks.compute.GceHook | +|airflow.contrib.hooks.gcp_compute_hook.GceHook |airflow.gcp.hooks.compute.ComputeEngineOperator | Review comment: Shouldn't be `ComputeEngineHook ` instead of `ComputeEngineOperator`? I am not sure is it allowed to change some older informations. Maybe it should be done in separate information. WDY @nuclearpinguin ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] TobKed commented on a change in pull request #6306: [AIRFLOW-5632] Rename ComputeEngine operators
TobKed commented on a change in pull request #6306: [AIRFLOW-5632] Rename ComputeEngine operators URL: https://github.com/apache/airflow/pull/6306#discussion_r334100277 ## File path: airflow/contrib/hooks/gcp_compute_hook.py ## @@ -16,14 +16,30 @@ # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. -"""This module is deprecated. Please use `airflow.gcp.hooks.compute`.""" +"""This module is deprecated. Please use `airflow.gcp.hooks.compute.ComputeEngineHook`.""" import warnings # pylint: disable=unused-import -from airflow.gcp.hooks.compute import GceHook, GceOperationStatus # noqa +from airflow.gcp.hooks.compute import ComputeEngineHook # noqa warnings.warn( -"This module is deprecated. Please use `airflow.gcp.hooks.compute`.", +"This module is deprecated. airflow.gcp.hooks.compute.ComputeEngineHook`", Review comment: Same as above. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6309: [AIRFLOW-3783][AIRFLOW-5395] Switch to HEADER arg to unload data from Redshift to S3
codecov-io edited a comment on issue #6309: [AIRFLOW-3783][AIRFLOW-5395] Switch to HEADER arg to unload data from Redshift to S3 URL: https://github.com/apache/airflow/pull/6309#issuecomment-541151113 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6309?src=pr=h1) Report > Merging [#6309](https://codecov.io/gh/apache/airflow/pull/6309?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/c0d98a72394b68df27db4d1cab4a94021980e0fb?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `90%`. [](https://codecov.io/gh/apache/airflow/pull/6309?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6309 +/- ## == + Coverage 80.34% 80.35% +<.01% == Files 616 616 Lines 3573335723 -10 == - Hits2871028704 -6 + Misses 7023 7019 -4 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6309?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/operators/redshift\_to\_s3\_operator.py](https://codecov.io/gh/apache/airflow/pull/6309/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcmVkc2hpZnRfdG9fczNfb3BlcmF0b3IucHk=) | `97.14% <90%> (+1.58%)` | :arrow_up: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6309/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.77% <0%> (+0.5%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6309?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6309?src=pr=footer). Last update [c0d98a7...08b3360](https://codecov.io/gh/apache/airflow/pull/6309?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6309: [AIRFLOW-3783][AIRFLOW-5395] Switch to HEADER arg to unload data from Redshift to S3
codecov-io commented on issue #6309: [AIRFLOW-3783][AIRFLOW-5395] Switch to HEADER arg to unload data from Redshift to S3 URL: https://github.com/apache/airflow/pull/6309#issuecomment-541151113 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6309?src=pr=h1) Report > Merging [#6309](https://codecov.io/gh/apache/airflow/pull/6309?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/c0d98a72394b68df27db4d1cab4a94021980e0fb?src=pr=desc) will **decrease** coverage by `0.25%`. > The diff coverage is `90%`. [](https://codecov.io/gh/apache/airflow/pull/6309?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6309 +/- ## == - Coverage 80.34% 80.08% -0.26% == Files 616 616 Lines 3573335723 -10 == - Hits2871028610 -100 - Misses 7023 7113 +90 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6309?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/operators/redshift\_to\_s3\_operator.py](https://codecov.io/gh/apache/airflow/pull/6309/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcmVkc2hpZnRfdG9fczNfb3BlcmF0b3IucHk=) | `97.14% <90%> (+1.58%)` | :arrow_up: | | [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6309/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6309/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6309/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `86.44% <0%> (-6.78%)` | :arrow_down: | | [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6309/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `85% <0%> (-5%)` | :arrow_down: | | [airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/airflow/pull/6309/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5) | `75.82% <0%> (-1.79%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6309/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `89.9% <0%> (-1.53%)` | :arrow_down: | | [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/6309/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `63.88% <0%> (-1.12%)` | :arrow_down: | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/6309/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `87.28% <0%> (-0.85%)` | :arrow_down: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6309/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.77% <0%> (+0.5%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6309?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6309?src=pr=footer). Last update [c0d98a7...08b3360](https://codecov.io/gh/apache/airflow/pull/6309?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6309: [AIRFLOW-3783][AIRFLOW-5395] Switch to HEADER arg to unload data from Redshift to S3
codecov-io edited a comment on issue #6309: [AIRFLOW-3783][AIRFLOW-5395] Switch to HEADER arg to unload data from Redshift to S3 URL: https://github.com/apache/airflow/pull/6309#issuecomment-541151113 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6309?src=pr=h1) Report > Merging [#6309](https://codecov.io/gh/apache/airflow/pull/6309?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/c0d98a72394b68df27db4d1cab4a94021980e0fb?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `90%`. [](https://codecov.io/gh/apache/airflow/pull/6309?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6309 +/- ## == + Coverage 80.34% 80.35% +<.01% == Files 616 616 Lines 3573335723 -10 == - Hits2871028704 -6 + Misses 7023 7019 -4 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6309?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/operators/redshift\_to\_s3\_operator.py](https://codecov.io/gh/apache/airflow/pull/6309/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcmVkc2hpZnRfdG9fczNfb3BlcmF0b3IucHk=) | `97.14% <90%> (+1.58%)` | :arrow_up: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6309/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.77% <0%> (+0.5%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6309?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6309?src=pr=footer). Last update [c0d98a7...08b3360](https://codecov.io/gh/apache/airflow/pull/6309?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6309: [AIRFLOW-3783][AIRFLOW-5395] Switch to HEADER arg to unload data from Redshift to S3
codecov-io edited a comment on issue #6309: [AIRFLOW-3783][AIRFLOW-5395] Switch to HEADER arg to unload data from Redshift to S3 URL: https://github.com/apache/airflow/pull/6309#issuecomment-541151113 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6309?src=pr=h1) Report > Merging [#6309](https://codecov.io/gh/apache/airflow/pull/6309?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/c0d98a72394b68df27db4d1cab4a94021980e0fb?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `90%`. [](https://codecov.io/gh/apache/airflow/pull/6309?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6309 +/- ## == + Coverage 80.34% 80.35% +<.01% == Files 616 616 Lines 3573335723 -10 == - Hits2871028704 -6 + Misses 7023 7019 -4 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6309?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/operators/redshift\_to\_s3\_operator.py](https://codecov.io/gh/apache/airflow/pull/6309/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcmVkc2hpZnRfdG9fczNfb3BlcmF0b3IucHk=) | `97.14% <90%> (+1.58%)` | :arrow_up: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6309/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.77% <0%> (+0.5%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6309?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6309?src=pr=footer). Last update [c0d98a7...08b3360](https://codecov.io/gh/apache/airflow/pull/6309?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6309: [AIRFLOW-3783][AIRFLOW-5395] Switch to HEADER arg to unload data from Redshift to S3
codecov-io edited a comment on issue #6309: [AIRFLOW-3783][AIRFLOW-5395] Switch to HEADER arg to unload data from Redshift to S3 URL: https://github.com/apache/airflow/pull/6309#issuecomment-541151113 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6309?src=pr=h1) Report > Merging [#6309](https://codecov.io/gh/apache/airflow/pull/6309?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/c0d98a72394b68df27db4d1cab4a94021980e0fb?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `90%`. [](https://codecov.io/gh/apache/airflow/pull/6309?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6309 +/- ## == + Coverage 80.34% 80.35% +<.01% == Files 616 616 Lines 3573335723 -10 == - Hits2871028704 -6 + Misses 7023 7019 -4 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6309?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/operators/redshift\_to\_s3\_operator.py](https://codecov.io/gh/apache/airflow/pull/6309/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcmVkc2hpZnRfdG9fczNfb3BlcmF0b3IucHk=) | `97.14% <90%> (+1.58%)` | :arrow_up: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6309/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.77% <0%> (+0.5%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6309?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6309?src=pr=footer). Last update [c0d98a7...08b3360](https://codecov.io/gh/apache/airflow/pull/6309?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] TobKed commented on a change in pull request #6297: [AIRFLOW-5224] Add encoding parameter to GoogleCloudStorageToBigQuery…
TobKed commented on a change in pull request #6297: [AIRFLOW-5224] Add encoding parameter to GoogleCloudStorageToBigQuery… URL: https://github.com/apache/airflow/pull/6297#discussion_r334079885 ## File path: airflow/gcp/hooks/bigquery.py ## @@ -1134,6 +1141,10 @@ def run_load(self, # pylint: disable=too-many-locals,too-many-arguments,invalid records, an invalid error is returned in the job result. Only applicable when soure_format is CSV. :type allow_jagged_rows: bool +:param encoding: The character encoding of the data. See: Review comment: Sure, good point. I've changed it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[airflow-site] branch aip-11 updated: Add more linters (#75)
This is an automated email from the ASF dual-hosted git repository.
kamilbregula pushed a commit to branch aip-11
in repository https://gitbox.apache.org/repos/asf/airflow-site.git
The following commit(s) were added to refs/heads/aip-11 by this push:
new 36f07ac Add more linters (#75)
36f07ac is described below
commit 36f07accc45cbf564030d421da05ddfb51c849fa
Author: Kamil Breguła
AuthorDate: Fri Oct 11 18:35:02 2019 +0200
Add more linters (#75)
---
.pre-commit-config.yaml| 15 +++
landing-pages/site/assets/scss/_accordion.scss | 2 --
landing-pages/site/assets/scss/main-custom.scss| 2 +-
landing-pages/site/config.toml | 13 ++---
landing-pages/site/content/en/blog/_index.md | 1 -
landing-pages/site/content/en/blog/news/_index.md | 2 --
.../site/content/en/blog/news/first-post/index.md | 2 --
landing-pages/site/content/en/blog/releases/_index.md | 2 --
landing-pages/site/content/en/docs/Concepts/_index.md | 4 +---
.../content/en/docs/Contribution guidelines/_index.md | 18 --
landing-pages/site/content/en/docs/Examples/_index.md | 2 --
.../site/content/en/docs/Getting started/_index.md | 1 -
.../content/en/docs/Getting started/example-page.md| 2 +-
landing-pages/site/content/en/docs/Overview/_index.md | 5 ++---
.../content/en/docs/Reference/parameter-reference.md | 2 +-
.../site/content/en/docs/Tasks/Ponycopters/_index.md | 1 -
.../docs/Tasks/Ponycopters/configuring-ponycopters.md | 2 +-
.../en/docs/Tasks/Ponycopters/launching-ponycopters.md | 2 +-
landing-pages/site/content/en/docs/Tasks/_index.md | 3 +--
landing-pages/site/content/en/docs/Tasks/beds.md | 2 +-
landing-pages/site/content/en/docs/Tasks/porridge.md | 2 +-
landing-pages/site/content/en/docs/Tasks/task.md | 2 +-
landing-pages/site/content/en/docs/Tutorials/_index.md | 1 -
.../site/content/en/docs/Tutorials/multi-bear.md | 2 +-
.../site/content/en/docs/Tutorials/tutorial2.md| 2 +-
landing-pages/site/content/en/docs/_index.md | 6 ++
landing-pages/site/content/en/search.md| 1 -
landing-pages/site/content/no/_index.html | 2 --
landing-pages/site/content/no/docs/_index.md | 2 --
.../site/content/no/docs/api-reference/_index.md | 2 --
.../content/no/docs/api-reference/examples/_index.md | 2 --
landing-pages/site/content/no/docs/big-data/_index.md | 2 --
.../site/content/no/docs/big-data/examples/_index.md | 2 --
.../site/content/no/docs/big-data/tutorials/_index.md | 2 --
.../site/content/no/docs/cloud-computing/_index.md | 2 --
.../content/no/docs/cloud-computing/examples/_index.md | 2 --
.../no/docs/cloud-computing/tutorials/_index.md| 2 --
.../site/content/no/docs/content-management/_index.md | 2 --
.../no/docs/content-management/tutorials/_index.md | 2 --
.../site/content/no/docs/cross-platform/_index.md | 2 --
.../content/no/docs/cross-platform/examples/_index.md | 2 --
.../content/no/docs/cross-platform/tutorials/_index.md | 2 --
landing-pages/site/content/no/search.md| 1 -
43 files changed, 45 insertions(+), 85 deletions(-)
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index ceeb03e..fac8e0a 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -55,3 +55,18 @@ repos:
language: system
files: \.js$
pass_filenames: false
+ - repo: https://github.com/pre-commit/pre-commit-hooks
+ rev: v2.3.0
+ hooks:
+- id: trailing-whitespace
+- id: check-executables-have-shebangs
+- id: check-json
+- id: check-merge-conflict
+- id: check-toml
+- id: check-yaml
+- id: detect-private-key
+- id: end-of-file-fixer
+- id: mixed-line-ending
+- id: requirements-txt-fixer
+- id: trailing-whitespace
+ args: [--markdown-linebreak-ext=md]
diff --git a/landing-pages/site/assets/scss/_accordion.scss
b/landing-pages/site/assets/scss/_accordion.scss
index 33f9fdd..2c9af12 100644
--- a/landing-pages/site/assets/scss/_accordion.scss
+++ b/landing-pages/site/assets/scss/_accordion.scss
@@ -76,5 +76,3 @@ details.accordion {
margin-top: 30px;
}
}
-
-
diff --git a/landing-pages/site/assets/scss/main-custom.scss
b/landing-pages/site/assets/scss/main-custom.scss
index abf9037..ca88a7d 100644
--- a/landing-pages/site/assets/scss/main-custom.scss
+++ b/landing-pages/site/assets/scss/main-custom.scss
@@ -16,6 +16,6 @@
* specific language governing permissions and limitations
* under the License.
*/
-
+
@import "accordion";
@import "typography";
diff --git a/landing-pages/site/config.toml b/landing-pages/site/config.toml
index 62d9cc0..c491265 100644
--- a/landing-pages/site/config.toml
+++ b/landing-pages/site/config.toml
@@
[GitHub] [airflow-site] mik-laj merged pull request #75: Add more linters
mik-laj merged pull request #75: Add more linters URL: https://github.com/apache/airflow-site/pull/75 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] bharathpalaksha opened a new pull request #6310: [AIRFLOW-5621] - Failure callback is not triggered when marked Failed on UI
bharathpalaksha opened a new pull request #6310: [AIRFLOW-5621] - Failure
callback is not triggered when marked Failed on UI
URL: https://github.com/apache/airflow/pull/6310
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW-5621/) issues and
references them in the PR title.
### Description
- [ ] When a task is marked 'Failed' on UI, on_failure_callback on the task
is not triggered.
### Tests
- [ ] My PR adds the following unit tests
test_mark_failure_on_failure_callback
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-5621) When task is marked as failed through UI, failure callback is not triggered
[
https://issues.apache.org/jira/browse/AIRFLOW-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949600#comment-16949600
]
ASF GitHub Bot commented on AIRFLOW-5621:
-
bharathpalaksha commented on pull request #6310: [AIRFLOW-5621] - Failure
callback is not triggered when marked Failed on UI
URL: https://github.com/apache/airflow/pull/6310
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW-5621/) issues and
references them in the PR title.
### Description
- [ ] When a task is marked 'Failed' on UI, on_failure_callback on the task
is not triggered.
### Tests
- [ ] My PR adds the following unit tests
test_mark_failure_on_failure_callback
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> When task is marked as failed through UI, failure callback is not triggered
> ---
>
> Key: AIRFLOW-5621
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5621
> Project: Apache Airflow
> Issue Type: Bug
> Components: core
>Affects Versions: 1.10.2
>Reporter: Bharath Palaksha
>Assignee: Bharath Palaksha
>Priority: Major
>
> When you mark a task as failed - Worker detects and kills the process but -
> on_failure_callback is not honoured.
> on_failure_callback needs to be called to complete it's intended
> functionality.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[airflow-site] branch aip-11 updated: Add forbit-tabs lint (#74)
This is an automated email from the ASF dual-hosted git repository.
kamilbregula pushed a commit to branch aip-11
in repository https://gitbox.apache.org/repos/asf/airflow-site.git
The following commit(s) were added to refs/heads/aip-11 by this push:
new 505decf Add forbit-tabs lint (#74)
505decf is described below
commit 505decf88ab5ce9f8f70bf152b473da4a2e6e893
Author: Kamil Breguła
AuthorDate: Fri Oct 11 17:56:31 2019 +0200
Add forbit-tabs lint (#74)
---
.gitmodules | 4 ++--
.pre-commit-config.yaml | 4
landing-pages/.gitmodules | 4 ++--
landing-pages/site/config.toml| 36 +++
landing-pages/site/content/en/_index.html | 20 -
landing-pages/site/content/no/_index.html | 20 -
6 files changed, 46 insertions(+), 42 deletions(-)
diff --git a/.gitmodules b/.gitmodules
index 55ca2eb..030e750 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -1,3 +1,3 @@
[submodule "landing-pages/site/themes/docsy"]
- path = landing-pages/site/themes/docsy
- url = https://github.com/google/docsy.git
+path = landing-pages/site/themes/docsy
+url = https://github.com/google/docsy.git
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index fd6c2d1..ceeb03e 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -24,6 +24,10 @@ repos:
- repo: meta
hooks:
- id: check-hooks-apply
+ - repo: https://github.com/Lucas-C/pre-commit-hooks
+rev: v1.1.7
+hooks:
+ - id: forbid-tabs
- repo: local
hooks:
- id: shellcheck
diff --git a/landing-pages/.gitmodules b/landing-pages/.gitmodules
index 29c209d..b2afb66 100644
--- a/landing-pages/.gitmodules
+++ b/landing-pages/.gitmodules
@@ -1,3 +1,3 @@
[submodule "site/themes/docsy"]
- path = site/themes/docsy
- url = https://github.com/google/docsy.git
+path = site/themes/docsy
+url = https://github.com/google/docsy.git
diff --git a/landing-pages/site/config.toml b/landing-pages/site/config.toml
index 20c7dc6..62d9cc0 100644
--- a/landing-pages/site/config.toml
+++ b/landing-pages/site/config.toml
@@ -134,34 +134,34 @@ no = 'Sorry to hear that. Please https://github.com/USERNAME/REPOSITORY
[params.links]
# End user relevant links. These will show up on left side of footer and in
the community page if you have one.
[[params.links.user]]
- name = "User mailing list"
- url = "https://example.org/mail;
- icon = "fa fa-envelope"
+name = "User mailing list"
+url = "https://example.org/mail;
+icon = "fa fa-envelope"
desc = "Discussion and help from your fellow users"
[[params.links.user]]
- name ="Twitter"
- url = "https://example.org/twitter;
- icon = "fab fa-twitter"
+name ="Twitter"
+url = "https://example.org/twitter;
+icon = "fab fa-twitter"
desc = "Follow us on Twitter to get the latest news!"
[[params.links.user]]
- name = "Stack Overflow"
- url = "https://example.org/stack;
- icon = "fab fa-stack-overflow"
+name = "Stack Overflow"
+url = "https://example.org/stack;
+icon = "fab fa-stack-overflow"
desc = "Practical questions and curated answers"
# Developer relevant links. These will show up on right side of footer and in
the community page if you have one.
[[params.links.developer]]
- name = "GitHub"
- url = "https://github.com/google/docsy;
- icon = "fab fa-github"
+name = "GitHub"
+url = "https://github.com/google/docsy;
+icon = "fab fa-github"
desc = "Development takes place here!"
[[params.links.developer]]
- name = "Slack"
- url = "https://example.org/slack;
- icon = "fab fa-slack"
+name = "Slack"
+url = "https://example.org/slack;
+icon = "fab fa-slack"
desc = "Chat with other project developers"
[[params.links.developer]]
- name = "Developer mailing list"
- url = "https://example.org/mail;
- icon = "fa fa-envelope"
+name = "Developer mailing list"
+url = "https://example.org/mail;
+icon = "fa fa-envelope"
desc = "Discuss development issues around the project"
diff --git a/landing-pages/site/content/en/_index.html
b/landing-pages/site/content/en/_index.html
index 213f9af..dbdcffd 100644
--- a/landing-pages/site/content/en/_index.html
+++ b/landing-pages/site/content/en/_index.html
@@ -6,16 +6,16 @@ linkTitle = "Goldydocs"
{{< blocks/cover title="Welcome to Goldydocs: A Docsy Example Project!"
image_anchor="top" height="full" color="orange" >}}
- }}">
- Learn More
-
- https://github.com/google/docsy-example;>
- Download
-
- Porridge temperature assessment - in the cloud!
-
- {{< blocks/link-down color="info" >}}
-
+}}">
+Learn More
+
+
[GitHub] [airflow-site] mik-laj merged pull request #74: Add forbit-tabs lint
mik-laj merged pull request #74: Add forbit-tabs lint URL: https://github.com/apache/airflow-site/pull/74 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[airflow-site] branch aip-11 updated: Fix all files permisssions (#73)
This is an automated email from the ASF dual-hosted git repository. kamilbregula pushed a commit to branch aip-11 in repository https://gitbox.apache.org/repos/asf/airflow-site.git The following commit(s) were added to refs/heads/aip-11 by this push: new 6c117f8 Fix all files permisssions (#73) 6c117f8 is described below commit 6c117f81c5f5f6b2f45feff36312a98b89c67263 Author: Kamil Breguła AuthorDate: Fri Oct 11 17:36:17 2019 +0200 Fix all files permisssions (#73) --- landing-pages/site/content/en/blog/news/second-post.md| 0 .../site/content/en/blog/releases/in-depth-monoliths-detailed-spec.md | 0 landing-pages/site/content/en/docs/Examples/_index.md | 0 landing-pages/site/content/en/docs/Tasks/Ponycopters/_index.md| 0 landing-pages/site/content/en/docs/Tasks/_index.md| 0 landing-pages/site/content/en/docs/Tutorials/_index.md| 0 landing-pages/site/content/en/docs/_index.md | 0 landing-pages/site/content/no/docs/_index.md | 0 landing-pages/site/content/no/docs/api-reference/_index.md| 0 landing-pages/site/content/no/docs/api-reference/examples/_index.md | 0 .../no/docs/api-reference/examples/in-depth-monoliths-detailed-spec.md| 0 .../api-reference/examples/the-inside-of-cryptography-detailed-spec.md| 0 .../examples/the-inside-of-microservices-how-does-it-work.md | 0 .../no/docs/api-reference/examples/the-math-of-java-how-does-it-work.md | 0 .../api-reference/examples/the-math-of-monographs-how-does-it-work.md | 0 .../no/docs/api-reference/the-inside-of-java-the-inner-workings.md| 0 .../content/no/docs/api-reference/the-math-of-monoliths-detailed-spec.md | 0 landing-pages/site/content/no/docs/big-data/_index.md | 0 landing-pages/site/content/no/docs/big-data/examples/_index.md| 0 .../no/docs/big-data/examples/the-math-of-monographs-how-does-it-work.md | 0 .../content/no/docs/big-data/the-math-of-monographs-how-does-it-work.md | 0 landing-pages/site/content/no/docs/big-data/tutorials/_index.md | 0 .../content/no/docs/big-data/tutorials/in-depth-go-the-inner-workings.md | 0 .../big-data/tutorials/the-inside-of-cryptography-how-does-it-work.md | 0 .../no/docs/big-data/tutorials/the-inside-of-java-detailed-spec.md| 0 .../docs/big-data/tutorials/the-inside-of-microservices-detailed-spec.md | 0 .../big-data/tutorials/the-math-of-cryptography-the-inner-workings.md | 0 landing-pages/site/content/no/docs/cloud-computing/_index.md | 0 landing-pages/site/content/no/docs/cloud-computing/examples/_index.md | 0 .../cloud-computing/examples/in-depth-cryptography-the-core-concepts.md | 0 .../no/docs/cloud-computing/examples/in-depth-go-how-does-it-work.md | 0 .../cloud-computing/examples/the-inside-of-monoliths-how-does-it-work.md | 0 .../site/content/no/docs/cloud-computing/in-depth-go-detailed-spec.md | 0 .../content/no/docs/cloud-computing/in-depth-monoliths-detailed-spec.md | 0 .../no/docs/cloud-computing/the-inside-of-java-how-does-it-work.md| 0 landing-pages/site/content/no/docs/cloud-computing/tutorials/_index.md| 0 .../cloud-computing/tutorials/in-depth-monographs-the-inner-workings.md | 0 .../docs/cloud-computing/tutorials/the-inside-of-go-the-inner-workings.md | 0 .../tutorials/the-inside-of-microservices-detailed-spec.md| 0 .../docs/cloud-computing/tutorials/the-math-of-java-the-inner-workings.md | 0 .../cloud-computing/tutorials/the-math-of-microservices-detailed-spec.md | 0 .../cloud-computing/tutorials/the-math-of-monographs-detailed-spec.md | 0 .../cloud-computing/tutorials/the-math-of-monoliths-the-inner-workings.md | 0 landing-pages/site/content/no/docs/content-management/_index.md | 0 .../content-management/the-inside-of-cryptography-how-does-it-work.md | 0 .../docs/content-management/the-inside-of-monographs-the-core-concepts.md | 0 .../docs/content-management/the-math-of-microservices-how-does-it-work.md | 0 .../docs/content-management/the-math-of-monoliths-the-inner-workings.md | 0 landing-pages/site/content/no/docs/content-management/tutorials/_index.md | 0 .../tutorials/in-depth-microservices-the-core-concepts.md | 0 .../tutorials/the-inside-of-cryptography-the-core-concepts.md | 0 .../tutorials/the-inside-of-microservices-the-core-concepts.md| 0 .../tutorials/the-inside-of-recursion-the-inner-workings.md | 0 landing-pages/site/content/no/docs/cross-platform/_index.md | 0 landing-pages/site/content/no/docs/cross-platform/examples/_index.md | 0 .../no/docs/cross-platform/examples/in-depth-java-detailed-spec.md| 0 .../docs/cross-platform/examples/the-inside-of-java-the-core-concepts.md | 0
[GitHub] [airflow-site] mik-laj merged pull request #73: Fix all files permisssions
mik-laj merged pull request #73: Fix all files permisssions URL: https://github.com/apache/airflow-site/pull/73 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-5640) BaseOperator email parameter is wrongly typed and not documented
Cedrik Neumann created AIRFLOW-5640:
---
Summary: BaseOperator email parameter is wrongly typed and not
documented
Key: AIRFLOW-5640
URL: https://issues.apache.org/jira/browse/AIRFLOW-5640
Project: Apache Airflow
Issue Type: Bug
Components: operators
Affects Versions: 1.10.5
Reporter: Cedrik Neumann
The {{email}} field is not documented in BaseOperator and furthermore the type
annotation {{str}} is wrong
[here|https://github.com/apache/airflow/blob/master/airflow/models/baseoperator.py#L273].
The method {{get_email_address_list}} clearly accepts lists of strings as well
as comma and semicolon delimited lists:
[here|https://github.com/apache/airflow/blob/88989200a66291580088188f06a6db503ac823e2/airflow/utils/email.py#L123]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] codecov-io edited a comment on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized DAG in DB for webserver scalability
codecov-io edited a comment on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized DAG in DB for webserver scalability URL: https://github.com/apache/airflow/pull/5743#issuecomment-529755241 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5743?src=pr=h1) Report > Merging [#5743](https://codecov.io/gh/apache/airflow/pull/5743?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/7c9b44dbfc72df951aadbda006229977beb6d47c?src=pr=desc) will **decrease** coverage by `70.72%`. > The diff coverage is `14.95%`. [](https://codecov.io/gh/apache/airflow/pull/5743?src=pr=tree) ```diff @@Coverage Diff @@ ## master #5743 +/- ## == - Coverage 80.41% 9.68% -70.73% == Files 612 617+5 Lines 35473 36018 +545 == - Hits285253490-25035 - Misses 6948 32528+25580 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5743?src=pr=tree) | Coverage Δ | | |---|---|---| | [...rflow/dag/serialization/serialized\_baseoperator.py](https://codecov.io/gh/apache/airflow/pull/5743/diff?src=pr=tree#diff-YWlyZmxvdy9kYWcvc2VyaWFsaXphdGlvbi9zZXJpYWxpemVkX2Jhc2VvcGVyYXRvci5weQ==) | `0% <0%> (-56.67%)` | :arrow_down: | | [airflow/www/utils.py](https://codecov.io/gh/apache/airflow/pull/5743/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdXRpbHMucHk=) | `0% <0%> (-75.61%)` | :arrow_down: | | [airflow/gcp/operators/cloud\_sql.py](https://codecov.io/gh/apache/airflow/pull/5743/diff?src=pr=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2Nsb3VkX3NxbC5weQ==) | `0% <0%> (-88.27%)` | :arrow_down: | | [airflow/dag/serialization/serialized\_dag.py](https://codecov.io/gh/apache/airflow/pull/5743/diff?src=pr=tree#diff-YWlyZmxvdy9kYWcvc2VyaWFsaXphdGlvbi9zZXJpYWxpemVkX2RhZy5weQ==) | `0% <0%> (-54.55%)` | :arrow_down: | | [airflow/dag/serialization/json\_schema.py](https://codecov.io/gh/apache/airflow/pull/5743/diff?src=pr=tree#diff-YWlyZmxvdy9kYWcvc2VyaWFsaXphdGlvbi9qc29uX3NjaGVtYS5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/dag/serialization/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/5743/diff?src=pr=tree#diff-YWlyZmxvdy9kYWcvc2VyaWFsaXphdGlvbi9fX2luaXRfXy5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/dag/serialization/enums.py](https://codecov.io/gh/apache/airflow/pull/5743/diff?src=pr=tree#diff-YWlyZmxvdy9kYWcvc2VyaWFsaXphdGlvbi9lbnVtcy5weQ==) | `0% <0%> (ø)` | | | [airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/5743/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==) | `0% <0%> (-65.23%)` | :arrow_down: | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/5743/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `0% <0%> (-75.2%)` | :arrow_down: | | [airflow/dag/serialization/serialization.py](https://codecov.io/gh/apache/airflow/pull/5743/diff?src=pr=tree#diff-YWlyZmxvdy9kYWcvc2VyaWFsaXphdGlvbi9zZXJpYWxpemF0aW9uLnB5) | `0% <0%> (-30.51%)` | :arrow_down: | | ... and [522 more](https://codecov.io/gh/apache/airflow/pull/5743/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5743?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5743?src=pr=footer). Last update [7c9b44d...e049fb8](https://codecov.io/gh/apache/airflow/pull/5743?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-3783) Switch to HEADER arg to unload data from Redshift to S3
[ https://issues.apache.org/jira/browse/AIRFLOW-3783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-3783: -- Component/s: operators > Switch to HEADER arg to unload data from Redshift to S3 > --- > > Key: AIRFLOW-3783 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3783 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Affects Versions: 1.10.5 >Reporter: Shreya Chakraborty >Assignee: Felix Uellendall >Priority: Major > > Redshift has added a new *HEADER* option to the s3unload parameters. > So we can remove the default PARALLEL OFF in the existing > RedshiftToS3Transfer operator. The unload operation can now run parallely, > hence faster, along with headers, if required. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] feluelle commented on issue #6309: [AIRFLOW-3783][AIRFLOW-5395] Switch to HEADER arg to unload data from Redshift to S3
feluelle commented on issue #6309: [AIRFLOW-3783][AIRFLOW-5395] Switch to HEADER arg to unload data from Redshift to S3 URL: https://github.com/apache/airflow/pull/6309#issuecomment-541094348 FYI @kurtqq This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5395) Fix unloading with column names containing a dot in RedshiftToS3Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-5395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949504#comment-16949504 ] ASF GitHub Bot commented on AIRFLOW-5395: - feluelle commented on pull request #5994: [AIRFLOW-5395] Fix unloading with column names containing a dot in RedshiftToS3Operator URL: https://github.com/apache/airflow/pull/5994 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Fix unloading with column names containing a dot in RedshiftToS3Operator > > > Key: AIRFLOW-5395 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5395 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Affects Versions: 1.10.5 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (AIRFLOW-5395) Fix unloading with column names containing a dot in RedshiftToS3Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-5395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-5395 started by Felix Uellendall. - > Fix unloading with column names containing a dot in RedshiftToS3Operator > > > Key: AIRFLOW-5395 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5395 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Affects Versions: 1.10.5 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-3783) Switch to HEADER arg to unload data from Redshift to S3
[ https://issues.apache.org/jira/browse/AIRFLOW-3783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-3783: -- Affects Version/s: 1.10.5 > Switch to HEADER arg to unload data from Redshift to S3 > --- > > Key: AIRFLOW-3783 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3783 > Project: Apache Airflow > Issue Type: Improvement >Affects Versions: 1.10.5 >Reporter: Shreya Chakraborty >Assignee: Felix Uellendall >Priority: Major > > Redshift has added a new *HEADER* option to the s3unload parameters. > So we can remove the default PARALLEL OFF in the existing > RedshiftToS3Transfer operator. The unload operation can now run parallely, > hence faster, along with headers, if required. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] feluelle commented on issue #5994: [AIRFLOW-5395] Fix unloading with column names containing a dot in RedshiftToS3Operator
feluelle commented on issue #5994: [AIRFLOW-5395] Fix unloading with column names containing a dot in RedshiftToS3Operator URL: https://github.com/apache/airflow/pull/5994#issuecomment-541087863 Closed in favor of #6309 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3783) Switch to HEADER arg to unload data from Redshift to S3
[ https://issues.apache.org/jira/browse/AIRFLOW-3783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949503#comment-16949503 ] ASF GitHub Bot commented on AIRFLOW-3783: - feluelle commented on pull request #4610: [AIRFLOW-3783] Switch to HEADER option being provided already by Redshift URL: https://github.com/apache/airflow/pull/4610 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Switch to HEADER arg to unload data from Redshift to S3 > --- > > Key: AIRFLOW-3783 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3783 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Shreya Chakraborty >Assignee: Felix Uellendall >Priority: Major > > Redshift has added a new *HEADER* option to the s3unload parameters. > So we can remove the default PARALLEL OFF in the existing > RedshiftToS3Transfer operator. The unload operation can now run parallely, > hence faster, along with headers, if required. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] feluelle closed pull request #5994: [AIRFLOW-5395] Fix unloading with column names containing a dot in RedshiftToS3Operator
feluelle closed pull request #5994: [AIRFLOW-5395] Fix unloading with column names containing a dot in RedshiftToS3Operator URL: https://github.com/apache/airflow/pull/5994 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle commented on issue #4610: [AIRFLOW-3783] Switch to HEADER option being provided already by Redshift
feluelle commented on issue #4610: [AIRFLOW-3783] Switch to HEADER option being provided already by Redshift URL: https://github.com/apache/airflow/pull/4610#issuecomment-541087610 Closed in favor of #6309 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle closed pull request #4610: [AIRFLOW-3783] Switch to HEADER option being provided already by Redshift
feluelle closed pull request #4610: [AIRFLOW-3783] Switch to HEADER option being provided already by Redshift URL: https://github.com/apache/airflow/pull/4610 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3783) Switch to HEADER arg to unload data from Redshift to S3
[
https://issues.apache.org/jira/browse/AIRFLOW-3783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949500#comment-16949500
]
ASF GitHub Bot commented on AIRFLOW-3783:
-
feluelle commented on pull request #6309: [AIRFLOW-3783][AIRFLOW-5395] Switch
to HEADER arg to unload data from Redshift to S3
URL: https://github.com/apache/airflow/pull/6309
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3783
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
This PR adapts #4610 and resolves its open issues, because the original
author stopped working on it.
It also fixes issue
[AIRFLOW-5395](https://issues.apache.org/jira/browse/AIRFLOW-5395) which was
addressed in PR #5994
Changes:
- replace the default unload_option 'PARALLEL OFF' by 'HEADER'
- fix pylint issues
- update docs
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
`./breeze --backend postgres --test-target
tests.operators.test_redshift_to_s3_operator`
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
CC: @shrechak
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Switch to HEADER arg to unload data from Redshift to S3
> ---
>
> Key: AIRFLOW-3783
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3783
> Project: Apache Airflow
> Issue Type: Improvement
>Reporter: Shreya Chakraborty
>Assignee: Felix Uellendall
>Priority: Major
>
> Redshift has added a new *HEADER* option to the s3unload parameters.
> So we can remove the default PARALLEL OFF in the existing
> RedshiftToS3Transfer operator. The unload operation can now run parallely,
> hence faster, along with headers, if required.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] feluelle opened a new pull request #6309: [AIRFLOW-3783][AIRFLOW-5395] Switch to HEADER arg to unload data from Redshift to S3
feluelle opened a new pull request #6309: [AIRFLOW-3783][AIRFLOW-5395] Switch
to HEADER arg to unload data from Redshift to S3
URL: https://github.com/apache/airflow/pull/6309
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3783
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
This PR adapts #4610 and resolves its open issues, because the original
author stopped working on it.
It also fixes issue
[AIRFLOW-5395](https://issues.apache.org/jira/browse/AIRFLOW-5395) which was
addressed in PR #5994
Changes:
- replace the default unload_option 'PARALLEL OFF' by 'HEADER'
- fix pylint issues
- update docs
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
`./breeze --backend postgres --test-target
tests.operators.test_redshift_to_s3_operator`
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
CC: @shrechak
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6295: [AIRFLOW-XXX] Adding Task re-run documentation
codecov-io edited a comment on issue #6295: [AIRFLOW-XXX] Adding Task re-run documentation URL: https://github.com/apache/airflow/pull/6295#issuecomment-540752672 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6295?src=pr=h1) Report > Merging [#6295](https://codecov.io/gh/apache/airflow/pull/6295?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/7c9b44dbfc72df951aadbda006229977beb6d47c?src=pr=desc) will **decrease** coverage by `0.07%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6295?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6295 +/- ## == - Coverage 80.41% 80.34% -0.08% == Files 612 616 +4 Lines 3547335733 +260 == + Hits2852528709 +184 - Misses 6948 7024 +76 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6295?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/6295/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==) | `58.89% <0%> (-6.34%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6295/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `89.9% <0%> (-1.53%)` | :arrow_down: | | [airflow/gcp/operators/bigquery.py](https://codecov.io/gh/apache/airflow/pull/6295/diff?src=pr=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2JpZ3F1ZXJ5LnB5) | `86.6% <0%> (ø)` | :arrow_up: | | [...gle/marketing\_platform/sensors/campaign\_manager.py](https://codecov.io/gh/apache/airflow/pull/6295/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlci9nb29nbGUvbWFya2V0aW5nX3BsYXRmb3JtL3NlbnNvcnMvY2FtcGFpZ25fbWFuYWdlci5weQ==) | `100% <0%> (ø)` | | | [...oogle/marketing\_platform/hooks/campaign\_manager.py](https://codecov.io/gh/apache/airflow/pull/6295/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlci9nb29nbGUvbWFya2V0aW5nX3BsYXRmb3JtL2hvb2tzL2NhbXBhaWduX21hbmFnZXIucHk=) | `100% <0%> (ø)` | | | [...\_platform/example\_dags/example\_campaign\_manager.py](https://codecov.io/gh/apache/airflow/pull/6295/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlci9nb29nbGUvbWFya2V0aW5nX3BsYXRmb3JtL2V4YW1wbGVfZGFncy9leGFtcGxlX2NhbXBhaWduX21hbmFnZXIucHk=) | `0% <0%> (ø)` | | | [...e/marketing\_platform/operators/campaign\_manager.py](https://codecov.io/gh/apache/airflow/pull/6295/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlci9nb29nbGUvbWFya2V0aW5nX3BsYXRmb3JtL29wZXJhdG9ycy9jYW1wYWlnbl9tYW5hZ2VyLnB5) | `91.73% <0%> (ø)` | | | [airflow/gcp/operators/cloud\_sql.py](https://codecov.io/gh/apache/airflow/pull/6295/diff?src=pr=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2Nsb3VkX3NxbC5weQ==) | `88.98% <0%> (+0.72%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6295?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6295?src=pr=footer). Last update [7c9b44d...52b0bb5](https://codecov.io/gh/apache/airflow/pull/6295?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #6090: [AIRFLOW-5470] Add Apache Livy REST operator
mik-laj commented on issue #6090: [AIRFLOW-5470] Add Apache Livy REST operator URL: https://github.com/apache/airflow/pull/6090#issuecomment-541086750 Can you do a rebase? We have new rules on master brach. It is related to operators-and-hooks-ref.rst. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] lucacavazzana commented on a change in pull request #6090: [AIRFLOW-5470] Add Apache Livy REST operator
lucacavazzana commented on a change in pull request #6090: [AIRFLOW-5470] Add
Apache Livy REST operator
URL: https://github.com/apache/airflow/pull/6090#discussion_r334014562
##
File path: airflow/contrib/hooks/livy_hook.py
##
@@ -0,0 +1,297 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+This module contains the Apache Livy hook.
+"""
+
+import re
+from enum import Enum
+import json
+import requests
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class BatchState(Enum):
+"""
+Batch session states
+"""
+NOT_STARTED = 'not_started'
+STARTING = 'starting'
+RUNNING = 'running'
+IDLE = 'idle'
+BUSY = 'busy'
+SHUTTING_DOWN = 'shutting_down'
+ERROR = 'error'
+DEAD = 'dead'
+KILLED = 'killed'
+SUCCESS = 'success'
+
+
+TERMINAL_STATES = {
+BatchState.SUCCESS,
+BatchState.DEAD,
+BatchState.KILLED,
+BatchState.ERROR,
+}
+
+
+class LivyHook(BaseHook, LoggingMixin):
+"""
+Hook for Apache Livy through the REST API.
+
+For more information about the API refer to
+https://livy.apache.org/docs/latest/rest-api.html
+
+:param livy_conn_id: reference to a pre-defined Livy Connection.
+:type livy_conn_id: str
+"""
+def __init__(self, livy_conn_id='livy_default'):
+super(LivyHook, self).__init__(livy_conn_id)
+self._livy_conn_id = livy_conn_id
+self._build_base_url()
+
+def _build_base_url(self):
+"""
+Build connection URL
+"""
+params = self.get_connection(self._livy_conn_id)
+
+base_url = params.host
+
+if not base_url:
+raise AirflowException("Missing Livy endpoint hostname")
+
+if '://' not in base_url:
+base_url = '{}://{}'.format('http', base_url)
+if not re.search(r':\d+$', base_url):
+base_url = '{}:{}'.format(base_url, str(params.port or 8998))
+
+self._base_url = base_url
+
+def get_conn(self):
+pass
+
+def post_batch(self, *args, **kwargs):
+"""
+Perform request to submit batch
+"""
+
+batch_submit_body = json.dumps(LivyHook.build_post_batch_body(*args,
**kwargs))
+headers = {'Content-Type': 'application/json'}
+
+self.log.info("Submitting job {} to {}".format(batch_submit_body,
self._base_url))
+response = requests.post(self._base_url + '/batches',
data=batch_submit_body, headers=headers)
Review comment:
the current HttpHook implementation would try to put the `auth` as a header.
Probably the best solution is to update directly HttpHook, but that would
require either adding a new field to the connection or change the `extra` usage
(which would break someone's code)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (AIRFLOW-4771) Initialization of hook in the GCP operator constructors
[
https://issues.apache.org/jira/browse/AIRFLOW-4771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor updated AIRFLOW-4771:
---
Fix Version/s: (was: 1.10.6)
2.0.0
This one is not easy to backport due to all the renames that have gone on with
GCP.
> Initialization of hook in the GCP operator constructors
> ---
>
> Key: AIRFLOW-4771
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4771
> Project: Apache Airflow
> Issue Type: Bug
> Components: gcp
>Affects Versions: 1.10.3
>Reporter: Kamil Bregula
>Priority: Major
> Fix For: 2.0.0
>
>
> Hello,
> This affects following operators:
> * BigtableInstanceCreateOperator
> * BigtableInstanceDeleteOperator
> * BigtableTableCreateOperator
> * BigtableTableDeleteOperator
> * BigtableClusterUpdateOperator
> * BigtableTableWaitForReplicationSensor
> * GceBaseOperator
> * GcfFunctionDeployOperator
> * GcfFunctionDeleteOperator
> * CloudSpannerInstanceDeployOperator
> * CloudSpannerInstanceDeleteOperator
> * CloudSpannerInstanceDatabaseQueryOperator
> * CloudSpannerInstanceDatabaseDeployOperator
> * CloudSpannerInstanceDatabaseUpdateOperator
> * CloudSpannerInstanceDatabaseDeleteOperator
> * CloudSqlBaseOperator
> * CloudVisionProductSetCreateOperator
> * and more...
> This is not good practice for two reasons:
> * The gcp_conn_id parameter can not be a Jinja template. Templates are
> resolved before calling the execute method, but after constrcutor.
> * Hook initialization calls the database connection, because In the
> GoogleCloudBaseHook class, we have a code snippet:
> {code:java}
> def __init__(self, gcp_conn_id='google_cloud_default', delegate_to=None):
> self.gcp_conn_id = gcp_conn_id
> self.delegate_to = delegate_to
> self.extras = self.get_connection(self.gcp_conn_id).extra_dejson
> {code}
> self.get_connection triggers a database query.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Reopened] (AIRFLOW-5423) Type annotations for GCP sensors
[ https://issues.apache.org/jira/browse/AIRFLOW-5423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor reopened AIRFLOW-5423: > Type annotations for GCP sensors > > > Key: AIRFLOW-5423 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5423 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.5 >Reporter: Tobiasz Kedzierski >Assignee: Tobiasz Kedzierski >Priority: Minor > Fix For: 1.10.6 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5423) Type annotations for GCP sensors
[ https://issues.apache.org/jira/browse/AIRFLOW-5423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor resolved AIRFLOW-5423. Fix Version/s: (was: 1.10.6) 2.0.0 Resolution: Fixed > Type annotations for GCP sensors > > > Key: AIRFLOW-5423 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5423 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.5 >Reporter: Tobiasz Kedzierski >Assignee: Tobiasz Kedzierski >Priority: Minor > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5624) Use mock decorator in MLEngine operators test
[ https://issues.apache.org/jira/browse/AIRFLOW-5624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949479#comment-16949479 ] ASF GitHub Bot commented on AIRFLOW-5624: - potiuk commented on pull request #6292: [AIRFLOW-5624] Use mock decorator in MLEngine operators tests URL: https://github.com/apache/airflow/pull/6292 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Use mock decorator in MLEngine operators test > - > > Key: AIRFLOW-5624 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5624 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-3783) Switch to HEADER arg to unload data from Redshift to S3
[ https://issues.apache.org/jira/browse/AIRFLOW-3783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-3783: -- Summary: Switch to HEADER arg to unload data from Redshift to S3 (was: Fix header rendering in the redshift-to-s3 plugin) > Switch to HEADER arg to unload data from Redshift to S3 > --- > > Key: AIRFLOW-3783 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3783 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Shreya Chakraborty >Assignee: Felix Uellendall >Priority: Major > > Redshift has added a new *HEADER* option to the s3unload parameters. > So we can remove the default PARALLEL OFF in the existing > RedshiftToS3Transfer operator. The unload operation can now run parallely, > hence faster, along with headers, if required. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-3978) Add missing types in MySqlToGoogleCloudStorageOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-3978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949471#comment-16949471 ] Jarek Potiuk commented on AIRFLOW-3978: --- >From looking at the code - no matter what type field it is, if the field is >represented by "bytes", it will be automatically converted into base64 encoded >strings, which is exactly what BigQuery expects. I guess that should cover the >BINARY and VARBINARY case. [~RosterIn] - can you please verify if the case can be closed? If so - I will close the issue. > Add missing types in MySqlToGoogleCloudStorageOperator > -- > > Key: AIRFLOW-3978 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3978 > Project: Apache Airflow > Issue Type: Bug > Components: gcp >Affects Versions: 1.10.2 >Reporter: Roster >Assignee: Roster >Priority: Minor > Labels: gcs > > There fields are missing and can not be mapped: > TIME, BINARY , VARBINARY -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-3978) Add missing types in MySqlToGoogleCloudStorageOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-3978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk closed AIRFLOW-3978. - Resolution: Won't Fix Closing for now - [~RosterIn], can you please reopen if you think this is still an issue? > Add missing types in MySqlToGoogleCloudStorageOperator > -- > > Key: AIRFLOW-3978 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3978 > Project: Apache Airflow > Issue Type: Bug > Components: gcp >Affects Versions: 1.10.2 >Reporter: Roster >Assignee: Roster >Priority: Minor > Labels: gcs > > There fields are missing and can not be mapped: > TIME, BINARY , VARBINARY -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5624) Use mock decorator in MLEngine operators test
[ https://issues.apache.org/jira/browse/AIRFLOW-5624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5624. --- Fix Version/s: 2.0.0 Resolution: Fixed > Use mock decorator in MLEngine operators test > - > > Key: AIRFLOW-5624 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5624 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5624) Use mock decorator in MLEngine operators test
[ https://issues.apache.org/jira/browse/AIRFLOW-5624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949480#comment-16949480 ] ASF subversion and git services commented on AIRFLOW-5624: -- Commit c0d98a72394b68df27db4d1cab4a94021980e0fb in airflow's branch refs/heads/master from Kamil Breguła [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=c0d98a7 ] [AIRFLOW-5624] Use mock decorator in MLEngine operators tests (#6292) > Use mock decorator in MLEngine operators test > - > > Key: AIRFLOW-5624 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5624 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk merged pull request #6292: [AIRFLOW-5624] Use mock decorator in MLEngine operators tests
potiuk merged pull request #6292: [AIRFLOW-5624] Use mock decorator in MLEngine operators tests URL: https://github.com/apache/airflow/pull/6292 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-3978) Add missing types in MySqlToGoogleCloudStorageOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-3978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-3978. --- Resolution: Not A Problem > Add missing types in MySqlToGoogleCloudStorageOperator > -- > > Key: AIRFLOW-3978 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3978 > Project: Apache Airflow > Issue Type: Bug > Components: gcp >Affects Versions: 1.10.2 >Reporter: Roster >Assignee: Roster >Priority: Minor > Labels: gcs > > There fields are missing and can not be mapped: > TIME, BINARY , VARBINARY -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (AIRFLOW-3978) Add missing types in MySqlToGoogleCloudStorageOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-3978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk reopened AIRFLOW-3978: --- > Add missing types in MySqlToGoogleCloudStorageOperator > -- > > Key: AIRFLOW-3978 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3978 > Project: Apache Airflow > Issue Type: Bug > Components: gcp >Affects Versions: 1.10.2 >Reporter: Roster >Assignee: Roster >Priority: Minor > Labels: gcs > > There fields are missing and can not be mapped: > TIME, BINARY , VARBINARY -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] kaxil commented on a change in pull request #6302: [AIRFLOW-5636] Allow adding or overriding existing Operator Links for…
kaxil commented on a change in pull request #6302: [AIRFLOW-5636] Allow adding
or overriding existing Operator Links for…
URL: https://github.com/apache/airflow/pull/6302#discussion_r333998545
##
File path: tests/www/test_views.py
##
@@ -1854,6 +1856,24 @@ def test_extra_links_no_response(self,
get_dag_function):
'url': None,
'error': 'No URL found for no_response'})
+@mock.patch('airflow.www.views.dagbag.get_dag')
+def test_operator_extra_link_override_plugin(self, get_dag_function):
+get_dag_function.return_value = self.dag
+
+response = self.client.get(
+
"{0}?dag_id={1}_id={2}_date={3}_name=airflow".format(
+self.ENDPOINT, self.dag.dag_id, self.task_2.task_id,
self.DEFAULT_DATE),
+follow_redirects=True)
+
+self.assertEqual(response.status_code, 200)
+response_str = response.data
+if isinstance(response.data, bytes):
+response_str = response_str.decode()
+self.assertEqual(json.loads(response_str), {
+'url': 'https://airflow.apache.org/1.10.5/',
Review comment:
Added the comment
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
